Zimbrabackend and ZCS 8.7.0 Support

[6125amartin]

Hello,

I recently upgraded to ZCS 8.7.0 on Ubuntu 14.04. After the upgrade, I'm seeing a lot of errors in /var/log/z-push/z-push-error.log of this format:

Code: Select all

29/08/2016 15:06:52 [ 2889] [ERROR] [user1] Zimbra->SoapRequest(): SOAP FAULT: Error Code Returned [mail.NO_SUCH_WAITSET]
29/08/2016 15:06:52 [ 2889] [ERROR] [user1] Zimbra->SoapRequest(): SOAP FAULT: Error Code Returned [service.PERM_DENIED]
29/08/2016 15:06:52 [ 4700] [ERROR] [user2] Zimbra->SoapRequest(): SOAP FAULT: Error Code Returned [admin.NO_SUCH_WAITSET]
29/08/2016 15:06:52 [ 2889] [ERROR] [user1] Zimbra->SoapRequest(): SOAP FAULT: Error Code Returned [admin.NO_SUCH_WAITSET]
29/08/2016 15:06:52 [ 4469] [ERROR] [user3] Zimbra->SoapRequest(): SOAP FAULT: Error Code Returned [mail.NO_SUCH_WAITSET]
29/08/2016 15:06:52 [ 4860] [ERROR] [user4] Zimbra->SoapRequest(): SOAP FAULT: HTML Error Returned - Error 504 Connection to Upstream is Time Out - Enable ZIMBRA_DEBUG for more details - returning { false }
29/08/2016 15:06:52 [ 2889] [ERROR] [user1] Zimbra->SoapRequest(): SOAP FAULT: HTML Error Returned - Error 502 Connection to Upstream is Refused - Enable ZIMBRA_DEBUG for more details - returning { false }
29/08/2016 15:06:52 [ 2889] [ERROR] [user1] Zimbra->SoapRequest(): SOAP FAULT: HTML Error Returned - Error 502 Connection to Upstream is Refused - Enable ZIMBRA_DEBUG for more details - returning { false }

Enabling ZIMBRA_DEBUG doesn't seem to produce any additional output in this log. Syncing mail with mobile clients appears to work fine, but I'm also seeing these 502 errors in the web client as well. Has anyone else tested z-push + zimbrabackend with ZCS 8.7.0?

[liverpoolfcfan]

The 50n errors look like nginx errors returned when it is having a hard time communicating with the mailbox server.

Which zimbra release did you update from?

Which z-push and zimbra backend releases are you running?

[liverpoolfcfan]

For the 504 error the most likely culprit is the nginx upstream read timeout. It defaults to just 60 seconds on a fresh install. For an ActiveSync connection that must allow up to 3600 seconds.

Code: Select all

su - zimbra
[zimbra@centos7 ~]$ zmprov gcf zimbraReverseProxyUpstreamReadTimeout
zimbraReverseProxyUpstreamReadTimeout: 60s


You need to change this setting to 3600s

Code: Select all

[zimbra@centos7 ~]$ zmprov mcf zimbraReverseProxyUpstreamReadTimeout 3600s
[zimbra@centos7 ~]$ zmprov gcf zimbraReverseProxyUpstreamReadTimeout
zimbraReverseProxyUpstreamReadTimeout: 3600s


That should eliminate the 504 errors.

502 errors usually occur when the MTA is stopping - it first refuses connections from the proxy. Could you have tweaked a setting that caused an MTA restart?

I would change the setting above and then monitor the logs to see if you still see any errors.

[6125amartin]

Thanks for the help, I will try that out! I actually ended up disabling zimbra-proxy after the upgrade due to a constant flood of 502 errors in the web client, see this thread. None of the debugging I tried with help from the #zimbra IRC worked to get the proxy functioning. The odd thing is the 502 errors would occur intermittently - e.g. if you got a 502 error, dismissed it, and then tried the exact same action again, it would succeed. I wasn't tweaking settings frequently enough to trigger as many mailboxd restarts as 502 error frequency. Note that stopping z-push did seem to significantly decrease the number of 502s we were seeing in the web client. Any ideas on how or if the 502 may be related to z-push traffic?

[liverpoolfcfan]

No idea - but I have just started testing with zimbra 8.7. I just ran into the 504 myself last night and quickly realized what the issue was there. I haven't seen a 502 as yet. But at the end of the day z-push is simply like an extra set of web client users. It logs in the same way through https/https so anything that affects the web client will affect z-push and vice-versa.

One thought - did you increase the number of soap sessions allowed? Search for SOAP sessions in the INSTALL notes

[liverpoolfcfan]

Another possibility I guess is that it is directly related to the 504 timeout error. The proxy will have closed it's session due to the timeout, and thereafter z-push is making another request on the same session as it assumes the session is still alive. It could be that the request is being refused because the session is no longer valid.

I would suggest upping the session read timeout as mentioned above and then re-testing to see if it resolves both problems.

[liverpoolfcfan]

I also see another new setting on 8.7 that was not on 8.6

zimbraReverseProxySSLSessionTimeout: 10m

I am not aware what this setting controls - but this may also need to be bumped up to 60m. If you have a ticket open with zimbra perhaps you could ask them.

[6125amartin]

Which zimbra release did you update from?

From 8.0.6 on Ubuntu 12.04 to 8.7.0 on Ubuntu 14.04 (with an intermediate stop at 8.7.0 on 12.04 as part of the upgrade process)

Which z-push and zimbra backend releases are you running?

Z-Push 2.2.9 and zimbrabackend 64

One thought - did you increase the number of soap sessions allowed? Search for SOAP sessions in the INSTALL notes

I don't see an INSTALL file provided in the tarball (nor is it in README.txt) - where did you see this SOAP sessions info mentioned?

Another possibility I guess is that it is directly related to the 504 timeout error. The proxy will have closed it's session due to the timeout, and thereafter z-push is making another request on the same session as it assumes the session is still alive. It could be that the request is being refused because the session is no longer valid.

Would this also cause web clients to frequently experience 502s? Wouldn't the sessions for zpush be separate from the sessions for the web client? I even had one user do a test - leave z-push enabled but turn airplane mode on on his mobile so no more z-push traffic for his user account would go through. Even with airplane mode on, he still saw 502 errors in the web interface with no apparent change in frequency.

I also see another new setting on 8.7 that was not on 8.6

zimbraReverseProxySSLSessionTimeout: 10m

I am not aware what this setting controls - but this may also need to be bumped up to 60m. If you have a ticket open with zimbra perhaps you could ask them.

I'm using OSE so unfortunately no official support from the Zimbra developers. I can try increasing zimbraReverseProxyUpstreamReadTimeout and zimbraReverseProxySSLSessionTimeout, but I'll have to set up a separate test instance since I disabled zimbra-proxy to get this working on the main server which is now in production.

[liverpoolfcfan]

There are 2 copies of the INSTALL text file in the Release 64 tarball. One at the top level, and another in the z-push-2 folder.

In any case, here is the relevant note

ZIMBRA 7.2 AND LATER - Configure Max SOAP Sessions per User
If you have users who have multiple devices synching against their account at the same time there is
a potential issue with the number of allowed SOAP sessions per user. This was highlighted in a zimbra
bugzilla ticket and was fixed in zimbra release 7.2 and later - where you can now increase the number.
See https://bugzilla.zimbra.com/show_bug.cgi?id=75338 ; Providing the ability to increase the number
of Soap sessions. I set the value to 7 on my server and typically don't see the issue any more.
Typical evidence of this problem is sync loops and content disappearing from the view pane of the Web
client while a device is synching through activesync.

And, since you mentioned it earlier in the ticket I am sure you are aware of the need to white list the z-push box

ZIMBRA 8.0 AND LATER - DoSFilter MUST whitelist z-push server
Assuming that z-push is not running on the same IP address as zimbra, if you are running
zimbra 8.0 or later, you MUST whitelist the IP Address of the z-push server to avoid
connection issues. See http://wiki.zimbra.com/wiki/DoSFilter

Point on sessions getting closed is that z-push will open a session - after 60 seconds it will close it - z-push will continue on believing that the session is still alive and will try to send commands to it. On the first attempt of z-push to talk to zimbra after nginx has closed the session nginx returns the 504 error. I am just guessing that it may subsequently try to pass the request back to zimbra but maybe zimbra refuses it as there is no open session. Please note that this is just a best guess.

I subsequently found info on the nginx web site that suggests the SSL Session Timeout is only related to cacheing SSL credentials. I believe it has no bearing on the duration of a session.

[6125amartin]

There are 2 copies of the INSTALL text file in the Release 64 tarball. One at the top level, and another in the z-push-2 folder.

In any case, here is the relevant note

ZIMBRA 7.2 AND LATER - Configure Max SOAP Sessions per User
If you have users who have multiple devices synching against their account at the same time there is
a potential issue with the number of allowed SOAP sessions per user. This was highlighted in a zimbra
bugzilla ticket and was fixed in zimbra release 7.2 and later - where you can now increase the number.
See https://bugzilla.zimbra.com/show_bug.cgi?id=75338 ; Providing the ability to increase the number
of Soap sessions. I set the value to 7 on my server and typically don't see the issue any more.
Typical evidence of this problem is sync loops and content disappearing from the view pane of the Web
client while a device is synching through activesync.

Confirmed, zimbra_session_limit_soap is set to the default of 5 on my install.

And, since you mentioned it earlier in the ticket I am sure you are aware of the need to white list the z-push box

ZIMBRA 8.0 AND LATER - DoSFilter MUST whitelist z-push server
Assuming that z-push is not running on the same IP address as zimbra, if you are running
zimbra 8.0 or later, you MUST whitelist the IP Address of the z-push server to avoid
connection issues. See http://wiki.zimbra.com/wiki/DoSFilter

Yes, I added z-push's IP address to zimbraHttpThrottleSafeIPs (but the 502 errors continued after it was added).

Point on sessions getting closed is that z-push will open a session - after 60 seconds it will close it - z-push will continue on believing that the session is still alive and will try to send commands to it. On the first attempt of z-push to talk to zimbra after nginx has closed the session nginx returns the 504 error. I am just guessing that it may subsequently try to pass the request back to zimbra but maybe zimbra refuses it as there is no open session. Please note that this is just a best guess.

Okay, that seems like a reasonable hypothesis. Would this somehow interact with the user when logged into the webui too?

I subsequently found info on the nginx web site that suggests the SSL Session Timeout is only related to cacheing SSL credentials. I believe it has no bearing on the duration of a session.

So zimbraReverseProxySSLSessionTimeout probably isn't related and won't help?

I'll try tweaking zimbra_session_limit_soap and zimbraReverseProxyUpstreamReadTimeout and see if I can improve the behavior with those tunables, if I can reproduce the problem on my test server