Page 1 of 1

zimbra proxy don't fail-over memecached

Posted: Thu Apr 20, 2017 7:53 am
by maxyang2008
Hi buddies,

I am testing zimbra version zcs-8.7.5_GA_1764
I have a 3 node zimbra setup like below:
    ldap: zimbra-ldap
    mailbox&proxy server 1 : zimbra-mailbox
    mailbox&proxy server 2 : zimbra-mailbox2

services running on each node are as below:

Code: Select all

[zimbra@zimbra-ldap ~]$ zmcontrol status
Host zimbra-ldap.example.com
   ldap                    Running
   stats                   Running
   zmconfigd               Running

   
[zimbra@zimbra-mailbox ~]$ zmcontrol status
Host zimbra-mailbox.example.com
   amavis                  Running
   antispam                Running
   antivirus               Running
   dnscache                Running
   logger                  Running
   mailbox                 Running
   memcached               Running
   mta                     Running
   opendkim                Running
   proxy                   Running
   service webapp          Running
   snmp                    Running
   spell                   Running
   stats                   Running
   zimbra webapp           Running
   zimbraAdmin webapp      Running
   zimlet webapp           Running
   zmconfigd               Running

   
[zimbra@zimbra-mailbox2 ~]$ zmcontrol status
Host zimbra-mailbox2.example.com
   amavis                  Running
   antispam                Running
   antivirus               Running
   dnscache                Running
   mailbox                 Running
   memcached               Running
   mta                     Running
   opendkim                Running
   proxy                   Running
   service webapp          Running
   snmp                    Running
   spell                   Running
   stats                   Running
   zimbra webapp           Running
   zimbraAdmin webapp      Running
   zimlet webapp           Running
   zmconfigd               Running


the installation order is as below:
    1st: zimbra-ldap
    2nd: zimbra-mailbox
    3rd: zimbra-mailbox2

when I startup all service on zimbra-mailbox and shutdown all service on zimbra-mailbox2, I cannot login https://zimbra-mailbox
nginx.log have the following error msg:

Code: Select all

2017/04/20 01:43:58 [error] 28822#0: recv() failed (111: Connection refused)
2017/04/20 01:43:58 [error] 28822#0: reconnect to memcached channel 192.168.56.102:11211 fails
2017/04/20 01:43:58 [notice] 28822#0: memcached channel:192.168.56.102:11211 down, reconnect after:60000 ms
2017/04/20 01:43:58 [error] 28821#0: recv() failed (111: Connection refused)



the output of zmprov getAllMemcachedServers shows there 2 memcached server exist:

Code: Select all

[zimbra@zimbra-mailbox ~]$ zmprov getAllMemcachedServers
zimbra-mailbox.example.com:11211 zimbra-mailbox2.example.com:11211


and I checked memcached on zimbra-mailbox is in running mode.
It seems nginx didn't fail-over memcached connection to zimbra-mailbox.

Did you ever encounter this problem or Can you help me figure it out why?

Re: zimbra proxy don't fail-over memecached

Posted: Thu Apr 20, 2017 10:23 am
by DualBoot
It says connection refused, are you running a FireWall ?

Re: zimbra proxy don't fail-over memecached

Posted: Thu Apr 20, 2017 12:10 pm
by maxyang2008
DualBoot wrote:It says connection refused, are you running a FireWall ?

no, there is no firewall running.
It says connetion refused because I stop memcached servivce on node zimbra-mailbox2, but I have 2 memcached server configured, proxy doesn't look for the live memcached on zimbra-mailbox.

Re: zimbra proxy don't fail-over memecached

Posted: Thu Apr 20, 2017 1:00 pm
by DualBoot
Sorry for misunderstanding.
Can you explain what you try to aim to ?

Re: zimbra proxy don't fail-over memecached

Posted: Thu Apr 20, 2017 1:12 pm
by maxyang2008
DualBoot wrote:Sorry for misunderstanding.
Can you explain what you try to aim to ?

Suppose I have 2 zimbra proxy server, each have its own memcached service, I will put a load-balancer in front of the 2 proxy server, when either 1 of the 2 proxy goes down, the whole system can still operate.

The problem now is that proxy server 1 always want to connect memcached on proxy server 2, when I shutdown proxy server 2, the whole system cannot operate.

Re: zimbra proxy don't fail-over memecached

Posted: Thu Apr 20, 2017 1:55 pm
by L. Mark Stone
I think you'll find that this behavior appears in at least one other place too: when a mailbox server is down in a proxy environment, the upstream server list (e.g. "zmprov garpu") does not get updated.

So in a 5-mailbox server environment with a separate proxy server in front, if one of the mailbox servers is down, 1/5 of the users browsing to the proxy URL to try to login to Zimbra will be greeted with a failure, as Zimbra proxy tries to get the login screen from the next random mailbox server, and chooses the one that is offline.

There are a few bugs for this, to enable Zimbra to update the upstream mailbox server list, but in the interim I would recommend the following.

First, have separate proxy/memcached servers and separate mailbox servers. This will make upgrades much easier as you can avoid the downtime from upgrading a mailbox server by just deploying a new mailbox server and moving all of the mailboxes there, then run zmdedupe and then destroy the old mailbox server. Virtually zero downtime.

Second, if you put a load balancer in front of the pair of proxy servers, you can do maintenance on each proxy server also with no downtime perceived by the end users.

Hope that helps,
Mark