zimbra proxy don't fail-over memecached

Ask questions about your setup or get help installing ZCS server (ZD section below).
maxyang2008
Posts: 3
Joined: Sat Apr 15, 2017 12:03 pm

zimbra proxy don't fail-over memecached

Postby maxyang2008 » Thu Apr 20, 2017 7:53 am

Hi buddies,

I am testing zimbra version zcs-8.7.5_GA_1764
I have a 3 node zimbra setup like below:
    ldap: zimbra-ldap
    mailbox&proxy server 1 : zimbra-mailbox
    mailbox&proxy server 2 : zimbra-mailbox2

services running on each node are as below:

Code: Select all

[zimbra@zimbra-ldap ~]$ zmcontrol status
Host zimbra-ldap.example.com
   ldap                    Running
   stats                   Running
   zmconfigd               Running

   
[zimbra@zimbra-mailbox ~]$ zmcontrol status
Host zimbra-mailbox.example.com
   amavis                  Running
   antispam                Running
   antivirus               Running
   dnscache                Running
   logger                  Running
   mailbox                 Running
   memcached               Running
   mta                     Running
   opendkim                Running
   proxy                   Running
   service webapp          Running
   snmp                    Running
   spell                   Running
   stats                   Running
   zimbra webapp           Running
   zimbraAdmin webapp      Running
   zimlet webapp           Running
   zmconfigd               Running

   
[zimbra@zimbra-mailbox2 ~]$ zmcontrol status
Host zimbra-mailbox2.example.com
   amavis                  Running
   antispam                Running
   antivirus               Running
   dnscache                Running
   mailbox                 Running
   memcached               Running
   mta                     Running
   opendkim                Running
   proxy                   Running
   service webapp          Running
   snmp                    Running
   spell                   Running
   stats                   Running
   zimbra webapp           Running
   zimbraAdmin webapp      Running
   zimlet webapp           Running
   zmconfigd               Running


the installation order is as below:
    1st: zimbra-ldap
    2nd: zimbra-mailbox
    3rd: zimbra-mailbox2

when I startup all service on zimbra-mailbox and shutdown all service on zimbra-mailbox2, I cannot login https://zimbra-mailbox
nginx.log have the following error msg:

Code: Select all

2017/04/20 01:43:58 [error] 28822#0: recv() failed (111: Connection refused)
2017/04/20 01:43:58 [error] 28822#0: reconnect to memcached channel 192.168.56.102:11211 fails
2017/04/20 01:43:58 [notice] 28822#0: memcached channel:192.168.56.102:11211 down, reconnect after:60000 ms
2017/04/20 01:43:58 [error] 28821#0: recv() failed (111: Connection refused)



the output of zmprov getAllMemcachedServers shows there 2 memcached server exist:

Code: Select all

[zimbra@zimbra-mailbox ~]$ zmprov getAllMemcachedServers
zimbra-mailbox.example.com:11211 zimbra-mailbox2.example.com:11211


and I checked memcached on zimbra-mailbox is in running mode.
It seems nginx didn't fail-over memcached connection to zimbra-mailbox.

Did you ever encounter this problem or Can you help me figure it out why?


User avatar
DualBoot
Outstanding Member
Outstanding Member
Posts: 494
Joined: Mon Apr 18, 2016 8:18 pm
Location: Earth
ZCS/ZD Version: ZCS FLOSS - 8.7.7 Mutli servers

Re: zimbra proxy don't fail-over memecached

Postby DualBoot » Thu Apr 20, 2017 10:23 am

It says connection refused, are you running a FireWall ?
maxyang2008
Posts: 3
Joined: Sat Apr 15, 2017 12:03 pm

Re: zimbra proxy don't fail-over memecached

Postby maxyang2008 » Thu Apr 20, 2017 12:10 pm

DualBoot wrote:It says connection refused, are you running a FireWall ?

no, there is no firewall running.
It says connetion refused because I stop memcached servivce on node zimbra-mailbox2, but I have 2 memcached server configured, proxy doesn't look for the live memcached on zimbra-mailbox.
User avatar
DualBoot
Outstanding Member
Outstanding Member
Posts: 494
Joined: Mon Apr 18, 2016 8:18 pm
Location: Earth
ZCS/ZD Version: ZCS FLOSS - 8.7.7 Mutli servers

Re: zimbra proxy don't fail-over memecached

Postby DualBoot » Thu Apr 20, 2017 1:00 pm

Sorry for misunderstanding.
Can you explain what you try to aim to ?
maxyang2008
Posts: 3
Joined: Sat Apr 15, 2017 12:03 pm

Re: zimbra proxy don't fail-over memecached

Postby maxyang2008 » Thu Apr 20, 2017 1:12 pm

DualBoot wrote:Sorry for misunderstanding.
Can you explain what you try to aim to ?

Suppose I have 2 zimbra proxy server, each have its own memcached service, I will put a load-balancer in front of the 2 proxy server, when either 1 of the 2 proxy goes down, the whole system can still operate.

The problem now is that proxy server 1 always want to connect memcached on proxy server 2, when I shutdown proxy server 2, the whole system cannot operate.
User avatar
L. Mark Stone
Elite member
Elite member
Posts: 1497
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine
ZCS/ZD Version: 8.6.0 Patch 8
Contact:

Re: zimbra proxy don't fail-over memecached

Postby L. Mark Stone » Thu Apr 20, 2017 1:55 pm

I think you'll find that this behavior appears in at least one other place too: when a mailbox server is down in a proxy environment, the upstream server list (e.g. "zmprov garpu") does not get updated.

So in a 5-mailbox server environment with a separate proxy server in front, if one of the mailbox servers is down, 1/5 of the users browsing to the proxy URL to try to login to Zimbra will be greeted with a failure, as Zimbra proxy tries to get the login screen from the next random mailbox server, and chooses the one that is offline.

There are a few bugs for this, to enable Zimbra to update the upstream mailbox server list, but in the interim I would recommend the following.

First, have separate proxy/memcached servers and separate mailbox servers. This will make upgrades much easier as you can avoid the downtime from upgrading a mailbox server by just deploying a new mailbox server and moving all of the mailboxes there, then run zmdedupe and then destroy the old mailbox server. Virtually zero downtime.

Second, if you put a load balancer in front of the pair of proxy servers, you can do maintenance on each proxy server also with no downtime perceived by the end users.

Hope that helps,
Mark
________________________________________________
L. Mark Stone, General Manager
reliable networks, a Division of OTT Communications
HIPAA-Compliant Zimbra Hosting Provider since 2006 http://www.reliablenetworks.com
Zeta Alliancehttp://www.zetalliance.org/

Return to “Installation and Upgrade”

Who is online

Users browsing this forum: No registered users and 9 guests