Possible Bug - nginx proxy not seeing memcache when DoS workaround applied

Ask questions about your setup or get help installing ZCS server (ZD section below).
guitardood
Posts: 19
Joined: Sat Sep 13, 2014 3:30 am

Possible Bug - nginx proxy not seeing memcache when DoS workaround applied

Postby guitardood » Thu Jan 24, 2019 11:45 am

Hello All,
I just found a problem while troubleshooting another issue. For some reason, in the nginx.log file, I found it littered with the error "no memcache server available". I remembered modifying the memcached IP address to 127.0.0.1 to mitigate DoS against memcached (per https://wiki.zimbra.com/wiki/Blocking_Memcached_Attack). While looking through the config files for nginx, I found the line containing the "servers" entry still had the zmhostname's IP address rather than the IP from the zimbraMemcachedBindAddress setting.

This is on ZCS Version - Release 8.8.10.GA.3039.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.10_P5.

As a temporary workaround, I found the servers entry in the template (/opt/zimbra/conf/nginx/templates/nginx.conf.memcache.template), "${memcache.:servers}", and commented it out. I then added a new line with the entry "servers 127.0.0.1:11211;".

The workaround works, but I thought I'd mention it here for anyone else who might be experiencing this issue.

Best,
Guitardood


smclinden
Posts: 19
Joined: Mon Aug 28, 2017 7:54 pm

Re: Possible Bug - nginx proxy not seeing memcache when DoS workaround applied

Postby smclinden » Fri Feb 22, 2019 3:16 pm

Can somebody please address a fix for this.

Making the changes in https://wiki.zimbra.com/wiki/Blocking_Memcached_Attack does NOT appear to affect the value as defined in

/opt/zimbra/conf/nginx/templates/nginx.conf.memcache.template


specifically:

Code: Select all

${memcache.:servers}


The suggested "workaround", to hard code the localhost in the template is not acceptable and potentially not persistent across updates.
smclinden
Posts: 19
Joined: Mon Aug 28, 2017 7:54 pm

Re: Possible Bug - nginx proxy not seeing memcache when DoS workaround applied

Postby smclinden » Mon Feb 25, 2019 5:27 pm

So, apparently, making the changes as described in the above link DOES NOT change the value which is returned by:

Code: Select all


$zmprov gs server.domain.tld zimbaMemcachedBindAddress
# name server.domain.tld
zimbraMemcachedBindAddress: 127.0.0.1

$ zmprov gamcs
server.domain.tld:11211



This strikes me as being a bug. What value is there to setting zimbraMemcachedBindAddress if it is ignored when nginx is configured?
guitardood
Posts: 19
Joined: Sat Sep 13, 2014 3:30 am

Re: Possible Bug - nginx proxy not seeing memcache when DoS workaround applied

Postby guitardood » Mon Feb 25, 2019 5:49 pm

smclinden wrote:This strikes me as being a bug.

It is.

smclinden wrote:What value is there to setting zimbraMemcachedBindAddress if it is ignored when nginx is configured?

It does set the bind address on which memcached runs, just doesn't effect the memcached address which nginx uses.

I'm not familiar enough with the innards of Zimbra to know if memcached is used for other purposes besides nginx, I just happened to find this one while troubleshooting another issue.

The workaround I posted works, but definitely should only be a temporary workaround until this bug gets resolved.

Best,
guitardood
smclinden
Posts: 19
Joined: Mon Aug 28, 2017 7:54 pm

Re: Possible Bug - nginx proxy not seeing memcache when DoS workaround applied

Postby smclinden » Mon Feb 25, 2019 6:08 pm

Another workaround (as someone posted) would be to enable the memcached service on the public IP, then use IP tables to block access from other than the local host.

To me this is, at best, a less efficient solution than simply not listening, but it would work with the existing distribution.

But, in general, I favor not listening on any IPs that are not necessary for the proper operation of the system (slapd comes to mind).

Sean
guitardood
Posts: 19
Joined: Sat Sep 13, 2014 3:30 am

Re: Possible Bug - nginx proxy not seeing memcache when DoS workaround applied

Postby guitardood » Mon Feb 25, 2019 6:16 pm

That would definitely work, as would adding a deny rule on a gateway server, if installed. Though, on a busy Zimbra server, adding iptable filtering could be the straw to break the camel's back. Once turned on, whether the rule is for that connection or not, it will have to check every packet going in/out of the server. Seems it would be a bit inefficient.

Best,
guitardood

Return to “Installation and Upgrade”

Who is online

Users browsing this forum: No registered users and 9 guests