Architecture - running Centos 6.4 on all machines
One machine - (zimbra1) used for ldap and logging (running snmp, ldap, proxy, logger, apache, mta, spell, memcached, store and core) no public ip addresses
Two machines - (mx1 and mx2) mx servers (running mta and core only)
One machine - (mail4) used for proxy only from the Internet (mta, memcached, core, store, spell, proxy, apache)
One machine - (mailstore2) used for mail store - all ,mailboxes are on it - has no public ip addresses (store, core, spell, mta, memcached, proxy, apache)
Senerio: two weeks ago Mailstore1 (now turned off) had a dying drive in the raid array. Size of drives, memory, processor, etc, we decided to upgrade to a new machine. Set up new machine, and over the weekend moved mailboxes from Mailstore1 to Mailstore2 (using a script I have used multiple times in the past). Monday morning - everything is running good. Friday - we remove Mailstore1 and turn it off (zmprov deleteServer Mailstore1.domain.com) About 30 minutes later, we have three accounts out of 263 that cannot POP their email. The web interface works fine. Just cannot use POP3.
The following line appears in the nginx.log
2013/11/18 08:20:10 [error] 18520#0: *71 recv() failed (111: Connection refused) while reading response from upstream, client: 172.17.2.121, server: 0.0.0.0:110, login: "email@example.com", upstream: 172.17.2.48:7110 (172.17.2.121:44025-172.17.2.21:110) <=> (172.17.2.21:51570-)
(the above are actual ip addresses)
After many hours of pulling hair out the two following items were discovered:
1. Adding an alias for each of those three users (example: firstname.lastname@example.org) and using that as the username for POP3, they could log in
2. Modifying their email client to bypass the proxy server by using port 7110 and their email address (not the alias) they could log in (obviously this will not work via the Internet as the mailstore has no public ip addresses)
I have done a
/opt/zimbra/libexec/zmslapcat -c .
and studied the results and cannot see anything different about these accounts.
Can anyone suggest some other place to look?
Does the proxy server have a cache we can flush?