Stopping zimlet webapp... takes 10-15 minutes

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
PaperAdvocate
Posts: 7
Joined: Tue Oct 11, 2016 9:28 pm

Stopping zimlet webapp... takes 10-15 minutes

Postby PaperAdvocate » Wed Dec 06, 2017 1:07 am

When shutting Zimbra down either via a system reboot or "zmcontrol stop", the process hangs at "stopping zimlet webapp" for 10-15 minutes. If I close all firewall ports (except SSH) and restart Zimbra it will take the same 10-15 minutes to shutdown, however on all following attempts it takes only 1 minute. This shows to me that it's established connections which are related to the issue somehow.

A netstat shows 20-30 IMAPS connections before the firewall is closed and Zimbra is restarted, afterward it shows only the LDAP and other internal connections, and SSH. The logs show E-Mail being received as well so SMTP connections were obviously active.

Single server Zimbra install, version is 8.7.11 FOSS on CentOS 7x64 with Firewalld and Selinux enabled.

The only thing that I think points to something in the logs are these (taken when firewalld allows traffic):

Dec 4 20:56:37 mail zmmailboxdmgr[9641]: stop requested
Dec 4 21:07:14 mail amavis-mc[26756]: Process [26757] exited (/opt/zimbra/common/sbin/amavis-services msg-forwarder) after 332559.5 s: KILLED, signal 9 (0009)

Dec 4 21:29:49 mail zmmailboxdmgr[27589]: stop requested
Dec 4 21:40:28 mail amavis-mc[23759]: Process [23760] exited (/opt/zimbra/common/sbin/amavis-services msg-forwarder) after 867.0 s: KILLED, signal 9 (0009)

All other attempts to shutdown with the firewall blocking traffic takes about 1 minute.

Any ideas on other items I might try to narrow this down further?

Thank you,

Jacob


cpaul_carling
Posts: 33
Joined: Sat Sep 13, 2014 1:36 am

Re: Stopping zimlet webapp... takes 10-15 minutes

Postby cpaul_carling » Wed Dec 06, 2017 3:58 am

I am having the same issue, on 8.7.9. Reached out to support, but no solution so far.
PaperAdvocate
Posts: 7
Joined: Tue Oct 11, 2016 9:28 pm

Re: Stopping zimlet webapp... takes 10-15 minutes

Postby PaperAdvocate » Wed Apr 18, 2018 6:03 am

Still trying to resolve this issue. I've compared logs of a server with same host OS and Zimbra version.

Release 8.7.11_GA_1854.RHEL7_64_20170531151956 RHEL7_64 FOSS edition.

The difference found was from Zimbra.log
Server with delay:

Code: Select all

Apr  3 23:37:24 mail zmconfigd[24046]: Shutting down. Received signal 15
Apr  3 23:37:25 mail zimbramon[14031]: 14031:info: Stopping zimlet via zmcontrol
Apr  3 23:37:25 mail zmmailboxdmgr[14122]: stop requested
Apr  3 23:37:25 mail zmmailboxdmgr[14122]: waiting for manager process 26108 to die
Apr  3 23:37:25 mail zmmailboxdmgr[26108]: shutdown requested, sending TERM signal to 26109
Apr  3 23:37:26 mail zmmailboxdmgr[26108]: mailboxd/JVM process exited (waitpid expected 26109 got 26109)
Apr  3 23:37:26 mail zmmailboxdmgr[26108]: manager woke up from wait on mailboxd/JVM with pid 26109
Apr  3 23:37:27 mail zmmailboxdmgr[14122]: manager process 26108 died, shutdown completed
Apr  3 23:37:49 mail postfix/amavisd/smtpd[11477]: timeout after END-OF-MESSAGE from localhost[127.0.0.1]
Apr  3 23:37:49 mail postfix/amavisd/smtpd[11477]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 commands=4
...
Apr  3 23:47:44 mail zimbramon[14031]: 14031:info: Stopping zimbraAdmin via zmcontrol


Server without delay:

Code: Select all

Apr 17 21:18:54 mail zimbramon[491]: 491:info: Stopping zimlet via zmcontrol
Apr 17 21:18:55 mail slapd[20055]: slap_queue_csn: queueing 0xf6c2e40 20180418041855.422478Z#000000#000#000000
Apr 17 21:18:55 mail slapd[20055]: slap_graduate_commit_csn: removing 0xf6c2e40 20180418041855.422478Z#000000#000#000000
Apr 17 21:18:55 mail zmmailboxdmgr[592]: stop requested
Apr 17 21:18:55 mail zmmailboxdmgr[592]: waiting for manager process 21758 to die
Apr 17 21:18:55 mail zmmailboxdmgr[21758]: shutdown requested, sending TERM signal to 21759
Apr 17 21:18:57 mail zmmailboxdmgr[21758]: mailboxd/JVM process exited (waitpid expected 21759 got 21759)
Apr 17 21:18:57 mail zmmailboxdmgr[21758]: manager woke up from wait on mailboxd/JVM with pid 21759
Apr 17 21:18:58 mail zmmailboxdmgr[592]: manager process 21758 died, shutdown completed
Apr 17 21:19:02 mail zimbramon[491]: 491:info: Stopping zimbraAdmin via zmcontrol


Notice on the server with the delay it takes 10 minutes between "stopping zimlet" and "stopping zimbraAdmin" where the other takes 8 seconds.

Both servers are on CentOS7, same patch level and running same underlying hardware, similar # of users. The server with the delay has been updated from 6.x versions where the other started at 8.5+.

On the server with a delay the 2 extra lines that have me wondering are:

Code: Select all

Apr  3 23:37:49 mail postfix/amavisd/smtpd[11477]: timeout after END-OF-MESSAGE from localhost[127.0.0.1]
Apr  3 23:37:49 mail postfix/amavisd/smtpd[11477]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 commands=4


Any ideas on what I can do next to try to drill down further?
zulhairiseman
Posts: 2
Joined: Wed May 24, 2017 8:34 am

Re: Stopping zimlet webapp... takes 10-15 minutes

Postby zulhairiseman » Tue Apr 24, 2018 10:06 am

Same issue and still not find any solution. :(
User avatar
L. Mark Stone
Elite member
Elite member
Posts: 1738
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine
ZCS/ZD Version: 8.8.8 Patch 3 Network Edition
Contact:

Re: Stopping zimlet webapp... takes 10-15 minutes

Postby L. Mark Stone » Tue Apr 24, 2018 2:09 pm

PaperAdvocate wrote:Still trying to resolve this issue. I've compared logs of a server with same host OS and Zimbra version.

Release 8.7.11_GA_1854.RHEL7_64_20170531151956 RHEL7_64 FOSS edition.

The difference found was from Zimbra.log
Server with delay:

Code: Select all

Apr  3 23:37:24 mail zmconfigd[24046]: Shutting down. Received signal 15
Apr  3 23:37:25 mail zimbramon[14031]: 14031:info: Stopping zimlet via zmcontrol
Apr  3 23:37:25 mail zmmailboxdmgr[14122]: stop requested
Apr  3 23:37:25 mail zmmailboxdmgr[14122]: waiting for manager process 26108 to die
Apr  3 23:37:25 mail zmmailboxdmgr[26108]: shutdown requested, sending TERM signal to 26109
Apr  3 23:37:26 mail zmmailboxdmgr[26108]: mailboxd/JVM process exited (waitpid expected 26109 got 26109)
Apr  3 23:37:26 mail zmmailboxdmgr[26108]: manager woke up from wait on mailboxd/JVM with pid 26109
Apr  3 23:37:27 mail zmmailboxdmgr[14122]: manager process 26108 died, shutdown completed
Apr  3 23:37:49 mail postfix/amavisd/smtpd[11477]: timeout after END-OF-MESSAGE from localhost[127.0.0.1]
Apr  3 23:37:49 mail postfix/amavisd/smtpd[11477]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 commands=4
...
Apr  3 23:47:44 mail zimbramon[14031]: 14031:info: Stopping zimbraAdmin via zmcontrol


Server without delay:

Code: Select all

Apr 17 21:18:54 mail zimbramon[491]: 491:info: Stopping zimlet via zmcontrol
Apr 17 21:18:55 mail slapd[20055]: slap_queue_csn: queueing 0xf6c2e40 20180418041855.422478Z#000000#000#000000
Apr 17 21:18:55 mail slapd[20055]: slap_graduate_commit_csn: removing 0xf6c2e40 20180418041855.422478Z#000000#000#000000
Apr 17 21:18:55 mail zmmailboxdmgr[592]: stop requested
Apr 17 21:18:55 mail zmmailboxdmgr[592]: waiting for manager process 21758 to die
Apr 17 21:18:55 mail zmmailboxdmgr[21758]: shutdown requested, sending TERM signal to 21759
Apr 17 21:18:57 mail zmmailboxdmgr[21758]: mailboxd/JVM process exited (waitpid expected 21759 got 21759)
Apr 17 21:18:57 mail zmmailboxdmgr[21758]: manager woke up from wait on mailboxd/JVM with pid 21759
Apr 17 21:18:58 mail zmmailboxdmgr[592]: manager process 21758 died, shutdown completed
Apr 17 21:19:02 mail zimbramon[491]: 491:info: Stopping zimbraAdmin via zmcontrol


Notice on the server with the delay it takes 10 minutes between "stopping zimlet" and "stopping zimbraAdmin" where the other takes 8 seconds.

Both servers are on CentOS7, same patch level and running same underlying hardware, similar # of users. The server with the delay has been updated from 6.x versions where the other started at 8.5+.

On the server with a delay the 2 extra lines that have me wondering are:

Code: Select all

Apr  3 23:37:49 mail postfix/amavisd/smtpd[11477]: timeout after END-OF-MESSAGE from localhost[127.0.0.1]
Apr  3 23:37:49 mail postfix/amavisd/smtpd[11477]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 commands=4


Any ideas on what I can do next to try to drill down further?


It's not clear to me from your posts, but are you saying that the Zimbra server which is running firewalld has the delay, and the server which is not running firewalld does not have the shutdown delay?

FWIW Zimbra typically suggests not running a local firewall on the Zimbra servers themselves, but instead to run a firewall in front of the Zimbra server(s).

Hope that helps,
Mark
_____________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP and Consulting https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 13 guests