stats and logger services restart often

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Chris-M
Posts: 22
Joined: Thu Jul 30, 2015 2:24 pm

stats and logger services restart often

Postby Chris-M » Wed Mar 30, 2016 4:04 pm

On our ZCS 8.6.0 install the stats and logger services restart at least once a day. In the logs we see this:

Code: Select all

Mar 30 07:16:19 mail zmconfigd[23752]: Service status change: mail.domain.com stats changed from stopped to running
Mar 30 07:17:21 mail zmconfigd[23752]: Service status change: mail.domain.com stats changed from running to stopped
Mar 29 03:17:21 mail zmconfigd[23752]: Service status change: mail.domain.com logger changed from running to stopped
Mar 29 03:18:22 mail zmconfigd[23752]: Service status change: mail.domain.com logger changed from stopped to running
Mar 28 03:25:20 mail zmconfigd[31300]: Service status change: mail.domain.com logger changed from running to stopped
Mar 28 03:26:21 mail zmconfigd[31300]: Service status change: mail.domain.com logger changed from stopped to running
Mar 28 23:07:42 mail zmconfigd[23752]: Service status change: mail.domain.com stats changed from stopped to running
Mar 28 23:08:43 mail zmconfigd[23752]: Service status change: mail.domain.com stats changed from running to stopped


Currently the stats service is showing as stopped:

Code: Select all

[zimbra@mail log]$ zmcontrol status
Host mail.domain.com
        amavis                  Running
        antispam                Running
        antivirus               Running
        ldap                    Running
        logger                  Running
        mailbox                 Running
        mta                     Running
        opendkim                Running
        service webapp          Running
        snmp                    Running
        spell                   Running
        stats                   Stopped
        zimbra webapp           Running
        zimbraAdmin webapp      Running
        zimlet webapp           Running
        zmconfigd               Running


I have searched quite a bit but haven't found any relevant information for this issue affecting Zimbra 8.6 (only much older versions) so i'm not sure where to begin troubleshooting.

Can anyone suggest how to troubleshoot this? The Zimbra server itself is not a busy server and only serves a couple of users - the loads are very low.

Thanks in advance.


User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1655
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Re: stats and logger services restart often

Postby quanah » Wed Mar 30, 2016 6:00 pm

The logger restarts are clearly from nightly log rotation. The stats bit, what is the output of zmstatctl status?
--
Quanah Gibson-Mount https://github.com/quanah/
Systems Architect, Synacor http://wwwsynacor.com/
A Zetalliance Founder http://www.zetalliance.org/
OpenLDAP Core team http://www.openldap.org/project/
Chris-M
Posts: 22
Joined: Thu Jul 30, 2015 2:24 pm

Re: stats and logger services restart often

Postby Chris-M » Wed Mar 30, 2016 7:41 pm

Thanks for your response.

The output is as follows:

Code: Select all

[zimbra@mail /]$ zmstatctl status
Running: zmstat-df
Running: zmstat-mtaqueue
Running: zmstat-io
Running: zmstat-ldap
Running: zmstat-proc
Running: zmstat-mysql
Running: zmstat-cpu
Running: zmstat-vm
Running: zmstat-io-x
process 26304 in /opt/zimbra/zmstat/pid/zmstat-fd.pid not running
Running: zmstat-allprocs


Any ideas?
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1655
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Re: stats and logger services restart often

Postby quanah » Thu Mar 31, 2016 12:42 am

For a first pass, I'd probably do something like

Code: Select all

zmstatctl stop
rm -f /opt/zimbra/zmstat/pid/zmstat-fd.pid
zmstatctl start
zmstatctl status


And see if that resolves it.
--
Quanah Gibson-Mount https://github.com/quanah/
Systems Architect, Synacor http://wwwsynacor.com/
A Zetalliance Founder http://www.zetalliance.org/
OpenLDAP Core team http://www.openldap.org/project/
Chris-M
Posts: 22
Joined: Thu Jul 30, 2015 2:24 pm

Re: stats and logger services restart often

Postby Chris-M » Thu Mar 31, 2016 11:12 am

Thanks for your reply.

I just did as you suggested and it seems the stats process is still not running after this:

Code: Select all

[zimbra@mail /]$ zmstatctl stop
Terminating process 26283
Terminating process 26325
Terminating process 26297
Terminating process 26332
Terminating process 26275
Terminating process 26313
Terminating process 26277
Terminating process 26279
Terminating process 26281
Terminating process 26304
Terminating process 26307

[zimbra@mail /]$ rm -f /opt/zimbra/zmstat/pid/zmstat-fd.pid

[zimbra@mail /]$ zmstatctl start
Invoking: /opt/zimbra/libexec/zmstat-proc
Invoking: /opt/zimbra/libexec/zmstat-cpu
Invoking: /opt/zimbra/libexec/zmstat-vm
Invoking: /opt/zimbra/libexec/zmstat-io -x
Invoking: /opt/zimbra/libexec/zmstat-df
Invoking: /opt/zimbra/libexec/zmstat-io
Invoking: /opt/zimbra/libexec/zmstat-fd
Invoking: /opt/zimbra/libexec/zmstat-allprocs
Invoking: /opt/zimbra/libexec/zmstat-mysql
Invoking: /opt/zimbra/libexec/zmstat-mtaqueue
Invoking: /opt/zimbra/libexec/zmstat-ldap

[zimbra@mail /]$ zmstatctl status
Running: zmstat-df
Running: zmstat-mtaqueue
Running: zmstat-io
Running: zmstat-ldap
Running: zmstat-proc
Running: zmstat-mysql
Running: zmstat-cpu
Running: zmstat-vm
Running: zmstat-io-x
process 20564 in /opt/zimbra/zmstat/pid/zmstat-fd.pid not running
Running: zmstat-allprocs
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1655
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Re: stats and logger services restart often

Postby quanah » Thu Mar 31, 2016 4:54 pm

Ok, next I would run:

Code: Select all

/opt/zimbra/libexec/zmstat-fd


and see if it reports any errors.
--
Quanah Gibson-Mount https://github.com/quanah/
Systems Architect, Synacor http://wwwsynacor.com/
A Zetalliance Founder http://www.zetalliance.org/
OpenLDAP Core team http://www.openldap.org/project/
Chris-M
Posts: 22
Joined: Thu Jul 30, 2015 2:24 pm

Re: stats and logger services restart often

Postby Chris-M » Fri Apr 01, 2016 12:41 pm

Output from that command is:

Code: Select all

[zimbra@mail /]$ /opt/zimbra/libexec/zmstat-fd
zmstat-fd-real.pid: Already running as pid 7015
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1655
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Re: stats and logger services restart often

Postby quanah » Fri Apr 01, 2016 6:20 pm

Ok, if you're seeing that after running

Code: Select all

zmstatctl stop
it seems like there's either (a) a runaway process for the zmstat-fd bits, or (b) a stale pid for the zmstat-fd root portion of the process. So I'd check first to see if that process ID exists, AND that it is related to zmstat-fd. If so, then kill it, and clean up the pid file. It's possible that some other process got that pid on a reboot and it's causing the script to exit out incorrectly. If there is no process with that PID, or the PID with that process is unrelated, remove the on disk file tracking the zmstat-fd root PID, so it'll stop getting confused, and then run

Code: Select all

zmstatctl start
and see if things are happier.
--
Quanah Gibson-Mount https://github.com/quanah/
Systems Architect, Synacor http://wwwsynacor.com/
A Zetalliance Founder http://www.zetalliance.org/
OpenLDAP Core team http://www.openldap.org/project/
Chris-M
Posts: 22
Joined: Thu Jul 30, 2015 2:24 pm

Re: stats and logger services restart often

Postby Chris-M » Fri Apr 01, 2016 10:07 pm

Thanks so much for your replies and support.

I killed the process and then started zmstatctl and all seems good now:

Code: Select all

[root@mail /]# ps aux | grep 7015
root      7015  0.0  0.0  48816  6324 ?        S    Mar13   0:23 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd

[root@mail /]# kill -9 7015

[zimbra@mail /]$ zmstatctl start
zmstat-df already running, skipping.
zmstat-mtaqueue already running, skipping.
zmstat-io already running, skipping.
zmstat-ldap already running, skipping.
zmstat-proc already running, skipping.
zmstat-mysql already running, skipping.
zmstat-cpu already running, skipping.
zmstat-vm already running, skipping.
zmstat-io-x already running, skipping.
zmstat-allprocs already running, skipping.
Invoking: /opt/zimbra/libexec/zmstat-fd

[zimbra@mail /]$ /opt/zimbra/libexec/zmstat-fd
zmstat-fd.pid: Already running as pid 13814

[zimbra@mail /]$ zmstatctl status
Running: zmstat-df
Running: zmstat-mtaqueue
Running: zmstat-io
Running: zmstat-ldap
Running: zmstat-proc
Running: zmstat-mysql
Running: zmstat-cpu
Running: zmstat-vm
Running: zmstat-io-x
Running: zmstat-fd
Running: zmstat-allprocs


I will monitor for the next few days to see if the service stops/starts unexpectedly.

Once again, thank you!
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1655
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Re: stats and logger services restart often

Postby quanah » Sat Apr 02, 2016 12:23 am

Yay, I'm glad it's finally solved!
--
Quanah Gibson-Mount https://github.com/quanah/
Systems Architect, Synacor http://wwwsynacor.com/
A Zetalliance Founder http://www.zetalliance.org/
OpenLDAP Core team http://www.openldap.org/project/

Return to “Administrators”

Who is online

Users browsing this forum: Baidu [Spider] and 18 guests