Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: 5.011 Bug - zmlogger fails to stop

  1. #1
    Join Date
    Jul 2007
    Location
    San Jose, CA
    Posts
    1,027
    Rep Power
    10

    Exclamation 5.011 Bug - zmlogger fails to stop

    Hey everyone,

    I'm putting this out there because it could drive a bunch of you nuts trying to debug something we've already discovered. . .on 5.011, when you do a zmcontrol stop, the zmlogger process doesn't stop. Each stop and restart (and for those who use open source backups, that's once a night) will leave you with a new "ghost" instance of zmlogger still running on your system, as you can see by doing a
    Code:
    ps ax | grep zmlogger
    You can safely kill those extra processes (although I'd recommend doing it with zimbra services down unless you are absolutely sure which are the ghosts and which is the real thing).

    However, if you fail to kill them, the number of running processes builds up until one day you try to log onto the admin gui and you get this familiar error:
    Message: system failure: getting database logger connection Error code: service.FAILURE Method: GetServiceStatusRequest Details:soap:Receiver
    NOTE, this is not the only cause of this error, but it is the one I saw today. Killing off excess zmlogger processes will eliminate the error (as will a reboot of the system if you prefer the Microsoft way )

    I have submitted Bug 33730 on this issue and will let you know more if I hear/discover it.
    Cheers,

    Dan

  2. #2
    Join Date
    Aug 2008
    Posts
    14
    Rep Power
    7

    Default

    Same error with me.
    5.0.11 Network edition.

    I've also noticed this:

    logmysql
    ERROR 1040 (08004): Too many connections


    Paolo
    Last edited by Paolo; 12-05-2008 at 01:17 AM.

  3. #3
    Join Date
    Aug 2008
    Posts
    14
    Rep Power
    7

    Default

    A possible workaround for now could be to edit this file /opt/zimbra/conf/my.logger.cnf
    and change from 10 to more this line:
    max_connections = 10

    Paolo

  4. #4
    Join Date
    Nov 2007
    Location
    Wilmore, KY
    Posts
    28
    Rep Power
    8

    Default Here's the fix

    I added the follow to the bug report here Bug 33730 Updated

    The 5.0.10 version of /opt/zimbra/bin/zmlogswatchctl "stop" function uses "kill -TERM" which correctly shuts down the logswatch process and it's child .swatch_script.??? process.

    The 5.0.11 version of /opt/zimbra/bin/zmlogswatchctl "stop" function uses "kill -9" which shuts down the logswatch process but does NOT shut down it's child .swatch_script.??? process.

    This change incorrectly allows the .swatch_script.??? processes to remain active which keeps corresponding zmlogger processes active which take up logger's mysql server connections until there are no more connections to be spared.

    When no more connections are available errors can show up such as the following error seen when logging into Zimbra's admin web interface:

    Server error encountered Message: system failure: getting database logger connection Error code: service.FAILURE Method: GetServiceStatusRequest Details:soap:Receiver

    To fix the problem you must change "kill -9" to "kill -15" (or it's equivalent "kill -TERM") in the file /opt/zimbra/bin/zmlogswatchctl

    Then as the zimbra user run the following command:
    zmloggerctl restart

    You will probably still have to kill old swatch_script processes and manually delete their corresponding .swatch_script.??? files from /tmp to cleanup the mess.

  5. #5
    Join Date
    Jul 2007
    Location
    San Jose, CA
    Posts
    1,027
    Rep Power
    10

    Default

    I would not recommend we mark this solved until the solution has been verified by Zimbra. Not because I don't trust you, Jeremiah (and I appreciate your identifying this issue) but because I don't know if someone made the change from "kill -TERM" to "kill -9" to fix a different problem, or whether it was just an error. If the former, then merely reverting to the .10 syntax isn't moving us forward, just whacking a different mole.

    FWIW
    Cheers,

    Dan

  6. #6
    Join Date
    Jul 2007
    Location
    San Jose, CA
    Posts
    1,027
    Rep Power
    10

    Default

    In fact, this fix is not workable for anyone who uses backup scripts or other shell scripts that kill Zimbra using an su - zimbra type command. Calling zmcontrol stop from a script, with kill -TERM results in a failure too:
    Code:
    root@mail:/opt/zimbra/bin# su - zimbra "/opt/zimbra/bin/zmcontrol stop"
    Host mail.mydomain.net
            Stopping stats...Done
            Stopping mta...Done
            Stopping spell...Done
            Stopping snmp...Done
            Stopping archiving...Done
            Stopping antivirus...Done
            Stopping antispam...Done
            Stopping imapproxy...Done
            Stopping mailbox...Done
            Stopping logger...FAILED
    Stopping mysqld...done.
    Stopping logswatch...failed.
    
    
            Stopping ldap...Done
    root@mail:/opt/zimbra/bin# ps ax | grep zmlog
    14040 ?        R      0:01 /usr/bin/perl /opt/zimbra/libexec/zmlogger
     8685 pts/0    S+     0:00 grep zmlog
    So something needs to forcibly kill these processes, but it needs to find ALL of them and kill them. . .
    Cheers,

    Dan

  7. #7
    phoenix is offline Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,587
    Rep Power
    58

    Default

    This command should kill all remaining processes:

    Code:
    kill -9 `ps -u zimbra -o "pid="`
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  8. #8
    Join Date
    Jul 2007
    Location
    San Jose, CA
    Posts
    1,027
    Rep Power
    10

    Default

    Quote Originally Posted by phoenix View Post
    This command should kill all remaining processes:

    Code:
    kill -9 `ps -u zimbra -o "pid="`
    Yeah, I'm just going to add that to my backup script. Thanks, Bill!
    Cheers,

    Dan

  9. #9
    Join Date
    Jul 2007
    Location
    Wilmington, DE
    Posts
    18
    Rep Power
    8

    Default similar thread

    I see similar behavior described in thread: http://www.zimbra.com/forums/install...rade-zcs5.html. I'm investigating if the kill change fixes my issue.

  10. #10
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default

    Curious if anyone has tried the command switch I discovered listed in the Admin Guide:

    Code:
    zmcontrol shutdown
    We are waiting for 5.0.12 so can't test this for you.

    Hope that helps,
    Mark

Similar Threads

  1. Zimbra .pids / service monitoring
    By bin2hex in forum Administrators
    Replies: 24
    Last Post: 04-03-2010, 10:12 PM
  2. Zimbra Cluster Installation
    By veronica in forum Installation
    Replies: 1
    Last Post: 06-25-2008, 02:55 AM
  3. 5.0.6 is out!
    By mmorse in forum Announcements
    Replies: 0
    Last Post: 05-23-2008, 02:16 PM
  4. zmlogger not stopping on zimbra stop
    By holunde in forum Installation
    Replies: 2
    Last Post: 12-17-2007, 05:27 PM
  5. zmlogger not stop
    By mikets in forum Administrators
    Replies: 1
    Last Post: 07-31-2007, 09:56 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •