Results 1 to 3 of 3

Thread: Upgrade from 7.1.1 to 7.2 on Ubuntu 10.04.4 LTS broke rsyslog and zmlogger

  1. #1
    Join Date
    Jun 2012
    Posts
    3
    Rep Power
    3

    Default Upgrade from 7.1.1 to 7.2 on Ubuntu 10.04.4 LTS broke rsyslog and zmlogger

    Hi,

    I just upgraded from zcs-7.1.1_GA_3196.UBUNTU10_64 to zcs-7.2.0_GA_2669.UBUNTU10_64 and at first it seemed to go without a hitch however a few minutes later I got a Nagios warning on disk space remaining on / dropping to 0%. For some reason I cannot yet figure out rsyslogd, zmlogger and perl processes were all chugging along with 100% CPU use and the /var/log/mail.log, mail.info, zimbra.log aimbra-stats.log files all exploded in size and filled up /. Looking at the log files it looks like the log messages are being duplicated over and over again and entries are not listed chronologically. I stopped the rsyslog service freed up some space and rebooted the server and the same problem reoccurred immediatly. I've since stopped the rsyslog service (it also stops zmlogger) and the rest of the server is functioning normally which is nice but I would really like to get the loggers running again soon.

    One thing I have noticed after the upgrade is that when I start or stop the Zimbra service I now get the following errors:

    Use of uninitialized value $current_proto in string eq at /usr/lib/perl5/Sys/Syslog.pm line 371.
    Use of uninitialized value $current_proto in string eq at /usr/lib/perl5/Sys/Syslog.pm line 374.

    which I've never seen before the update. I think the problem has to do with that warning, I've tried setting the initial value of that var to 0 as suggested by http://www.zimbra.com/forums/adminis...tml#post218843 but that didn't help at all with my problem.

    My rsyslog conf files are listed below as I think that is probably where the problem is as I haven't been able to find anything the Zimbra logs that would indicate an issue.

    root@mail:~$ cat /etc/rsyslog.conf
    Code:
    #  /etc/rsyslog.conf    Configuration file for rsyslog.
    #
    #                       For more information see
    #                       /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
    #
    #  Default logging rules can be found in /etc/rsyslog.d/50-default.conf
    
    
    #################
    #### MODULES ####
    #################
    
    $ModLoad imuxsock # provides support for local system logging
    $ModLoad imklog   # provides kernel logging support (previously done by rklogd)
    #$ModLoad immark  # provides --MARK-- message capability
    
    $KLogPath /proc/kmsg
    
    # provides UDP syslog reception
    $ModLoad imudp
    $UDPServerRun 514
    
    # provides TCP syslog reception
    #$ModLoad imtcp
    #$InputTCPServerRun 514
    
    
    ###########################
    #### GLOBAL DIRECTIVES ####
    ###########################
    
    #
    # Use traditional timestamp format.
    # To enable high precision timestamps, comment out the following line.
    #
    $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
    
    # Filter duplicated messages
    $RepeatedMsgReduction on
    
    #
    # Set the default permissions for all log files.
    #
    $FileOwner syslog
    $FileGroup adm
    $FileCreateMode 0640
    $DirCreateMode 0755
    $Umask 0022
    $PrivDropToUser syslog
    $PrivDropToGroup syslog
    
    #
    # Include all config files in /etc/rsyslog.d/
    #
    $IncludeConfig /etc/rsyslog.d/*.conf
    root@mail:~$ cat /etc/rsyslog.d/20-ufw.conf
    Code:
    # Log kernel generated UFW log messages to file
    :msg,contains,"[UFW " /var/log/ufw.log
    
    # Uncomment the following to stop logging anything that matches the last rule.
    # Doing this will stop logging kernel generated UFW log messages to the file
    # normally containing kern.* messages (eg, /var/log/kern.log)
    #& ~
    root@mail:~$ cat /etc/rsyslog.d/50-default.conf
    Code:
    #  Default rules for rsyslog.
    #
    #                       For more information see rsyslog.conf(5) and /etc/rsyslog.conf
    
    #
    # First some standard log files.  Log by facility.
    #
    auth,authpriv.*                 /var/log/auth.log
    *.*;auth,authpriv.none;local0.none;local1.none;mail.none                -/var/log/syslog
    #cron.*                         /var/log/cron.log
    daemon.*                        -/var/log/daemon.log
    kern.*                          -/var/log/kern.log
    lpr.*                           -/var/log/lpr.log
    mail.*                          -/var/log/mail.log
    user.*                          -/var/log/user.log
    
    #
    # Logging for the mail system.  Split it up so that
    # it is easy to write scripts to parse these files.
    #
    mail.info                       -/var/log/mail.info
    mail.warn                       -/var/log/mail.warn
    mail.err                        /var/log/mail.err
    
    #
    # Logging for INN news system.
    #
    news.crit                       /var/log/news/news.crit
    news.err                        /var/log/news/news.err
    news.notice                     -/var/log/news/news.notice
    
    #
    # Some "catch-all" log files.
    #
    *.=debug;\
            local0,local1.none;\
            auth,authpriv.none;\
            news.none       -/var/log/debug
    *.=info;*.=notice;*.=warn;\
            local0,local1.none;\
            auth,authpriv.none;\
            cron,daemon.none;\
            mail,news.none          -/var/log/messages
    
    #
    # Emergencies are sent to everybody logged in.
    #
    *.emerg                         *
    
    #
    # I like to have messages displayed on the console, but only on a virtual
    # console I usually leave idle.
    #
    #daemon,mail.*;\
    #       news.=crit;news.=err;news.=notice;\
    #       *.=debug;*.=info;\
    #       *.=notice;*.=warn       /dev/tty8
    
    # The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
    # you must invoke `xconsole' with the `-file' option:
    #
    #    $ xconsole -file /dev/xconsole [...]
    #
    # NOTE: adjust the list below, or you'll go crazy if you have a reasonably
    #      busy site..
    #
    daemon.*;mail.*;\
            news.err;\
            *.=debug;*.=info;\
            *.=notice;*.=warn       |/dev/xconsole
    root@mail:~$ cat /etc/rsyslog.d/60-zimbra.conf
    Code:
    local0.*                @mail.feds.uwaterloo.ca
    local1.*                @mail.feds.uwaterloo.ca
    auth.*                  @mail.feds.uwaterloo.ca
    local0.*                -/var/log/zimbra.log
    local1.*                -/var/log/zimbra-stats.log
    auth.*                  -/var/log/zimbra.log
    mail.*                @mail.feds.uwaterloo.ca
    mail.*                -/var/log/zimbra.log

    Any help is greatly appreciated. Thanks!

  2. #2
    Join Date
    Jun 2012
    Posts
    3
    Rep Power
    3

    Default

    So a good buddy of mine figured it out. It turns out that during the upgrade process the installer added and additional rsyslog configuration file /etc/rsyslog.d/60-zimbra.conf that included 4 directives that caused rsyslogd to pass the same message back to itself in an infinite loop. Changing the /etc/rsyslog.d/60-zimbra.conf file to the following solved the problem:

    root-@mail:~$ cat /etc/rsyslog.d/60-zimbra.conf
    Code:
    #local0.*                @mail.feds.uwaterloo.ca
    #local1.*                @mail.feds.uwaterloo.ca
    #auth.*                  @mail.feds.uwaterloo.ca
    local0.*                -/var/log/zimbra.log
    local1.*                -/var/log/zimbra-stats.log
    auth.*                  -/var/log/zimbra.log
    #mail.*                @mail.feds.uwaterloo.ca
    mail.*                -/var/log/zimbra.log

  3. #3
    Join Date
    Oct 2007
    Location
    Woodland, WA
    Posts
    4
    Rep Power
    8

    Default

    Same upgrade path, same distro, same arch. I just wanted to let you know the above fix worked for me as well. It got rid of not only the rsyslog cpu usage but also perl and zmlogger cpu usage. Probably all related I'm sure. Big thanks

Similar Threads

  1. zmlogger, syslog, rsyslog and stats
    By loopietky in forum Administrators
    Replies: 2
    Last Post: 08-08-2012, 06:05 AM
  2. zmlogger, syslog, rsyslog and stats
    By loopietky in forum Installation
    Replies: 0
    Last Post: 11-19-2010, 07:29 PM
  3. Upgrade from 4.5.10 to 5.0.0b3 broke permissions
    By toolcaserp in forum Administrators
    Replies: 9
    Last Post: 12-11-2007, 03:25 PM
  4. dspam broke after 4.5.7 upgrade?
    By Krishopper in forum Administrators
    Replies: 9
    Last Post: 09-24-2007, 09:07 AM
  5. Upgrade to 3.0.1_GA_160 broke
    By jwilso2 in forum Installation
    Replies: 2
    Last Post: 02-24-2006, 09:44 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •