Results 1 to 5 of 5

Thread: [SOLVED] Stats on 6.0.5 cant be started - /opt/zimbra/libexec/zmstat-fd - Multi-serve

  1. #1
    Join Date
    Sep 2009
    Posts
    99
    Rep Power
    6

    Default [SOLVED] Stats on 6.0.5 cant be started - /opt/zimbra/libexec/zmstat-fd - Multi-serve

    I'm testing to upgrade from 6.0.3 to 6.0.5
    Our setup is multi-server setup

    All stats on our servers, except mailbox servers can't be started
    then when I check zmstatctl status, only zmstat-fd is not running

    after scratching head for half a day, I find out that the script on 6.0.5 and 6.0.3 are different.

    Looks like (sorry not a perl programmer) the 6.0.5 script assumes that it is running on mailbox server.
    I replace 6.0.5 script with 6.0.3 one then stats magically can be started again...

    any comment??

    6.0.3 /opt/zimbra/libexec/zmstat-fd script
    Code:
    #!/usr/bin/perl -w
    #
    # ***** BEGIN LICENSE BLOCK *****
    # Zimbra Collaboration Suite Server
    # Copyright (C) 2008, 2009 Zimbra, Inc.
    #
    # The contents of this file are subject to the Yahoo! Public License
    # Version 1.0 ("License"); you may not use this file except in
    # compliance with the License.  You may obtain a copy of the License at
    # http://www.zimbra.com/license.
    #
    # Software distributed under the License is distributed on an "AS IS"
    # basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.
    # ***** END LICENSE BLOCK *****
    #
    
    # Periodically print FD open stats obtained from /proc/sys/fs/file-nr
    # The first column of /proc/sys/fs/file-nr gives an instantaneous reading
    # of all file descriptors open throughout all processes.  The third column
    # gives a maximum file descriptor count for the entire system.
    
    use strict;
    use Getopt::Long;
    use lib "/opt/zimbra/zimbramon/lib";
    use Zimbra::Mon::Zmstat;
    use Zimbra::Mon::Logger;
    use vars qw($LOGFH $CONSOLE $LOGFILE $ROTATE_NOW $ROTATE_DEFER);
    
    zmstatInit();
    
    my $STAT = '/proc/sys/fs/file-nr';
    my $HEADING = 'timestamp, fd_count';
    
    sub get_fd_stat() {
        open(STAT, "<$STAT") || die "Can't open $STAT: $!";
        my $line = <STAT>;
        $line =~ /^(\d+)\s+/;
        my $stat = $1;
        close(STAT);
        $stat;
    }
    sub sighup {
        if (!$CONSOLE) {
            $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING);
        } else {
            $ROTATE_NOW = 1;
        }
    }
    $SIG{HUP} = \&sighup;
    
    sub usage {
        print STDERR << '_USAGE_';
    Usage: zmstat-fd [options]
    Monitor system filedescriptor usage
    -i, --interval: output a line every N seconds
    -l, --log:      log file (default is /opt/zimbra/zmstat/fd.csv)
    -c, --console:  output to stdout
    If logging to a file, rotation occurs when a HUP signal is sent or when
    the date changes.  The current log is renamed to <dir>/YYY-MM-DD/fd.csv
    and a new file is created.
    _USAGE_
        exit(1);
    }
    
    $| = 1;
    
    $LOGFILE = getLogFilePath('fd.csv');
    my $interval = getZmstatInterval();
    my $opts_good = GetOptions(
        'interval=i' => \$interval,
        'log=s' => \$LOGFILE,
        'console' => \$CONSOLE,
    );
    if (!$opts_good) {
        print STDERR "\n";
        usage();
    }
    createPidFile('zmstat-fd.pid');
    
    my $date = getDate();
    if ($CONSOLE) {
        $LOGFH = \*STDOUT;
        $LOGFH->print($HEADING . "\n");
    
    } else {
        $LOGFH = openLogFile($LOGFILE, $HEADING);
    }
    waitUntilNiceRoundSecond($interval);
    while (1) {
        my $stat = get_fd_stat();
        my $tstamp = getTstamp();
        my $currDate = getDate();
        if ($currDate ne $date && !$CONSOLE) {
            $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING, $date);
            $date = $currDate;
        }
        $ROTATE_DEFER = 1;
        $LOGFH->print("$tstamp, $stat\n");
        Zimbra::Mon::Logger::LogStats( "info", "zmstat fd.csv: ${HEADING}:: $tstamp, $stat");
        $LOGFH->flush();
        $ROTATE_DEFER = 0;
        if ($ROTATE_NOW) {
            $ROTATE_NOW = 0;
            $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING);
        }
        sleep($interval);
    }
    6.0.5 /opt/zimbra/libexec/zmstat-fd script
    Code:
    #!/usr/bin/perl -w
    # 
    # ***** BEGIN LICENSE BLOCK *****
    # Zimbra Collaboration Suite Server
    # Copyright (C) 2008, 2009, 2010 Zimbra, Inc.
    # 
    # The contents of this file are subject to the Zimbra Public License
    # Version 1.3 ("License"); you may not use this file except in
    # compliance with the License.  You may obtain a copy of the License at
    # http://www.zimbra.com/license.
    # 
    # Software distributed under the License is distributed on an "AS IS"
    # basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.
    # ***** END LICENSE BLOCK *****
    # 
    
    # Periodically print FD open stats obtained from /proc/sys/fs/file-nr
    # The first column of /proc/sys/fs/file-nr gives an instantaneous reading
    # of all file descriptors open throughout all processes.  The third column
    # gives a maximum file descriptor count for the entire system.
    
    use strict;
    use Getopt::Long;
    use lib "/opt/zimbra/zimbramon/lib";
    use Zimbra::Mon::Zmstat;
    use Zimbra::Mon::Logger;
    use vars qw($LOGFH $CONSOLE $LOGFILE $ROTATE_NOW $ROTATE_DEFER);
    
    Zimbra::Mon::Zmstat::osCheck();
    Zimbra::Mon::Zmstat::getLocalConfig('zimbra_home', 'zimbra_user', 'zimbra_server_hostname',
                   'zmstat_log_directory', 'zmstat_interval',
                   'zmstat_disk_interval');
    my $zuser = $Zimbra::Mon::Zmstat::LC{zimbra_user};
    my ($zuid,$zgid) = (getpwnam($zuser))[2,3];
    
    if (@ARGV > 0 && $ARGV[0] eq 'stop') {
    	my $dir = getPidFileDir();
    	my $pidFile = "$dir/zmstat-fd-real.pid";
    	my $pid = readPidFile($pidFile);
        if (!kill(0, $pid)) {
            unlink($pidFile);
        } elsif (kill(15, $pid) == 1) {  # SIGTERM
            unlink($pidFile);
        } elsif (kill(9, $pid) == 1) {
        	unlink($pidFile);
        }
    	exit;
    }
    if (@ARGV > 0 && $ARGV[0] eq 'rotate') {
    	my $dir = getPidFileDir();
    	my $pidFile = "$dir/zmstat-fd-real.pid";
    	my $pid = readPidFile($pidFile);
        kill(1, $pid);
    }
    
    if ($< != 0) {
        Zimbra::Mon::Zmstat::userCheck();
        createPidFile('zmstat-fd.pid');
        $SIG{TERM} = sub {
        	system("sudo /opt/zimbra/libexec/zmstat-fd stop");
        };
        $SIG{INT} = sub {
        	system("sudo /opt/zimbra/libexec/zmstat-fd stop");
        };
        $SIG{HUP} = sub {
        	system("sudo /opt/zimbra/libexec/zmstat-fd rotate");
        };
        my $args = "";
        if (@ARGV > 0) {
        	$args = " " . join(' ', @ARGV);
        }
        system("sudo /opt/zimbra/libexec/zmstat-fd$args");
        exit;
    }
    
    my $STAT = '/proc/sys/fs/file-nr';
    my $HEADING = 'timestamp, fd_count, mailboxd_fd_count';
    
    sub get_fd_stat() {
        open(STAT, "<$STAT") || die "Can't open $STAT: $!";
        my $line = <STAT>;
        $line =~ /^(\d+)\s+/;
        my $stat = $1;
        close(STAT);
        $stat;
    }
    sub get_mbox_stat() {
        my $mbox_pid = get_mboxd_pid();
        return 0 if ($mbox_pid == 0);
        
        opendir(PROC, "/proc/$mbox_pid/fd") || warn $!;
        my @ents = grep { /^[0-9]+/ } readdir(PROC);
        closedir(PROC);
        return scalar @ents;
    }
    
    my $MBOX_PID_FILE = '/opt/zimbra/log/zmmailboxd_java.pid';
    sub get_mboxd_pid() {
        my $pid = 0;
        if (-f $MBOX_PID_FILE) {
            eval {
                open(PID_FILE, "<$MBOX_PID_FILE") || die $!;
                $pid = <PID_FILE>;
                chomp($pid); chomp($pid);
                close PID_FILE;
            };
            if ($@) {
                print STDERR "Unable to find pid file, falling back: $@\n";
            }
        }
        if (!$pid) {
            eval {
                open(PS, "ps -ef | grep [j]etty.xml | awk '{ print \$2 }' |")
                        || die $!;
                $pid = <PS>;
                chomp($pid); chomp($pid);
                close(PS);
            };
            if ($@) {
                print STDERR "Unable to manually obtain mailboxd pid: $@\n";
            } elsif (!$pid) {
                print STDERR "Unable to determine mailboxd pid\n";
            }
        }
        return $pid;
    }
    
    sub sighup {
        if (!$CONSOLE) {
            $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING);
        } else {
            $ROTATE_NOW = 1;
        }
    }
    $SIG{HUP} = \&sighup;
    
    sub usage {
        print STDERR << '_USAGE_';
    Usage: zmstat-fd [options]
    Monitor system filedescriptor usage
    -i, --interval: output a line every N seconds
    -l, --log:      log file (default is /opt/zimbra/zmstat/fd.csv)
    -c, --console:  output to stdout
    
    If logging to a file, rotation occurs when a HUP signal is sent or when
    the date changes.  The current log is renamed to <dir>/YYY-MM-DD/fd.csv
    and a new file is created.
    _USAGE_
        exit(1);
    }
    
    $| = 1;
    
    $LOGFILE = getLogFilePath('fd.csv');
    my $interval = getZmstatInterval();
    my $opts_good = GetOptions(
        'interval=i' => \$interval,
        'log=s' => \$LOGFILE,
        'console' => \$CONSOLE,
    );
    if (!$opts_good) {
        print STDERR "\n";
        usage();
    }
    createPidFile('zmstat-fd-real.pid');
    
    my $date = getDate();
    if ($CONSOLE) {
        $LOGFH = \*STDOUT;
        $LOGFH->print($HEADING . "\n");
        
    } else {
        $LOGFH = openLogFile($LOGFILE, $HEADING);
        chown $zuid, $zgid, $LOGFILE;
    }
    waitUntilNiceRoundSecond($interval);
    
    while (1) {
        my $stat = get_fd_stat();
        my $mbox_stat = get_mbox_stat();
        my $tstamp = getTstamp();
        my $currDate = getDate();
        if ($currDate ne $date && !$CONSOLE) {
            $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING, $date);
            chown $zuid, $zgid, $LOGFILE;
            $date = $currDate;
        }
        $ROTATE_DEFER = 1;
        $LOGFH->print("$tstamp, $stat, $mbox_stat\n");
        Zimbra::Mon::Logger::LogStats( "info", "zmstat fd.csv: ${HEADING}:: $tstamp, $stat, $mbox_stat"); 
        $LOGFH->flush();
        $ROTATE_DEFER = 0;
        if ($ROTATE_NOW) {
            $ROTATE_NOW = 0;
            $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING);
        }
        sleep($interval);
    }

  2. #2
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,017
    Rep Power
    25

  3. #3
    Join Date
    Sep 2009
    Posts
    99
    Rep Power
    6

    Default

    Ah.. Voted.. !

    .
    .


  4. #4
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,017
    Rep Power
    25

    Default

    And you can just make the change yourself to /etc/sudoers so that your stats will work.

  5. #5
    Join Date
    Nov 2006
    Posts
    31
    Rep Power
    9

    Default Very similar problem

    I had a similar problem on Release 6.0.5_GA_2213.UBUNTU8 UBUNTU8 FOSS edition. I upgraded from 6.0.4 to 6.0.5 several days ago. Last night, at 12:01 am, I got an email saying that service stats was stopped. All attempts to revive it failed. I have but a single server, so all services run locally, and I already had the line in my /etc/sudoers file:

    %zimbra ALL=NOPASSWD:/opt/zimbra/libexec/zmstat-fd *

    I looked on my company's Release 6.0.5_GA_2213.UBUNTU8 UBUNTU8 NETWORK edition to see what the status processes should look like, ran `/usr/bin/perl -w /opt/zimbra/libexec/zmstat-cpu' by hand, and got notice that it couldn't jigger log files around. I see that root wound up owning the stat logfile directory in /opt/zimbra/zmstat/2010-02-23. I changed the ownership back to zimbra, and things seem to be working again.

    Although this seems to be a different issue than having a split server setup, the errors looked exactly the same. I don't know why root took over that directory. I'm new to running Zimbra, but my personal server had been purring along for a week when this happened, and I can't think of what I must have done to cause this to happen. Anyway, I thought I'd leave this here to help someone else. It may or may not be completely appropriate. ;-)

    UPDATE: I keep coming back to reference this because it seems to be happening about once a week. I changed the perms on /opt/zimbra/zmstat/ to g+w and guid. Hopefully this will force any new directories created there to be group-owned by zimbra, and forced to be group writable, and this will avoid the service from getting hung.
    Last edited by dunkirk; 03-26-2010 at 06:31 AM. Reason: UPDATE

Similar Threads

  1. Replies: 6
    Last Post: 07-07-2013, 09:08 PM
  2. Replies: 9
    Last Post: 02-25-2009, 04:39 AM
  3. Zimbra fails after working for 2 weeks
    By Linsys in forum Administrators
    Replies: 10
    Last Post: 10-07-2008, 01:42 AM
  4. Error after installation
    By robsontuxlinux in forum Installation
    Replies: 13
    Last Post: 09-11-2008, 10:48 PM
  5. Replies: 1
    Last Post: 08-09-2008, 10:15 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •