Results 1 to 10 of 13

Thread: logswatch.out and zmlogger.out BIG!

Hybrid View

  1. #1
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default logswatch.out and zmlogger.out BIG!

    /tmp/logswatch.out and /tmp/zmlogger.out are at 1.3GB and 750MB in size, respectively.

    Are these files required to be kept around or can I periodically delete them?

    Will Zimbra recreate them if they are not there? For example, is this OK:

    zmcontrol stop
    rm /tmp/logswatch.out
    rm /tmp/zmlogger.out
    zmcontrol start

    Thanks!
    Mark

  2. #2
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    21

    Default

    did you apply the patch for solving the bug for the logger stopping?

    If so, you may wish to revert back

    john

  3. #3
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default

    Quote Originally Posted by jholder View Post
    did you apply the patch for solving the bug for the logger stopping?

    If so, you may wish to revert back

    john
    Hi John,

    I did indeed apply the patch, but didn't think the patch would survive the upgrade to 4.5.6 from 4.5.5.

    I'll go back out the patch, and then may I assume it is safe to delete the files after stopping Zimbra?

    Thanks John,
    Mark

  4. #4
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    21

    Default

    hmm. good point.

    I'm gonna have to ask Brian to chime in on this.

  5. #5
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default

    Quote Originally Posted by jholder View Post
    hmm. good point.

    I'm gonna have to ask Brian to chime in on this.
    OK, so I backed out the patch but haven't run 'zmlogswatchctl restart' yet.

    Is this helpful?:

    viognier:/opt/zimbra/bin # ls -al zmlogsw*
    -rwxr-xr-x 1 root root 2019 Jul 17 23:36 zmlogswatchctl
    -rwxr-xr-x 1 root root 2019 Jul 17 23:36 zmlogswatchctl.456
    -rwxr-xr-x 1 root root 2019 Jun 7 10:10 zmlogswatchctl.orig
    viognier:/opt/zimbra/bin # diff zmlogswatchctl.456 zmlogswatchctl
    viognier:/opt/zimbra/bin #

    viognier:/opt/zimbra/libexec # ls -al zmlogprocess*
    -rwxr-xr-x 1 root root 37108 Jul 17 23:35 zmlogprocess
    -rwxr-xr-x 1 root root 37365 Jul 17 23:35 zmlogprocess.456
    -rwxr-xr-x 1 root root 37108 Jun 7 10:10 zmlogprocess.orig
    viognier:/opt/zimbra/libexec # diff zmlogprocess.456 zmlogprocess
    554,559d553
    < $sth = sqlExec("delete from disk_aggregate where ".
    < "period_start='".$periodStart."' and ".
    < "period_end='".$periodEnd."' and ".
    < "device='".$dev."' and ".
    < "host='".$host."'");
    <
    viognier:/opt/zimbra/libexec #

    viognier:/opt/zimbra/libexec # ls -al zmlogger*
    -rwxr-xr-x 1 root root 4448 Jul 17 23:35 zmlogger
    -rwxr-xr-x 1 root root 6065 Jul 17 23:34 zmlogger.456
    -rwxr-xr-x 1 root root 4448 Jun 7 10:11 zmlogger.orig
    -rwxr-xr-x 1 root root 3871 Jun 27 20:05 zmloggerinit
    -rwxr-xr-x 1 root root 1936 Jun 27 20:05 zmloggerprune
    viognier:/opt/zimbra/libexec # diff zmlogger.456 zmlogger
    29,31c29
    < my $zimbra_home = "/opt/zimbra";
    <
    < use lib "${zimbra_home}/zimbramon/lib";
    ---
    > use lib "/opt/zimbra/zimbramon/lib";
    37,39c35,37
    < #$SIG{QUIT} = \&quit(0, "Received sigQUIT");
    < #$SIG{KILL} = \&quit(0, "Received sigKILL");
    < #$SIG{INT} = \&quit(0, "Received sigINT");
    ---
    > my $baseDir = "/opt/zimbra";
    >
    > my $data_source="dbi:mysql:database=zimbra_logger;mys ql_read_default_file=/opt/zimbra/conf/my.logger.cnf;mysql_socket=/opt/zimbra/logger/db/mysql.sock";
    41,42d38
    < my $debug = 1;
    < my $data_source="dbi:mysql:database=zimbra_logger;mys ql_read_default_file=${zimbra_home}/conf/my.logger.cnf;mysql_socket=${zimbra_home}/logger/db/mysql.sock";
    44c40
    < my $password = `${zimbra_home}/bin/zmlocalconfig -s -m nokey zimbra_logger_mysql_password`;
    ---
    > my $password = `/opt/zimbra/bin/zmlocalconfig -s -m nokey zimbra_logger_mysql_password`;
    46,47d41
    < my $platform=`${zimbra_home}/libexec/get_plat_tag.sh`;
    < chomp $platform;
    49c43,54
    < my $dbh = dbConnect($data_source,$username,$password);
    ---
    > my $platform=`/opt/zimbra/libexec/get_plat_tag.sh`;
    > chomp $platform;
    > my $timeout=0;
    > while (! isSqlRunning() && ($timeout <= 120) ) {
    > $timeout += sleep 5;
    > }
    > my $dbh = DBI->connect($data_source, $username, $password);
    >
    > if (!$dbh) {
    > print STDERR "DB: Can't connect to $data_source: $DBI::errstr\n";
    > exit 1;
    > }
    52,64c57,69
    < 'Jan' => '01',
    < 'Feb' => '02',
    < 'Mar' => '03',
    < 'Apr' => '04',
    < 'May' => '05',
    < 'Jun' => '06',
    < 'Jul' => '07',
    < 'Aug' => '08',
    < 'Sep' => '09',
    < 'Oct' => '10',
    < 'Nov' => '11',
    < 'Dec' => '12'
    < );
    ---
    > 'Jan' => '01',
    > 'Feb' => '02',
    > 'Mar' => '03',
    > 'Apr' => '04',
    > 'May' => '05',
    > 'Jun' => '06',
    > 'Jul' => '07',
    > 'Aug' => '08',
    > 'Sep' => '09',
    > 'Oct' => '10',
    > 'Nov' => '11',
    > 'Dec' => '12'
    > );
    65a71,81
    > sub logdate_to_sqldate {
    > my $ts = shift;
    > if ($ts eq "" || !defined($ts)) {
    > return '';
    > }
    > # Sep 14 18:30:02
    >
    > my @p = split (' ', $ts);
    > my @s= localtime();
    > my $y = $s[5]+1900;
    > my $m = $MON{$p[0]};
    67,69c83,85
    < my $statement =
    < "insert into raw_logs(log_date, loghost, app, pid, msg, postfix_qid) values (?,?,?,?,?,?)";
    < my $sth = $dbh->prepare($statement);
    ---
    > my $dt = sprintf ("%4d-%02d-%02d %s",$y,$m,$p[1],$p[2]);
    > return $dt;
    > }
    71,73d86
    < my $status_statement =
    < "insert into service_status(server, service, time, status, loghostname) values (?,?,?,?,?)";
    < my $s_sth = $dbh->prepare($status_statement);
    74a88,89
    > sub get_last_timestamp {
    > my $ts;
    76,79c91
    < open FOO, ">/tmp/zmlogger.out" if $debug;
    < select FOO;
    < $|=1;
    < logMsg("INFO", "zmlogger starting up");
    ---
    > my $statement = "select max(time) from server_stat";
    81c93
    < my $logregex = qr/(^.{15}) ((\d+\.\d+\.\d+\.\d+) \S+|(\S+)) ([^[]+)\[(\d+)\]: (.*)$/o;
    ---
    > my $sth = $dbh->prepare($statement);
    83,84c95,98
    < while (<>) {
    < next if (/last message repeated/);
    ---
    > if (!$sth->execute) {
    > Zimbra::Logger::Log ("err", "DB: $sth->errstr");
    > return undef;
    > }
    86,95c100
    < # attempt to reconnect if mysql dropped
    < if (!$dbh->ping) {
    < logMsg("DEBUG", "DB Connection dropped. Attempting to reconnect");
    < undef $dbh;
    < $dbh = dbConnect($data_source,$username,$password);
    <
    < # prepare the statements with the new connection
    < $sth = $dbh->prepare($statement);
    < $s_sth = $dbh->prepare($status_statement);
    < }
    ---
    > my @ary = $sth->fetchrow_array;
    97,99c102,104
    < # parse the input
    < my ($log_date, $host, $ip, $name, $app, $pid, $msg) = ($_ =~ m/$logregex/);
    < $host = (($ip ne "") ? $ip : $name);
    ---
    > return (datetime_to_timestamp($ary[0]));
    >
    > }
    101,145c106,107
    < # skip slapd and master
    < if ($app eq 'slapd' || $app eq 'master') {
    < logMsg("DEBUG", "Skipping update for app $app");
    < next;
    < } else {
    < logMsg("DEBUG", "hostname:$host app:$app name:$name log_date:$log_date");
    < }
    <
    < next if ($host eq "");
    < my $qid = undef;
    <
    < $qid = $1 if ($app =~ /^postfix/ && $msg =~ /^(\S{8,12}): /);
    <
    < if ($msg =~ /info: ([^,]+), STATUS: (\S+): (\S+): (Running|Stopped).*$/) {
    < my $tm = $1;
    < my $hostname = $2;
    < my $service = $3;
    < my $status = ($4 eq "Running")?1:0;
    <
    < # delete previous entries
    < my $statement = "delete from service_status ".
    < "where server=\'".$hostname."\' and service=\'".$service."\'";
    < my $dsth = $dbh->prepare($statement);
    < if (!$dsth->execute() ) {
    < logMsg("ERROR", $dsth->errstr);
    < next;
    < }
    < # insert new status entries
    < if (!$s_sth->execute($hostname, $service, $tm, $status, $host)) {
    < logMsg("ERROR", "Status update failed for service:$service host:$host status:$status");
    < logMsg("ERROR", $s_sth->errstr);
    < } else {
    < logMsg("INFO", "Status update for service:$service host:$host status:$status tm:$tm");
    < }
    < next;
    < }
    <
    < if (!$sth->execute(logdate_to_sqldate($log_date), $host, $app, $pid, $msg, $qid) ) {
    < logMsg("ERROR", "Error inserting $log_date, $host, $app, $pid, $msg");
    < logMsg("DEBUG", "From $_");
    < logMsg("DEBUG", "DBI errstr: ",$sth->errstr);
    < #logMsg("ERROR", "zmlogger insert failed $sth->errstr");
    < } else {
    < logMsg("INFO", "zmlogger insert completed: $host $app $pid $msg $qid");
    < }
    ---
    > my $statement =
    > "insert into raw_logs(log_date, loghost, app, pid, msg, postfix_qid) values (?,?,?,?,?,?)";
    147,148c109
    < }
    < quit(1, "Loop failed this should never be reached");
    ---
    > my $sth = $dbh->prepare($statement);
    150c111,112
    < # Sub Routines
    ---
    > my $status_statement =
    > "insert into service_status(server, service, time, status, loghostname) values (?,?,?,?,?)";
    152,155c114
    < sub isSqlRunning {
    < `${zimbra_home}/bin/logmysqladmin status > /dev/null 2>&1`;
    < return(($? == 0) ? 1 : 0);
    < }
    ---
    > my $s_sth = $dbh->prepare($status_statement);
    157,188c116
    < sub quit($$) {
    < my ($rc,$msg) = @_;
    < logMsg("INFO", "$msg");
    < logMsg("INFO", "zmlogger shutting down");
    < close FOO if $debug;
    < exit $rc;
    < }
    <
    < sub logMsg {
    < my ($lvl, $msg) = @_;
    < print FOO localtime(time), " $msg\n" if $debug;
    < print "$msg\n" if ($lvl eq "INFO" || $lvl eq "ERROR");
    < }
    <
    < sub dbConnect($$$) {
    < my ($db_src,$db_user,$db_pass) = @_;
    < my $timeout=0;
    <
    < while (! isSqlRunning() && ($timeout <= 3600) ) {
    < logMsg("INFO", "database not running. retrying connection");
    < $timeout += sleep 5;
    < }
    < quit(1, "zmlogger database isn't running") unless isSqlRunning();
    <
    < my $dbh = DBI->connect($db_src,$db_user,$db_pass);
    < $dbh->{PrintError} = 0 unless $debug;
    < $dbh->{RaiseError} = 0 unless $debug;
    <
    < if (!$dbh) {
    < logMsg("ERROR", "DB: Can't connect to $db_src: $DBI::errstr");
    < quit(1, "Can not connect to database $DBI::errstr");
    < }
    ---
    > my $DEBUG = 0;
    190,191c118
    < return $dbh;
    < }
    ---
    > ($DEBUG) && open FOO, ">/tmp/zmlogger.out";
    193,203c120
    < sub logdate_to_sqldate {
    < my $ts = shift;
    < if ($ts eq "" || !defined($ts)) {
    < return '';
    < }
    < # Sep 14 18:30:02
    <
    < my @p = split (' ', $ts);
    < my @s= localtime();
    < my $y = $s[5]+1900;
    < my $m = $MON{$p[0]};
    ---
    > ($DEBUG) && print FOO localtime()," zmlogger starting up\n";
    205,207c122
    < my $dt = sprintf ("%4d-%02d-%02d %s",$y,$m,$p[1],$p[2]);
    < return $dt;
    < }
    ---
    > $SIG{QUIT} = \&bye;
    208a124
    > my $logregex = qr/(^.{15}) ((\d+\.\d+\.\d+\.\d+) \S+|(\S+)) ([^[]+)\[(\d+)\]: (.*)$/o;
    210,211c126,127
    < sub get_last_timestamp {
    < my $ts;
    ---
    > while (<>) {
    > (/last message repeated/) && next;
    213c129,130
    < my $statement = "select max(time) from server_stat";
    ---
    > my ($log_date, $host, $ip, $name, $app, $pid, $msg) = ($_ =~ m/$logregex/);
    > $host = (($ip ne "") ? $ip : $name);
    215c132
    < my $sth = $dbh->prepare($statement);
    ---
    > if ($app eq 'slapd' || $app eq 'master') {next;}
    217,220c134,135
    < if (!$sth->execute) {
    < Zimbra::Logger::Log ("err", "DB: $sth->errstr");
    < return undef;
    < }
    ---
    > if ($host eq "") { next;}
    > my $qid = undef;
    222c137,184
    < my @ary = $sth->fetchrow_array;
    ---
    > if ($app =~ /^postfix/ && $msg =~ /^(\S{8,12}): /) {
    > $qid = $1;
    > }
    >
    > if ($msg =~ /info: ([^,]+), STATUS: (\S+): (\S+): (Running|Stopped).*$/) {
    > my $tm = $1;
    > my $hostname = $2;
    > my $service = $3;
    > my $status = ($4 eq "Running")?1:0;
    > my $statement = "delete from service_status ".
    > "where server=\'".$hostname."\' and service=\'".$service."\'";
    >
    > my $dsth = $dbh->prepare($statement);
    >
    > #print "Executing $statement with @args\n\n";
    >
    > eval {
    > if (!$dsth->execute() ) {
    > warn $dsth->errstr;
    > next;
    > }
    > };
    >
    >
    > eval {
    > $s_sth->execute($hostname, $service, $tm, $status, $host);
    > };
    >
    > next;
    > }
    >
    > eval {
    > if (!$sth->execute(logdate_to_sqldate($log_date), $host,
    > $app, $pid, $msg, $qid) ) {
    > ($DEBUG) && print FOO localtime(),
    > " zmlogger insert failed: ",$sth->errstr,"\n";
    > } else {
    > #($DEBUG) && print FOO localtime(),
    > #" zmlogger insert completed: $host $app $pid $msg $qid\n";
    > }
    >
    > };
    > if ($@) {
    > print "Error inserting $log_date, $host, $app, $pid, $msg\n";
    > print "From $_\n";
    > print $sth->errstr,"\n";
    > print "$@\n";
    > }
    224,225d185
    < return (datetime_to_timestamp($ary[0]));
    <
    226a187,198
    > ($DEBUG) && close FOO;
    >
    > sub isSqlRunning {
    > `logmysqladmin status > /dev/null 2>&1`;
    > return(($? == 0) ? 1 : 0);
    > }
    >
    > sub bye {
    > ($DEBUG) && close FOO;
    > exit (0);
    > }
    >
    viognier:/opt/zimbra/libexec #

    All the best,
    Mark

  6. #6
    Join Date
    Jul 2006
    Posts
    623
    Rep Power
    10

    Default

    The patch was mostly rolled into 4.5.6 but unfortunately the debug mode got left on in a couple of the scripts so these files will continue to be large. 4.5.7 will lower the logging level back to the normal level.

    4.5.7 will also move the files from /tmp to /opt/zimbra/log and they will be part of the regular log rotation. In the meantime it is safe to truncate the files after stopping logger. They will be recreated upon restart.

    Code:
    % zmlogswatchctl stop
    % rm -f /tmp/zmlogger.out /tmp/logswatch.out
    % zmlogswatchctl start
    Bugzilla - Wiki - Downloads - Before posting... Search!

  7. #7
    phoenix is offline Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,587
    Rep Power
    58

    Default

    The raw logs shouldn't be too large even with the 31d retention if you're using a current build of Zimbra, it would depend to some extent on the activity of your server.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •