Results 1 to 8 of 8

Thread: System-Load with M4

  1. #1
    Join Date
    Sep 2005
    Posts
    51
    Rep Power
    10

    Default System-Load with M4

    Hi,

    I've been looking at Zimbra again having been sidetracked for the past few months. I wrote a small perl script to simulate sending messages to about a dozen users on the system, message-size of 5 - 10k and sending messages every 2-3 seconds (sometimes that means two messages as once, sometimes none for 6 - 10 seconds).

    Anyway - the system load became so high (30+) that I could barely run "uptime" or do anything else on the system. The test machine is an Athlon64 with 1gig of RAM running 32-bit CentOS4.2 (~RHEL4).

    Is this remotely normal? This is with no-one actually reading their mail at the same time.

    Java & amavisd seem to be the two processes I see at the top of "top" most often. I've tried with swap completely disabled and still get the same kind of load.

  2. #2
    Join Date
    Sep 2005
    Posts
    2,103
    Rep Power
    14

    Default system load

    This is on M4 (41, I assume?)

    I've run similar tests, and your system load shouldn't be that high. I'm not sure how much this is related to architecture, but some things to further investigate:

    When you're injecting the messages, how large do the postfix queues get? How many lmtp processes (in postfix) seem busy? Are the messages backing up before amavis, or after? Try tailing /opt/zimbra/log/zimbra.log and look for the message deliveries - are they coming fast, or slow?

    Is /opt/zimbra/amavisd-new-2.3.3/tmp mounted as tmpfs? That speeds up amavis quite a bit.

    What's the IOWAIT on the box like? What kind of disk are you using? (local, san, nfs, usb drive...)

    This isn't running in some kind of virtual machine, is it?

    What's in the process list other than zimbra?

  3. #3
    Join Date
    Oct 2005
    Location
    USA, Canada and India
    Posts
    777
    Rep Power
    11

    Default

    Hi..All the above is good if once really know how to do all this.

    Is it possible to write a sctipt which can dump ZIMBRA realated performenace stats so if a user can run this script every 5 min when they are LOAD TESTING/TROUBLESHOOTING.
    This way data collected by script can be analyzed or posted "here" for expert opinion to Optomize the Zimbra Setup.
    And yes some things Admin need to know on its own but those will be GLOBAL settings of server which is easy to give out.

    If anyone at Zimbra thinks its easy to write this script will be great help

    Thanks
    Raj

  4. #4
    Join Date
    Sep 2005
    Posts
    51
    Rep Power
    10

    Default

    Quote Originally Posted by marcmac
    This is on M4 (41, I assume?)
    Yup.

    When you're injecting the messages, how large do the postfix queues get?
    It's sitting after running for 10 minutes (system load up from 0.3 at idle to 20 now) with about 15 messages in incoming & active.

    How many lmtp processes (in postfix) seem busy? Are the messages backing up before amavis, or after? Try tailing /opt/zimbra/log/zimbra.log and look for the message deliveries - are they coming fast, or slow?
    There only seem to be few (6 or so) lmtp processes running - none seem to be using much system time.
    I'm not sure how to test whether they are backing up before amavis?
    zimbra.log shows a delivery - on average - 3 seconds or so.

    Is /opt/zimbra/amavisd-new-2.3.3/tmp mounted as tmpfs? That speeds up amavis quite a bit.
    Yup


    What's the IOWAIT on the box like? What kind of disk are you using? (local, san, nfs, usb drive...)
    iowait is 30 - 31 or there-abouts.
    Disk is a local sata.

    This isn't running in some kind of virtual machine, is it?
    Nope - regular install.


    What's in the process list other than zimbra?
    A ps listing gives me :

    Code:
    init
    ksoftirqd/0
    events/0
    khelper
    kacpid
    kblockd/0
    pdflush
    pdflush
    aio/0
    khubd
    kswapd0
    kseriod
    ata/0
    scsi_eh_0
    scsi_eh_1
    kmirrord
    kmir_mon
    kjournald
    udevd
    kauditd
    kjournald
    syslogd
    klogd
    portmap
    rpc.statd
    rpc.idmapd
    acpid
    cupsd
    sshd
    xinetd
    gpm
    crond
    xfs
    atd
    dbus-daemon-1
    cups-config-dae
    hald
    freshclam
    sshd
    bash
    mingetty
    mingetty
    mingetty
    mingetty
    mingetty
    mingetty
    gdm-binary
    gdm-binary
    X
    gdmgreeter
    sshd
    bash
    sshd
    bash
    slapd
    zmmtaconfig
    amavisd
    mysqld_safe
    logswatch
    freshclam
    perl
    mysqld
    tail
    clamd
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    zmlogger
    mysqld
    java
    mysqld_safe
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    master
    pickup
    qmgr
    saslauthd
    saslauthd
    saslauthd
    saslauthd
    saslauthd
    swatch
    perl
    httpd
    tail
    httpd
    httpd
    httpd
    httpd
    httpd
    mysqld
    mysqld
    tlsmgr
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    mysqld
    proxymap
    smtp
    smtp
    cleanup
    scache
    smtp
    smtp
    trivial-rewrite
    cleanup
    lmtp
    cleanup
    cleanup
    trivial-rewrite
    cleanup
    mysqld
    smtpd
    smtpd
    amavisd
    amavisd
    amavisd
    lmtp
    amavisd
    amavisd
    amavisd
    amavisd
    crond
    sh
    zmlogprocess
    smtpd
    smtpd
    amavisd
    smtpd
    amavisd
    amavisd
    crond
    zmstatuslog
    test_mail.pl
    zmcontrol
    test_mail.pl
    test_mail.pl
    crond
    zmstatuslog
    smtpd
    zmcontrol
    smtpd
    smtpd
    smtpd
    smtpd
    smtpd
    crond
    smtpd
    zmstatuslog
    smtpd
    trivial-rewrite
    smtpd
    smtpd
    smtpd
    zmcontrol
    java
    smtpd
    smtpd
    cleanup
    smtpd
    cleanup
    cleanup
    smtpd
    smtpd
    smtpd
    smtpd
    smtpd
    cleanup
    cleanup
    cleanup
    cleanup
    cleanup
    cleanup
    cleanup
    cleanup
    cleanup
    cleanup
    lmtp
    lmtp
    smtp
    lmtp
    smtp
    crond
    smtp
    zmstatuslog
    smtp
    smtpd
    zmcontrol
    smtpd
    tail
    java
    sendmail
    sendmail
    sendmail
    smtp
    lmtp
    crond
    sendmail
    sendmail
    sendmail
    lmtp
    lmtp
    zmstatuslog
    zmcontrol
    sh
    sh
    zmantispamctl
    sh
    zmantispamctl
    zmantispamctl
    sendmail
    sendmail
    ps
    bash

  5. #5
    Join Date
    Sep 2005
    Posts
    51
    Rep Power
    10

    Default

    Oh - and yes, that iowait does seem fantastically high Could well be that it's this HP system that's at fault...... I'll have to try the same test on another box and check.

  6. #6
    Join Date
    Sep 2005
    Posts
    2,103
    Rep Power
    14

    Default Performance

    I see 3 mysqld groupings - are you running another DB besides the message store and logger instances?

    I think I asked the IOWAIT question incorrectly - what I was after is what percentage of processes (as reported by top) are in the iowait state?

    It's hard to tell if they're backing up before or after amavis - but if amavis is eating a large amount of CPU, they're probably backing up before it.

    Couple of things to try to narrow this down:

    1 - Run a test that just blasts mail into the system - no delay between sends, multiple processes sending mail to multiple accounts at the same time. Load up the queues, stop sending, and watch what the server does as the queues drain - this will eliminate the variables in the send delay, etc.

    2 - check the postfix logs, look for delay=<num> - this will tell you how long it takes postfix to deliver a message. Each message takes 2 hops: postfix->amavis->postfix (via smtp) and postfix->mailstore (via lmtp). Which delay is higher?

    3 - turn of amavis - you can do this in the admin console, which should cut the amavis step out of the delivery process. This will tell us where things are slowing down.

  7. #7
    Join Date
    Sep 2005
    Posts
    2,103
    Rep Power
    14

    Default high iowait

    What's in iowait? Is it java (writing messages for final delivery) or is it sql (indexing messages) or is it amavis and/or postfix?

  8. #8
    Join Date
    Sep 2005
    Posts
    51
    Rep Power
    10

    Default

    Sorry for not replying to this - I didn't get a mail saying a new post had been added. I took zimbra off of the machine it was on and put it on a more standard P4 box. System load is fine running the same test.

    That's what I get for running a test on a dodgy HP box

Similar Threads

  1. 3.0 to 4.5.3 Upgrade failed (mysql error)
    By dealt in forum Installation
    Replies: 35
    Last Post: 03-19-2007, 11:30 PM
  2. domain coexistence
    By marcmac in forum Administrators
    Replies: 14
    Last Post: 06-30-2006, 02:19 PM
  3. Cannot determine services - exiting
    By tawas in forum Installation
    Replies: 7
    Last Post: 04-25-2006, 03:47 AM
  4. Yet another get.DirectContext issue
    By dccpark in forum Installation
    Replies: 5
    Last Post: 03-08-2006, 01:25 PM
  5. system failure: getDirectContext
    By avisser in forum Installation
    Replies: 3
    Last Post: 10-12-2005, 06:32 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •