Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: [SOLVED] total zimbra cs startup time 1.5 hours

  1. #1
    Join Date
    Apr 2010
    Posts
    7
    Rep Power
    5

    Default [SOLVED] total zimbra cs startup time 1.5 hours

    After few month of using Zimbra Collaboration Suite i've found that startup time became near 90 minutes (93 last time). After installation and test usage this time was less than 2 minutes.
    "zmcontrol status" command takes 3 minutes.
    During startup time processor usage by zimbra components is near 0 (by "top" utility), disk activity low, 20-100 block read/write operations in 10 seconds (by "iostat")
    I don't know, what was the reason of this. Every night cron launches backup script based on "Open Source Edition Backup Procedure", "Simple Shell Script Method" - shutdown zimbra, rsync files to another storage, startup server. Because night is more than 90 minutes, nobody have seen this problem.
    Server work 24/7 without restart.
    Total mail size on server near 16 GB, 20 GB with database and other software. 19 account, 11 with mailbox more than 100Mb. Statistic is not accumulated because it didn't work after installation, it had big size, it wasn't needed.

    CPU is Xeon E5420, RAM 2 GB + swap 2GB, Mail filesystem 60GB, 66% free.

    Startup part from /var/log/zimbra.log is in attachment - server started from init after reboot.
    Attached Files Attached Files

  2. #2
    Join Date
    Jan 2009
    Posts
    369
    Rep Power
    6

    Default

    Follow your log, I don't see any errors.
    Can you try to disable AV/AS service to check if because of them your server starts too slowly?

  3. #3
    Join Date
    Dec 2007
    Location
    Ypsilanti, MI
    Posts
    145
    Rep Power
    8

    Default

    Quote Originally Posted by morozovvs View Post
    After few month of using Zimbra Collaboration Suite i've found that startup time became near 90 minutes (93 last time). After installation and test usage this time was less than 2 minutes.
    "zmcontrol status" command takes 3 minutes.
    During startup time processor usage by zimbra components is near 0 (by "top" utility), disk activity low, 20-100 block read/write operations in 10 seconds (by "iostat")
    I don't know, what was the reason of this. Every night cron launches backup script based on "Open Source Edition Backup Procedure", "Simple Shell Script Method" - shutdown zimbra, rsync files to another storage, startup server. Because night is more than 90 minutes, nobody have seen this problem.
    Server work 24/7 without restart.
    Total mail size on server near 16 GB, 20 GB with database and other software. 19 account, 11 with mailbox more than 100Mb. Statistic is not accumulated because it didn't work after installation, it had big size, it wasn't needed.

    CPU is Xeon E5420, RAM 2 GB + swap 2GB, Mail filesystem 60GB, 66% free.

    Startup part from /var/log/zimbra.log is in attachment - server started from init after reboot.
    The better file to look at is likely the mailbox.log. You're also running the bare minimum amount of RAM in the system, and if the system is swapping alot then you're likely found your problem. What does the free output look like?

  4. #4
    Join Date
    Apr 2010
    Posts
    7
    Rep Power
    5

    Default

    There are no errors. The problem is that startup takes a long time. I'll try to disable Debian's clamav. There are no other working AV daemons.

    Output of memory usage tool:
    Code:
    # free
                 total       used       free     shared    buffers     cached
    Mem:       2061752    1894308     167444          0      12988     274656
    -/+ buffers/cache:    1606664     455088
    Swap:      1951856     439048    1512808
    I'll try reduce memory usage by shutdown of some non-zimbra daemons and removing them from startup. After shutdown of them i have following:
    Code:
    # free
                 total       used       free     shared    buffers     cached
    Mem:       2061752    1709736     352016          0      22788     295396
    -/+ buffers/cache:    1391552     670200
    Swap:      1951856     401484    1550372
    About mailbox.log compared to zimbra.log:
    I've stared Zimbra at 13:37, but first record in mailbox.log appeared at 14:32
    Code:
    2010-04-20 12:32:51,638 INFO  [Shutdown] [] log - Shutdown hook complete
    2010-04-20 14:32:20,788 INFO  [main] [] soap - Servlet SoapServlet starting up
    2010-04-20 14:32:20,835 INFO  [main] [] soap - Adding service AccountService to SoapServlet
    2010-04-20 14:32:21,051 INFO  [main] [] soap - Adding service MailService to SoapServlet
    2010-04-20 14:32:21,173 INFO  [main] [] soap - Adding service IMService to SoapServlet
    ...
    I don't know if there some ldap log.
    stacktrace.*.* contains some exceptions (most of them are TIMED_WAITING), but without timestamps.

    I'll look tomorrow at logs, if the problem is in memory.
    Last edited by morozovvs; 04-22-2010 at 01:29 AM.

  5. #5
    Join Date
    Dec 2007
    Location
    Ypsilanti, MI
    Posts
    145
    Rep Power
    8

    Default

    I wouldn't say the problem is necessarily in memory so much as it is in lack of it. As you can see in the free output, you're always swapping. Committing stuff to disk and having to access it from disk is going to be a lot slower than doing everything from memory. You'd do well to increase your RAM when you can and it would likely improve things quite a bit.

  6. #6
    Join Date
    Apr 2010
    Posts
    7
    Rep Power
    5

    Default

    I've disabled swap. So now there are only RAM is used. Also i've decreased quantity of amavis processes from 10 to 5, disabled one AV scaner.
    It didn't change anything. Startup time is still almost same.
    Night backup finish main work at 00:45 and launches zimbra. The first postfix response is near 02:18.
    I'ts also notable that 'zmcontrol status' command takes 2-3 minutes.
    All services, including postfix, amavis, mailbox work fast.
    Code:
    # free
                 total       used       free     shared    buffers     cached
    Mem:       2061752    1717704     344048          0      32484     358628
    -/+ buffers/cache:    1326592     735160
    Swap:            0          0          0

  7. #7
    Join Date
    Dec 2007
    Location
    Ypsilanti, MI
    Posts
    145
    Rep Power
    8

    Default

    Disabling swap is a pretty bad idea. That doesn't change the fact that the system wants to use more memory than it has, it just guarantees that at some point you're going to run out of memory and instead of swapping to disk, it's going to just start killing processes to get more memory.

  8. #8
    Join Date
    Apr 2010
    Posts
    7
    Rep Power
    5

    Default

    Possibly i've found the reason. It's not a memory problem. I think it was Linux ip6tables firewall (it prevented almost all incoming connections by ipv6 protocol). As i saw before, if application require ip connection, it tries IPv6 first, and if it fails, it tries IPv4.
    I've disabled IPv6 firewall (and also disabled IPv6 Internet connection). Now 'zmcontrol status' takes few seconds. Later i'll check Zimbra startup time.

  9. #9
    Join Date
    Apr 2010
    Posts
    7
    Rep Power
    5

    Default

    Yes, the reason was in IPv6 firewall. After disabling of IPv6 firewall (after resetting rules to accept everything) zimbra starts in less than 2 minutes.

  10. #10
    Join Date
    Sep 2009
    Location
    Philippines
    Posts
    11
    Rep Power
    6

    Default

    i know this is an old thread but i found it to be of help in identifying our problem. zmprov, as well as other related zm* commands, takes a very long time to run. and i found out that our IPv6 firewall is the culprit through this thread.

    with the advent of the World IPv6 Day, we enabled IPv6 on our Zimbra installation, following several sources online (see below if interested). we successfully enabled it (dual-stacked with v4), and we modified our firewall definitions for v6. and then this problem came out. since our v4 firewall is working w/o any issues before, i assumed that the same definitions would fit v6. question would be, are there ports other than the ones indicated here that should be enabled for IPv6? thanks in advance. will investigate this further tomorrow.


    resources for enabling IPv6 on Zimbra:

Similar Threads

  1. [SOLVED] Postfix unavailable - queue down
    By pmona in forum Administrators
    Replies: 20
    Last Post: 01-21-2010, 10:03 PM
  2. zimbra install with perpetually broken logger/stats
    By jptech in forum Installation
    Replies: 8
    Last Post: 09-29-2008, 03:33 PM
  3. Major Issue - 5.0RC2 NE to 5.0GA NE failed
    By DougWare in forum Installation
    Replies: 7
    Last Post: 01-06-2008, 09:56 PM
  4. huge log size
    By rmvg in forum Administrators
    Replies: 5
    Last Post: 01-02-2007, 10:39 AM
  5. Fedora Core 3, Clean Install - Not working!
    By pcjackson in forum Installation
    Replies: 17
    Last Post: 03-05-2006, 07:38 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •