Results 1 to 7 of 7

Thread: Zimbra server crashed

Hybrid View

  1. #1
    Join Date
    Nov 2005
    Location
    Austria
    Posts
    223
    Rep Power
    9

    Default Zimbra server crashed

    Hi,

    I didn't find my error messages somewhere else in the forums, if there was a similar problem, I am sorry for this thread.

    This morning I recognised that my Zimbra server wasn't running, so I tried a "zmcontrol status". It didn't work:
    Code:
    Cannot determine services - exiting
    After a "zmcontrol start" the server was running.

    I looked into zimbra.log and I am quite worried.

    I attached a part of zimbra.log. How can I look into the zimbra.log of yesterday? The server has been crashed in the time after 8 PM, that was the time I received the last mail.
    Attached Files Attached Files

  2. #2
    Join Date
    Sep 2005
    Posts
    2,103
    Rep Power
    14

    Default

    Looks like ldap crashed.

    The logs are rotated daily, so there should be a zimbra.log.1, zimbra.log.2, etc.

  3. #3
    Join Date
    Nov 2005
    Location
    Austria
    Posts
    223
    Rep Power
    9

    Default

    Quote Originally Posted by marcmac
    Looks like ldap crashed.

    The logs are rotated daily, so there should be a zimbra.log.1, zimbra.log.2, etc.
    I have attached the part of the log file from where the server was crashing until ~4 hours after it.

    Can you see why LDAP crashed?

    There seems to be a problem with "4-mail.net[62.2.100.201]" (in the log at Feb 27 20:21). This is my backup-MX server (I don't know if this information is relevant, but maybe you can need it).

    Thanks for your help,
    goetzi
    Attached Files Attached Files

  4. #4
    Join Date
    Sep 2005
    Posts
    2,103
    Rep Power
    14

    Default

    Quote Originally Posted by goetzi
    I have attached the part of the log file from where the server was crashing until ~4 hours after it.

    Can you see why LDAP crashed?

    There seems to be a problem with "4-mail.net[62.2.100.201]" (in the log at Feb 27 20:21). This is my backup-MX server (I don't know if this information is relevant, but maybe you can need it).

    Thanks for your help,
    goetzi
    I can't tell why it crashed from this log - apparently it died between
    Feb 27 20:20:52 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: ldap: Running


    and
    Feb 27 20:31:00 zimbra postfix/smtpd[20569]: warning: dict_ldap_lookup: Search error -5: Timed out


    It's logging disk space, which looks ok. What can you tell me about your hardware? (Virtual machine?)

    Also - can you send me the contents of /var/log/messages between 20:20 and 20:35 on that date?

    (It looks like you reduced the frequency of status checking in the crontab - is that correct?)

    Also - did you have any DNS problems around this time?

    This is merely statistical:

    Feb 27 20:32:36 zimbra postfix/anvil[20769]: statistics: max connection rate 1/60s for (smtp:62.2.100.201) at Feb 27 20:22:05
    Feb 27 20:33:36 zimbra postfix/anvil[20769]: statistics: max connection count 1 for (smtp:62.2.100.201) at Feb 27 20:22:05


    Reported by the postfix anvil service (man 8 anvil on your zimbra box for more info).

  5. #5
    Join Date
    Nov 2005
    Location
    Austria
    Posts
    223
    Rep Power
    9

    Default

    What can you tell me about your hardware? (Virtual machine?)
    No, it's no virtual machine.
    It's a PIII 600 Mhz with 386 MB RAM and 18 GB RAID 5 (3 Disks with 9 GB).


    Also - can you send me the contents of /var/log/messages between 20:20 and 20:35 on that date?
    Code:
    Feb 27 20:20:08 zimbra crond(pam_unix)[20526]: session opened for user zimbra by (uid=0)
    Feb 27 20:20:08 zimbra crond(pam_unix)[20529]: session opened for user zimbra by (uid=0)
    Feb 27 20:20:10 zimbra crond(pam_unix)[20527]: session opened for user zimbra by (uid=0)
    Feb 27 20:20:10 zimbra crond(pam_unix)[20528]: session opened for user zimbra by (uid=0)
    Feb 27 20:20:29 zimbra zimbramon[20558]: 20558:info: 2006-02-27 20:20:26, DISK: : dev: /dev/ida/c0d0p3, mp: /, tot: 16139, ava
    il: 5771
    Feb 27 20:20:34 zimbra zimbramon[20558]: 20558:info: 2006-02-27 20:20:26, DISK: : dev: /dev/ida/c0d0p1, mp: /boot, tot: 97, av
    ail: 81
    Feb 27 20:20:37 zimbra crond(pam_unix)[20528]: session closed for user zimbra
    Feb 27 20:20:45 zimbra zimbramon[20556]: 20556:info: 2006-02-27 20:20:26, QUEUE: 0 0
    Feb 27 20:20:48 zimbra crond(pam_unix)[20526]: session closed for user zimbra
    Feb 27 20:20:51 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: antispam: Running
    Feb 27 20:20:52 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: antivirus: Running
    Feb 27 20:20:52 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: ldap: Running
    Feb 27 20:20:53 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: logger: Running
    Feb 27 20:20:55 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: mailbox: Running
    Feb 27 20:20:55 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: mta: Running
    Feb 27 20:20:55 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: snmp: Running
    Feb 27 20:20:56 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: spell: Running
    Feb 27 20:20:54 zimbra crond(pam_unix)[15857]: session closed for user zimbra
    Feb 27 20:21:15 zimbra crond(pam_unix)[20755]: session opened for user zimbra by (uid=0)
    Feb 27 20:22:25 zimbra crond(pam_unix)[20794]: session opened for user zimbra by (uid=0)
    Feb 27 20:23:53 zimbra crond(pam_unix)[20804]: session opened for user zimbra by (uid=0)
    Feb 27 20:24:32 zimbra crond(pam_unix)[20871]: session opened for user zimbra by (uid=0)
    Feb 27 20:26:19 zimbra crond(pam_unix)[20926]: session opened for user zimbra by (uid=0)
    Feb 27 20:28:59 zimbra crond(pam_unix)[20940]: session opened for user zimbra by (uid=0)
    Feb 27 20:29:34 zimbra crond(pam_unix)[20945]: session opened for user zimbra by (uid=0)
    Feb 27 20:30:47 zimbra kernel: oom-killer: gfp_mask=0x1d2
    Feb 27 20:30:51 zimbra kernel: Mem-info:
    Feb 27 20:30:51 zimbra kernel: DMA per-cpu:
    Feb 27 20:30:51 zimbra kernel: cpu 0 hot: low 2, high 6, batch 1
    Feb 27 20:30:51 zimbra kernel: cpu 0 cold: low 0, high 2, batch 1
    Feb 27 20:30:51 zimbra kernel: Normal per-cpu:
    Feb 27 20:30:51 zimbra kernel: cpu 0 hot: low 32, high 96, batch 16
    Feb 27 20:30:51 zimbra kernel: cpu 0 cold: low 0, high 32, batch 16
    Feb 27 20:30:51 zimbra kernel: HighMem per-cpu: empty
    Feb 27 20:30:51 zimbra kernel:
    Feb 27 20:30:51 zimbra kernel: Free pages:         592kB (0kB HighMem)
    Feb 27 20:30:51 zimbra kernel: Active:9098 inactive:37654 dirty:0 writeback:0 unstable:0 free:148 slab:22514 mapped:46423 page
    tables:25807
    Feb 27 20:30:51 zimbra kernel: DMA free:24kB min:24kB low:48kB high:72kB active:212kB inactive:280kB present:16384kB pages_sca
    nned:2994 all_unreclaimable? yes
    Feb 27 20:30:54 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:30:57 zimbra kernel: Normal free:568kB min:596kB low:1192kB high:1788kB active:36180kB inactive:150336kB present:376
    816kB pages_scanned:328267 all_unreclaimable? yes
    Feb 27 20:30:59 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:31:02 zimbra kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scann
    ed:0 all_unreclaimable? no
    Feb 27 20:31:06 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:31:08 zimbra kernel: DMA: 6*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24kB
    Feb 27 20:31:09 zimbra kernel: Normal: 0*4kB 1*8kB 3*16kB 0*32kB 0*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5
    68kB
    Feb 27 20:31:12 zimbra kernel: HighMem: empty
    Feb 27 20:31:13 zimbra kernel: Swap cache: add 304430, delete 304430, find 16703/26938, race 0+4
    Feb 27 20:31:13 zimbra kernel: 0 bounce buffer pages
    Feb 27 20:31:15 zimbra kernel: Free swap:            0kB
    Feb 27 20:31:17 zimbra kernel: 98300 pages of RAM
    Feb 27 20:31:18 zimbra kernel: 0 pages of HIGHMEM
    Feb 27 20:31:19 zimbra kernel: 1784 reserved pages
    Feb 27 20:31:19 zimbra kernel: 11076 pages shared
    Feb 27 20:31:20 zimbra kernel: 0 pages swap cached
    Feb 27 20:31:22 zimbra kernel: Out of Memory: Killed process 20633 (java).
    Feb 27 20:31:29 zimbra crond(pam_unix)[20957]: session opened for user zimbra by (uid=0)
    Feb 27 20:31:29 zimbra crond(pam_unix)[20951]: session opened for user zimbra by (uid=0)
    Feb 27 20:31:29 zimbra crond(pam_unix)[20948]: session opened for user zimbra by (uid=0)
    Feb 27 20:33:32 zimbra crond(pam_unix)[20958]: session opened for user zimbra by (uid=0)
    Feb 27 20:35:44 zimbra kernel: oom-killer: gfp_mask=0x1d2
    Feb 27 20:35:44 zimbra kernel: Mem-info:
    Feb 27 20:35:44 zimbra kernel: DMA per-cpu:
    Feb 27 20:35:44 zimbra kernel: cpu 0 hot: low 2, high 6, batch 1
    Feb 27 20:35:44 zimbra kernel: cpu 0 cold: low 0, high 2, batch 1
    Feb 27 20:35:44 zimbra kernel: Normal per-cpu:
    Feb 27 20:35:44 zimbra kernel: cpu 0 hot: low 32, high 96, batch 16
    Feb 27 20:35:44 zimbra kernel: cpu 0 cold: low 0, high 32, batch 16
    Feb 27 20:35:44 zimbra kernel: HighMem per-cpu: empty
    Feb 27 20:35:44 zimbra kernel:
    Feb 27 20:35:44 zimbra kernel: Free pages:         568kB (0kB HighMem)
    Feb 27 20:35:44 zimbra kernel: Active:156 inactive:46434 dirty:0 writeback:0 unstable:0 free:142 slab:22517 mapped:46309 paget
    ables:25835
    Feb 27 20:35:44 zimbra kernel: DMA free:24kB min:24kB low:48kB high:72kB active:368kB inactive:124kB present:16384kB pages_sca
    nned:1246 all_unreclaimable? yes
    Feb 27 20:35:46 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:35:51 zimbra kernel: Normal free:544kB min:596kB low:1192kB high:1788kB active:256kB inactive:185612kB present:37681
    6kB pages_scanned:859980 all_unreclaimable? yes
    Feb 27 20:36:08 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:36:08 zimbra crond(pam_unix)[20964]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:08 zimbra crond(pam_unix)[20960]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scann
    ed:0 all_unreclaimable? no
    Feb 27 20:36:12 zimbra crond(pam_unix)[20974]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra crond(pam_unix)[20979]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra crond(pam_unix)[20970]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra crond(pam_unix)[20982]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra crond(pam_unix)[20977]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:13 zimbra crond(pam_unix)[20956]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:14 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:36:15 zimbra kernel: DMA: 6*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24kB
    Feb 27 20:36:17 zimbra kernel: Normal: 0*4kB 0*8kB 2*16kB 0*32kB 0*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5
    44kB
    Feb 27 20:36:17 zimbra kernel: HighMem: empty
    Feb 27 20:36:18 zimbra kernel: Swap cache: add 307028, delete 307028, find 17081/27658, race 0+19
    Feb 27 20:36:24 zimbra kernel: 0 bounce buffer pages
    Feb 27 20:36:28 zimbra kernel: Free swap:            0kB
    Feb 27 20:36:43 zimbra zimbramon[20986]: 20986:info: 2006-02-27 20:36:19, DISK: : dev: /dev/ida/c0d0p3, mp: /, tot: 16139, ava
    il: 5771
    Feb 27 20:36:31 zimbra kernel: 98300 pages of RAM
    Feb 27 20:36:33 zimbra kernel: 0 pages of HIGHMEM
    Feb 27 20:36:34 zimbra kernel: 1784 reserved pages
    Feb 27 20:36:34 zimbra kernel: 11098 pages shared
    Feb 27 20:36:34 zimbra kernel: 0 pages swap cached
    May it be a problem with my memory ("zimbra kernel: Out of Memory: Killed process 20633 (java).")?
    I know that 386 MB isn't very much, but it always worked right (I only have 3 users on my system at the moment).

    (It looks like you reduced the frequency of status checking in the crontab - is that correct?)
    No, I didn't do anything there.

    Also - did you have any DNS problems around this time?
    Not that I know. All other things worked well at this time.

  6. #6
    Join Date
    Sep 2005
    Posts
    2,103
    Rep Power
    14

    Default

    Yeah, it's the memory killing you.

    I'd suggest the following:

    Disable the logger package:
    zmprov ms SERVERNAME -zimbraServiceEnabled logger

    - Edit the zimbra crontab, remove zmlogprocess and zmgengraphs

    - Cut down the frequency of zmstatuslog and zmdisklog

    Edit amavisd.conf.in and reduce max_servers

    Reduce zimbraLmtpNumThreads (via zmprov) on the server

    Disable the spell server, or reduce the number http processes (in conf/httpd.conf, spareservers, minspareservers, maxspareservers - reduce to 2 or 3)

Similar Threads

  1. QUE Failure
    By tbullock in forum Administrators
    Replies: 31
    Last Post: 07-30-2008, 12:17 PM
  2. Replies: 9
    Last Post: 03-01-2008, 07:21 PM
  3. svn version still won't start
    By kinaole in forum Developers
    Replies: 0
    Last Post: 10-04-2006, 06:47 AM
  4. Logger
    By jholder in forum Installation
    Replies: 24
    Last Post: 03-31-2006, 10:50 AM
  5. Monitoring : Data not yet avalaible
    By s3nz3x in forum Installation
    Replies: 7
    Last Post: 11-30-2005, 06:18 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •