Results 1 to 6 of 6

Thread: tomcat/catalina frozen with high cpu

  1. #1
    Join Date
    Feb 2006
    Posts
    27
    Rep Power
    9

    Default tomcat/catalina frozen with high cpu

    We've had an intermittant problem with tomcat locking up with high cpu usage. I've seen other threads about similar problems but none seems exactly the same. Sometimes it happens 3-4 times a day and sometimes, we don't have a problem for 3 or more days at a time. We can't find a direct link to specific user activity nor do we see any smoking gun in the logs. I combed the forum and found a post by KevinH about getting a stack trace using 'kill -QUIT ' which did produce something very interesting about a deadlock.

    From tomcat.out as a result of 'kill -QUIT':
    Found one Java-level deadlock:
    =============================
    "LmtpServer-85":
    waiting to lock monitor 0x092a58cc (object 0xa55d9288, a java.lang.Object),
    which is held by "IMAPS-993-W1261"
    "IMAPS-993-W1261":
    waiting to lock monitor 0x092a588c (object 0xa22aa078, a
    com.zimbra.cs.mailbox.Mailbox),
    which is held by "LmtpServer-85"
    Entries relating to the two transctions in /opt/zimbra/log/zimbra.log:
    2006-06-05 10:23:02,092 INFO [IndexWritersSweeper] [] MailboxIndex - open index
    writers sweep: before=6, closed=0, after=6 (0ms) 2006-06-05 10:23:06,596 INFO [IMAPS-993-W1261] [cid=191;ip=XX.XX.XX.29;] imap
    - exception occurred during read task java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher. java:21)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java :233) at sun.nio.ch.IOUtil.read(IOUtil.java:206)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImp l.java:207) at com.zimbra.cs.ozserver.OzConnection$ReadTask.doTas k(OzConnection.java:428
    ) at com.zimbra.cs.ozserver.OzConnection$Task.run(OzCon nection.java:344)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:595) 2006-06-05 10:23:06,597 INFO [IMAPS-993-W1261] [cid=191;ip=XX.XX.XX.29;] imap
    - channel cleanup - CLEANUP 2006-06-05 10:23:21,516 INFO [IMAPS-993-W1261] [cid=196;ip=XX.XX.XX.29;] imap
    - connected

    2006-06-05 10:23:32,555 INFO [LmtpServer-85] [] LmtpHandler - [XX.XX.XX.199] co
    nnected 2006-06-05 10:23:32,659 INFO [LmtpServer-85] [name=USER1@XXXXX.com;] Fil
    eBlobStore - Stored size=6014 wrote=6014 path=/opt/zimbra/store/incoming/1149436431144-133.msg vol=1 digest=0uqCR,nwoNG86IWb,UfFbfU0Yuk=
    2006-06-05 10:23:32,680 INFO [LmtpServer-85] [name=USER1@XXXXX.com;] FileBlobStore - Linked id=2801 mbox=7 oldpath=/opt/zimbra/store/incoming/1149436431
    144-133.msg newpath=/opt/zimbra/store/0/7/msg/0/2801-6670.msg 2006-06-05 10:23:32,700 INFO [LmtpServer-85] [name=USER1@XXXXX.com;] mai
    lbox - Added message id=2801 digest=0uqCR,nwoNG86IWb,UfFbfU0Yuk= mailbox=7 rcpt=USER1@XXXXX.com
    2006-06-05 10:23:32,703 INFO [LmtpServer-85] [name=USER1@XXXXX.com;] FileBlobStore - Linked id=2802 mbox=7 oldpath=/opt/zimbra/store/0/7/msg/0/2801-6670.msg newpath=/opt/zimbra/store/0/7/msg/0/2802-6671.msg
    2006-06-05 10:23:32,721 INFO [LmtpServer-85] [name=USER1@XXXXX.com;] mailbox - Added message id=2802 digest=0uqCR,nwoNG86IWb,UfFbfU0Yuk= mailbox=7 rcpt=USER1@XXXXX.com
    2006-06-05 10:23:32,727 INFO [LmtpServer-85] [name=USER2@XXXXX.com;] FileBlobStore - Linked id=8219 mbox=6 oldpath=/opt/zimbra/store/incoming/1149436431144-133.msg newpath=/opt/zimbra/store/0/6/msg/2/8219-10467.msg
    2006-06-05 10:23:32,741 INFO [LmtpServer-85] [name=USER2@XXXXX.com;] mailbox - Added message id=8219 digest=0uqCR,nwoNG86IWb,UfFbfU0Yuk= mailbox=6 rcpt=USER2@XXXXX.com
    2006-06-05 10:23:32,744 INFO [LmtpServer-85] [name=USER3@XXXXX.com;] FileBlobStore - Linked id=708 mbox=14 oldpath=/opt/zimbra/store/incoming/1149436431144-133.msg newpath=/opt/zimbra/store/0/14/msg/0/708-1313.msg
    2006-06-05 10:23:32,795 INFO [LmtpServer-85] [name=USER3@XXXXX.com;] mailbox - Added message id=708 digest=0uqCR,nwoNG86IWb,UfFbfU0Yuk= mailbox=14 rcpt=USER3@XXXXX.com
    2006-06-05 10:23:32,799 INFO [LmtpServer-85] [name=USER4@XXXXX.com;] FileBlobStore - Linked id=4461 mbox=2 oldpath=/opt/zimbra/store/incoming/1149436431144-133.msg newpath=/opt/zimbra/store/0/2/msg/1/4461-9219.msg
    2006-06-05 10:23:32,812 INFO [LmtpServer-85] [name=USER4@XXXXX.com;] mailbox - Added message id=4461 digest=0uqCR,nwoNG86IWb,UfFbfU0Yuk= mailbox=2 rcpt=USER4@XXXXX.com
    2006-06-05 10:23:32,813 INFO [LmtpServer-85] [name=USER4@XXXXX.com;] mailbox - outofoffice not sent (not direct) mid=4461 rcpt='USER4@XXXXX.com' dest='tmallory@XXXXX.com'
    2006-06-05 10:23:32,821 INFO [LmtpServer-85] [name=USER5@XXXXX.com;] FileBlobStore - Linked id=2883 mbox=15 oldpath=/opt/zimbra/store/incoming/1149436431144-133.msg newpath=/opt/zimbra/store/0/15/msg/0/2883-5601.msg
    2006-06-05 10:23:32,857 INFO [LmtpServer-85] [name=USER5@XXXXX.com;] mailbox - Added message id=2883 digest=0uqCR,nwoNG86IWb,UfFbfU0Yuk= mailbox=15 rcpt=USER5@XXXXX.com
    2006-06-05 10:23:32,860 INFO [LmtpServer-85] [name=USER5@XXXXX.com;] FileBlobStore - Linked id=2884 mbox=15 oldpath=/opt/zimbra/store/0/15/msg/0/2883-5601.msg newpath=/opt/zimbra/store/0/15/msg/0/2884-5602.msg
    Can anyone explain what is going on and if there is a fix for this? Let me know if any more info is needed.

    Thanks.

  2. #2
    Join Date
    Feb 2006
    Posts
    27
    Rep Power
    9

    Default

    Forgot to mention that new IMAP, POP, and web connections all fail when we have this problem. Existing connections seem to continue to function (such as IDLE IMAP connections) but once the connection is broken (like restarting the client) then the user won't be able to re-establish. Doing a 'tomcat restart' brings everything back.

  3. #3
    Join Date
    Aug 2005
    Location
    San Mateo, CA
    Posts
    4,789
    Rep Power
    19

    Default

    Are you running with NIO IMAP enabled(looks like it from the stack traces)? If so I bet this is the problem. There are known dead-locks in that code which is the reason it's disabled by default.
    Looking for new beta users -> Co-Founder of Acompli. Previously worked at Zimbra (and Yahoo! & VMware) since 2005.

  4. #4
    Join Date
    Feb 2006
    Posts
    27
    Rep Power
    9

    Default

    Thanks Kevin. Based on your response, I looked up the other thread that talks about NIO IMAP which we did turn on because we need IDLE support.

    http://www.zimbra.com/forums/showthread.php?t=2084

    I assume there's no way to enable IDLE but not use NIO. We have a relatively small deployment right now so number of threads is not an issue but we do need the IDLE feature. Please let me know if there's anyway I can work around NIO but still have IDLE support.

    Thanks.
    Last edited by eddie; 06-05-2006 at 05:42 PM.

  5. #5
    Join Date
    Aug 2005
    Location
    San Mateo, CA
    Posts
    4,789
    Rep Power
    19

    Default

    There is no way to get IMAP IDLE today without being exposed to the deadlock bugs. We've removed IDLE from IMAP in both cases but if you turn on nio you'll get IDLE but be exposed to the deadlocks. So the only safe way to proceed is to wait for the NIO fixes that should be in a future release.
    Looking for new beta users -> Co-Founder of Acompli. Previously worked at Zimbra (and Yahoo! & VMware) since 2005.

  6. #6
    Join Date
    Sep 2006
    Location
    London
    Posts
    32
    Rep Power
    9

    Default NIO fixes

    Has NIO been fixed in the 4.0 release?

Similar Threads

  1. High CPU usage on server and client
    By jack-Z in forum Administrators
    Replies: 10
    Last Post: 01-07-2009, 10:12 PM
  2. High CPU cycle
    By chenthil in forum Administrators
    Replies: 10
    Last Post: 02-08-2007, 11:38 AM
  3. HIGH CPU use
    By brwatters in forum Administrators
    Replies: 3
    Last Post: 07-05-2006, 09:47 AM
  4. 3.1 on FC4 problems
    By cohnhead in forum Installation
    Replies: 8
    Last Post: 05-26-2006, 12:16 PM
  5. Zimbra server crashed
    By goetzi in forum Administrators
    Replies: 6
    Last Post: 03-25-2006, 01:00 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •