zimbra hanging - java process using 100%
We have been having a very rare issue. This problem has occured during medium load to off-time so it isn't related to load. What happens is a java process uses 100% of a single core and then zimbra will "hang". You can't access your email through a client or the webgui. However, looking at the logs it appears zimbra was fine. No errors or timeouts and email looks like it is being received, sent, queued, and scanned fine.
I was able to grab a ps aux while it was happening. This is the offending process...
Here are all the process running USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root - Pastebin.com
zimbra 18050 8.4 3.1 15518388 3100980 ? Sl Nov20 3676:56 /opt/zimbra/java/bin/java -Dfile.encoding=UTF-8 -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Djava.net.preferIPv4Stack=true -Xss256k -Xms2048m -Xmx2048m -Xmn512m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar OPTIONS=Server,jsp,jmx,resources,websocket,ext,jta,plus,rewrite,setuid /opt/zimbra/mailboxd/etc/jetty.properties /opt/zimbra/mailboxd/etc/jetty-setuid.xml /opt/zimbra/mailboxd/etc/jetty.xml
The problem usualy corrects itself if you wait long enough. It can take a couple minutes to 15 min. Restarting zimbra fixes the problem right away.
Any ideas? I know it is not much to go on but there is nothing in the logs and this problem only pops up a couple times a month so it is not a common occurance and like I said above it happens at random times.