Hi there,

Anybody can help? Our mail server suddenly start slowing as i see the cpu is growing around 80-90%. I see the java process consuming high cpu

top - 08:48:28 up 74 days, 22:41, 3 users, load average: 48.41, 33.43, 26.00
Tasks: 287 total, 3 running, 284 sleeping, 0 stopped, 0 zombie
Cpu(s): 93.0%us, 2.3%sy, 0.1%ni, 0.0%id, 0.0%wa, 0.1%hi, 4.5%si, 0.0%st
Mem: 9804136k total, 7921996k used, 1882140k free, 340884k buffers
Swap: 9215992k total, 293792k used, 8922200k free, 4066396k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22286 zimbra 20 0 6680m 1.2g 13m S 484.1 12.9 9:02.26 java
2851 zimbra 20 0 6836m 1.3g 4556 S 8.6 13.6 1279:01 mysqld
25028 zimbra 20 0 3682m 18m 7832 S 3.0 0.2 0:00.11 java
7409 zimbra 20 0 15220 1272 876 S 0.6 0.0 432:09.40 top
24906 zimbra 20 0 15220 1400 952 R 0.6 0.0 0:00.06 top
1591 root 20 0 321m 3936 1192 S 0.3 0.0 300:12.33 rsyslogd
2335 zimbra 30 10 45752 10m 968 S 0.3 0.1 0:16.22 perl
2479 zimbra 30 10 45764 11m 2036 R 0.3 0.1 0:35.87 perl
9109 zimbra 20 0 15220 1392 952 S 0.3 0.0 0:03.72 top
10091 zimbra 20 0 2152m 236m 12m S 0.3 2.5 0:59.75 java
1 root 20 0 19392 1152 928 S 0.0 0.0 0:01.29 init
And i checked at mailbox.log, i found this :

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at com.sun.jndi.ldap.Connection.<init>(Connection.jav a:216)
at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.jav a:116)
at com.sun.jndi.ldap.LdapClient.getInstance(LdapClien t.java:1580)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:267 8)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:296)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapC txFactory.java:175)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Ldap CtxFactory.java:193)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstanc e(LdapCtxFactory.java:136)
at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext (LdapCtxFactory.java:66)
at javax.naming.spi.NamingManager.getInitialContext(N amingManager.java:667)
at javax.naming.InitialContext.getDefaultInitCtx(Init ialContext.java:288)
at javax.naming.InitialContext.init(InitialContext.ja va:223)
at javax.naming.ldap.InitialLdapContext.<init>(Initia lLdapContext.java:134)
at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init >(ZimbraLdapContext.java:438)
at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init >(ZimbraLdapContext.java:409)
at com.zimbra.cs.account.ldap.LdapProvisioning.getAcc ountByQuery(LdapProvisioning.java:609)
at com.zimbra.cs.account.ldap.LdapProvisioning.getAcc ountByNameInternal(LdapProvisioning.java:784)
at com.zimbra.cs.account.ldap.LdapProvisioning.getAcc ountByName(LdapProvisioning.java:763)
at com.zimbra.cs.account.ldap.LdapProvisioning.getAcc ountByName(LdapProvisioning.java:758)
at com.zimbra.cs.account.ldap.LdapProvisioning.get(Ld apProvisioning.java:669)
at com.zimbra.cs.account.ldap.LdapProvisioning.get(Ld apProvisioning.java:654)
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.getAddr essStatus(ZimbraLmtpBackend.java:121)
at com.zimbra.cs.lmtpserver.LmtpHandler.doRCPT(LmtpHa ndler.java:334)
at com.zimbra.cs.lmtpserver.LmtpHandler.processComman d(LmtpHandler.java:164)
at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCom mand(TcpLmtpHandler.java:65)
at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:199)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:140)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013-09-16 08:27:32,898 INFO [main] [] soap - Servlet SoapServlet starting up
2013-09-16 08:27:32,969 INFO [main] [] soap - Adding service AccountService to SoapServlet
2013-09-16 08:27:33,104 INFO [main] [] soap - Adding service MailService to SoapServlet
2013-09-16 08:27:33,342 INFO [main] [] soap - Adding service IMService to SoapServlet
2013-09-16 08:27:33,366 INFO [main] [] misc - version=7.2.0_GA_2669 release=20120410002025 builddate=20120410-0031 buildhost=zre-rhel6-64.eng.vmware.com
2013-09-16 08:27:33,386 INFO [main] [] misc - LANG environment is set to: C
2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.home=/opt/zimbra/jdk1.6.0_31/jre
2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.runtime.version=1.6.0_31-b04
2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.version=1.6.0_31
2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.vm.info=mixed mode
2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.vm.name=Java HotSpot(TM) 64-Bit Server VM
2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.vm.version=20.6-b01
2013-09-16 08:27:33,386 INFO [main] [] misc - System property os.arch=amd64
2013-09-16 08:27:33,387 INFO [main] [] misc - System property os.name=Linux
2013-09-16 08:27:33,387 INFO [main] [] misc - System property os.version=2.6.32-220.el6.x86_64
2013-09-16 08:27:33,387 INFO [main] [] misc - System property sun.arch.data.model=64
2013-09-16 08:27:33,387 INFO [main] [] misc - System property sun.cpu.endian=little
2013-09-16 08:27:33,387 INFO [main] [] misc - System property sun.cpu.isalist=
2013-09-16 08:27:33,387 INFO [main] [] misc - System property sun.os.patch.level=unknown
2013-09-16 08:27:33,444 INFO [main] [] system - Setting mysql connector property: maxActive=100
2013-09-16 08:27:33,453 INFO [main] [] system - Setting mysql connector property: maxActive=100
2013-09-16 08:27:34,718 INFO [main] [] sqltrace - Setting slow SQL threshold to 2000ms.
2013-09-16 08:27:36,101 INFO [main] [] ZimbraHttpConnectionManager - Starting http client idle connection reaper thread for Internal http client connection manager - reaper sleep interval=300000, reaper connection timeout=300000
2013-09-16 08:27:36,104 INFO [main] [] ZimbraHttpConnectionManager - Starting http client idle connection reaper thread for External http client connection manager - reaper sleep interval=300000, reaper connection timeout=300000
2013-09-16 08:27:36,160 INFO [IncomingDirectorySweeper] [] store - IncomingDirectorySweeper thread starting
2013-09-16 08:27:36,173 INFO [main] [] FileDescriptorCache - Loading settings: zimbraMailFileDescriptorCacheSize=1000.
2013-09-16 08:27:36,686 INFO [main] [] extensions - Loading extensions from /opt/zimbra/lib/ext
2013-09-16 08:27:36,708 INFO [main] [] extensions - extension com.zimbra.cert.ZimbraCertMgrExt found in /opt/zimbra/lib/ext/com_zimbra_cert_manager/com_zimbra_cert_manager.jar
2013-09-16 08:27:36,712 INFO [main] [] extensions - extension com.zimbra.cs.versioncheck.VersionCheckExtension found in /opt/zimbra/lib/ext/zimbraadminversioncheck/zimbraadminversioncheck.jar
2013-09-16 08:27:36,715 INFO [main] [] extensions - extension com.zimbra.cs.nginx.NginxLookupExtension found in /opt/zimbra/lib/ext/nginx-lookup/nginx-lookup.jar
2013-09-16 08:27:36,732 INFO [main] [] extensions - extension com.zimbra.bp.ZimbraBulkProvisionExt found in /opt/zimbra/lib/ext/com_zimbra_bulkprovision/com_zimbra_bulkprovision.jar
2013-09-16 08:27:36,737 INFO [main] [] extensions - extension com.zimbra.ldaputils.ZimbraLDAPUtilsExtension found in /opt/zimbra/lib/ext/zimbraldaputils/zimbraldaputils.jar
2013-09-16 08:27:36,737 INFO [main] [] extensions - extension com.zimbra.clam.ClamScanner found in /opt/zimbra/lib/ext/clamscanner/clamscanner.jar
2013-09-16 08:27:36,738 INFO [main] [] extensions - Initializing extensions
2013-09-16 08:27:36,754 INFO [main] [] extensions - Initialized extension com_zimbra_cert_manager: com.zimbra.cert.ZimbraCertMgrExt@com... ension.ZimbraExtensionClassLoader@4799a84f
2013-09-16 08:27:36,760 INFO [main] [] extensions - Initialized extension versioncheck: com.zimbra.cs.versioncheck.VersionCh...tension.ZimbraExtensionClassLoader@52ec1f9e
2013-09-16 08:27:36,775 INFO [main] [] extensions - registered handler at /nginx-lookup
2013-09-16 08:27:36,775 INFO [main] [] extensions - Initialized extension nginx-lookup: com.zimbra.cs.nginx.NginxLookupExten...tension.ZimbraExtensionClassLoader@2804f2ec
2013-09-16 08:27:36,784 INFO [main] [] extensions - Initialized extension com_zimbra_bulkprovision: com.zimbra.bp.ZimbraBulkProvisionExt...tension.ZimbraExtensionClassLoader@3d2088db
2013-09-16 08:27:36,791 INFO [main] [] extensions - Initialized extension zimbrasamba: com.zimbra.ldaputils.ZimbraLDAPUtils...tension.ZimbraExtensionClassLoader@1c0b049a
2013-09-16 08:27:36,796 INFO [main] [] ClamScanner - attachment scan is disabled
2013-09-16 08:27:36,796 INFO [main] [] extensions - Initialized extension clamscanner: com.zimbra.clam.ClamScanner@com.zimb...nsio n.ZimbraExtensionClassLoader@9f528ab
2013-09-16 08:27:36,841 INFO [IndexReadersCache-Sweeper] [] IndexReadersCache - IndexReadersCache-Sweeper thread starting
2013-09-16 08:27:36,876 INFO [main] [] redolog - Starting pre-startup crash recovery
2013-09-16 08:27:46,821 INFO [main] [] redolog - Redoing 5 uncommitted transactions
2013-09-16 08:27:46,821 INFO [main] [] redolog - REDOING: txn 1379054272.931115 [CreateMessage] ver=1.31, tstamp=1379294847627, change=511368, mailbox=758, id=238682, rcpt=:API:, rcvDate=1379294847000, shared=false, blobDigest="CPmTYiKeoAjycFhpbuKAMpUReXw=", size=4962, dataLen=4962, folder=5, conv=238479, convFirstMsgId=-1, calItemPartStat=NE, noICal=true, flags=1, tags="", bodyType=1, path=/opt/zimbra/store/incoming/1379294619234-387.msg
2013-09-16 08:27:47,290 INFO [main] [] index - Initialized Index for mailbox 758 directory: LuceneIndex{mbox=758, dir=org.apache.lucene.store.NIOFSDirectory@/opt/zimbra/index/0/758/index/0 lockFactory=org.apache.lucene.store.SingleInstance LockFactory@65988886} Analyzer=com.zimbra.cs.index.ZimbraAnalyzer@12b498 6
2013-09-16 08:27:47,291 INFO [main] [] cache - initializing folder and tag caches for mailbox 758
2013-09-16 08:27:47,352 INFO [main] [] mbxmgr - Mailbox 758 account d891f707-559c-41d0-b43c-5098c1f2f05c LOADED
2013-09-16 08:27:47,901 INFO [main] [] cache - setting message cache size to 2000
2013-09-16 08:27:48,151 INFO [main] [] zimlet - Loaded class com.zimbra.cs.zimlet.handler.NANPHandler
2013-09-16 08:27:48,156 INFO [main] [] zimlet - Loaded class com.zimbra.cs.zimlet.handler.RegexHandler
2013-09-16 08:27:48,832 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:yqbuagqaeqau
2013-09-16 08:27:48,832 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:zgbyaguabgbn
2013-09-16 08:27:48,901 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:ywbzadeamwaz
2013-09-16 08:27:48,902 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:yqbtageababp
2013-09-16 08:27:48,950 WARN [main] [] index - Unable to index: message-id:!&!aaaaaaaaaaayaaaa
2013-09-16 08:27:49,020 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:zgbpagwaygbl
2013-09-16 08:27:49,020 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:uablahiabqbv
Version of zimbra is Release 7.2.0_GA_2669.RHEL6_64_20120410002025 RHEL6_64 FOSS edition

Is there something need to configure with the java memory?

[zimbra]$ ps -eaf | grep java
root 2914 1 0 Jul03 ? 00:00:00 /opt/zimbra/libexec/zmmailboxdmgr start -Dfile.encoding=UTF-8 -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xss256k -Xms1971m -Xmx1971m -Xmn492m
zimbra 25493 2914 99 08:49 ? 00:09:10 /opt/zimbra/java/bin/java -Dfile.encoding=UTF-8 -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xss256k -Xms1971m -Xmx1971m -Xmn492m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar /opt/zimbra/mailboxd/etc/jetty.properties /opt/zimbra/mailboxd/etc/jetty-setuid.xml /opt/zimbra/mailboxd/etc/jetty.xml
zimbra 26133 2408 32 08:49 ? 00:00:54 /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner -Djava.io.tmpdir=/opt/zimbra/data/tmp -Dpython.cachedir=/opt/zimbra/data/tmp/jython-cachedir org.python.util.jython /opt/zimbra/libexec/zmconfigd
zimbra 28487 28486 8 08:52 ? 00:00:02 /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l gs mailbox.domain.com
This error doesn't happen everytime, it's only happen once a day.
Sorry for my bad english.

Regards,