Results 1 to 3 of 3

Thread: java.lang.OutOfMemoryError: unable to create new native thread

Hybrid View

  1. #1
    Join Date
    May 2012
    Posts
    4
    Rep Power
    3

    Default java.lang.OutOfMemoryError: unable to create new native thread

    Hi there,

    Anybody can help? Our mail server suddenly start slowing as i see the cpu is growing around 80-90%. I see the java process consuming high cpu

    top - 08:48:28 up 74 days, 22:41, 3 users, load average: 48.41, 33.43, 26.00
    Tasks: 287 total, 3 running, 284 sleeping, 0 stopped, 0 zombie
    Cpu(s): 93.0%us, 2.3%sy, 0.1%ni, 0.0%id, 0.0%wa, 0.1%hi, 4.5%si, 0.0%st
    Mem: 9804136k total, 7921996k used, 1882140k free, 340884k buffers
    Swap: 9215992k total, 293792k used, 8922200k free, 4066396k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    22286 zimbra 20 0 6680m 1.2g 13m S 484.1 12.9 9:02.26 java
    2851 zimbra 20 0 6836m 1.3g 4556 S 8.6 13.6 1279:01 mysqld
    25028 zimbra 20 0 3682m 18m 7832 S 3.0 0.2 0:00.11 java
    7409 zimbra 20 0 15220 1272 876 S 0.6 0.0 432:09.40 top
    24906 zimbra 20 0 15220 1400 952 R 0.6 0.0 0:00.06 top
    1591 root 20 0 321m 3936 1192 S 0.3 0.0 300:12.33 rsyslogd
    2335 zimbra 30 10 45752 10m 968 S 0.3 0.1 0:16.22 perl
    2479 zimbra 30 10 45764 11m 2036 R 0.3 0.1 0:35.87 perl
    9109 zimbra 20 0 15220 1392 952 S 0.3 0.0 0:03.72 top
    10091 zimbra 20 0 2152m 236m 12m S 0.3 2.5 0:59.75 java
    1 root 20 0 19392 1152 928 S 0.0 0.0 0:01.29 init
    And i checked at mailbox.log, i found this :

    java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:640)
    at com.sun.jndi.ldap.Connection.<init>(Connection.jav a:216)
    at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.jav a:116)
    at com.sun.jndi.ldap.LdapClient.getInstance(LdapClien t.java:1580)
    at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:267 8)
    at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:296)
    at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapC txFactory.java:175)
    at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Ldap CtxFactory.java:193)
    at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstanc e(LdapCtxFactory.java:136)
    at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext (LdapCtxFactory.java:66)
    at javax.naming.spi.NamingManager.getInitialContext(N amingManager.java:667)
    at javax.naming.InitialContext.getDefaultInitCtx(Init ialContext.java:288)
    at javax.naming.InitialContext.init(InitialContext.ja va:223)
    at javax.naming.ldap.InitialLdapContext.<init>(Initia lLdapContext.java:134)
    at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init >(ZimbraLdapContext.java:438)
    at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init >(ZimbraLdapContext.java:409)
    at com.zimbra.cs.account.ldap.LdapProvisioning.getAcc ountByQuery(LdapProvisioning.java:609)
    at com.zimbra.cs.account.ldap.LdapProvisioning.getAcc ountByNameInternal(LdapProvisioning.java:784)
    at com.zimbra.cs.account.ldap.LdapProvisioning.getAcc ountByName(LdapProvisioning.java:763)
    at com.zimbra.cs.account.ldap.LdapProvisioning.getAcc ountByName(LdapProvisioning.java:758)
    at com.zimbra.cs.account.ldap.LdapProvisioning.get(Ld apProvisioning.java:669)
    at com.zimbra.cs.account.ldap.LdapProvisioning.get(Ld apProvisioning.java:654)
    at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.getAddr essStatus(ZimbraLmtpBackend.java:121)
    at com.zimbra.cs.lmtpserver.LmtpHandler.doRCPT(LmtpHa ndler.java:334)
    at com.zimbra.cs.lmtpserver.LmtpHandler.processComman d(LmtpHandler.java:164)
    at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCom mand(TcpLmtpHandler.java:65)
    at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:199)
    at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:140)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    2013-09-16 08:27:32,898 INFO [main] [] soap - Servlet SoapServlet starting up
    2013-09-16 08:27:32,969 INFO [main] [] soap - Adding service AccountService to SoapServlet
    2013-09-16 08:27:33,104 INFO [main] [] soap - Adding service MailService to SoapServlet
    2013-09-16 08:27:33,342 INFO [main] [] soap - Adding service IMService to SoapServlet
    2013-09-16 08:27:33,366 INFO [main] [] misc - version=7.2.0_GA_2669 release=20120410002025 builddate=20120410-0031 buildhost=zre-rhel6-64.eng.vmware.com
    2013-09-16 08:27:33,386 INFO [main] [] misc - LANG environment is set to: C
    2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.home=/opt/zimbra/jdk1.6.0_31/jre
    2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.runtime.version=1.6.0_31-b04
    2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.version=1.6.0_31
    2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.vm.info=mixed mode
    2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.vm.name=Java HotSpot(TM) 64-Bit Server VM
    2013-09-16 08:27:33,386 INFO [main] [] misc - System property java.vm.version=20.6-b01
    2013-09-16 08:27:33,386 INFO [main] [] misc - System property os.arch=amd64
    2013-09-16 08:27:33,387 INFO [main] [] misc - System property os.name=Linux
    2013-09-16 08:27:33,387 INFO [main] [] misc - System property os.version=2.6.32-220.el6.x86_64
    2013-09-16 08:27:33,387 INFO [main] [] misc - System property sun.arch.data.model=64
    2013-09-16 08:27:33,387 INFO [main] [] misc - System property sun.cpu.endian=little
    2013-09-16 08:27:33,387 INFO [main] [] misc - System property sun.cpu.isalist=
    2013-09-16 08:27:33,387 INFO [main] [] misc - System property sun.os.patch.level=unknown
    2013-09-16 08:27:33,444 INFO [main] [] system - Setting mysql connector property: maxActive=100
    2013-09-16 08:27:33,453 INFO [main] [] system - Setting mysql connector property: maxActive=100
    2013-09-16 08:27:34,718 INFO [main] [] sqltrace - Setting slow SQL threshold to 2000ms.
    2013-09-16 08:27:36,101 INFO [main] [] ZimbraHttpConnectionManager - Starting http client idle connection reaper thread for Internal http client connection manager - reaper sleep interval=300000, reaper connection timeout=300000
    2013-09-16 08:27:36,104 INFO [main] [] ZimbraHttpConnectionManager - Starting http client idle connection reaper thread for External http client connection manager - reaper sleep interval=300000, reaper connection timeout=300000
    2013-09-16 08:27:36,160 INFO [IncomingDirectorySweeper] [] store - IncomingDirectorySweeper thread starting
    2013-09-16 08:27:36,173 INFO [main] [] FileDescriptorCache - Loading settings: zimbraMailFileDescriptorCacheSize=1000.
    2013-09-16 08:27:36,686 INFO [main] [] extensions - Loading extensions from /opt/zimbra/lib/ext
    2013-09-16 08:27:36,708 INFO [main] [] extensions - extension com.zimbra.cert.ZimbraCertMgrExt found in /opt/zimbra/lib/ext/com_zimbra_cert_manager/com_zimbra_cert_manager.jar
    2013-09-16 08:27:36,712 INFO [main] [] extensions - extension com.zimbra.cs.versioncheck.VersionCheckExtension found in /opt/zimbra/lib/ext/zimbraadminversioncheck/zimbraadminversioncheck.jar
    2013-09-16 08:27:36,715 INFO [main] [] extensions - extension com.zimbra.cs.nginx.NginxLookupExtension found in /opt/zimbra/lib/ext/nginx-lookup/nginx-lookup.jar
    2013-09-16 08:27:36,732 INFO [main] [] extensions - extension com.zimbra.bp.ZimbraBulkProvisionExt found in /opt/zimbra/lib/ext/com_zimbra_bulkprovision/com_zimbra_bulkprovision.jar
    2013-09-16 08:27:36,737 INFO [main] [] extensions - extension com.zimbra.ldaputils.ZimbraLDAPUtilsExtension found in /opt/zimbra/lib/ext/zimbraldaputils/zimbraldaputils.jar
    2013-09-16 08:27:36,737 INFO [main] [] extensions - extension com.zimbra.clam.ClamScanner found in /opt/zimbra/lib/ext/clamscanner/clamscanner.jar
    2013-09-16 08:27:36,738 INFO [main] [] extensions - Initializing extensions
    2013-09-16 08:27:36,754 INFO [main] [] extensions - Initialized extension com_zimbra_cert_manager: com.zimbra.cert.ZimbraCertMgrExt@com... ension.ZimbraExtensionClassLoader@4799a84f
    2013-09-16 08:27:36,760 INFO [main] [] extensions - Initialized extension versioncheck: com.zimbra.cs.versioncheck.VersionCh...tension.ZimbraExtensionClassLoader@52ec1f9e
    2013-09-16 08:27:36,775 INFO [main] [] extensions - registered handler at /nginx-lookup
    2013-09-16 08:27:36,775 INFO [main] [] extensions - Initialized extension nginx-lookup: com.zimbra.cs.nginx.NginxLookupExten...tension.ZimbraExtensionClassLoader@2804f2ec
    2013-09-16 08:27:36,784 INFO [main] [] extensions - Initialized extension com_zimbra_bulkprovision: com.zimbra.bp.ZimbraBulkProvisionExt...tension.ZimbraExtensionClassLoader@3d2088db
    2013-09-16 08:27:36,791 INFO [main] [] extensions - Initialized extension zimbrasamba: com.zimbra.ldaputils.ZimbraLDAPUtils...tension.ZimbraExtensionClassLoader@1c0b049a
    2013-09-16 08:27:36,796 INFO [main] [] ClamScanner - attachment scan is disabled
    2013-09-16 08:27:36,796 INFO [main] [] extensions - Initialized extension clamscanner: com.zimbra.clam.ClamScanner@com.zimb...nsio n.ZimbraExtensionClassLoader@9f528ab
    2013-09-16 08:27:36,841 INFO [IndexReadersCache-Sweeper] [] IndexReadersCache - IndexReadersCache-Sweeper thread starting
    2013-09-16 08:27:36,876 INFO [main] [] redolog - Starting pre-startup crash recovery
    2013-09-16 08:27:46,821 INFO [main] [] redolog - Redoing 5 uncommitted transactions
    2013-09-16 08:27:46,821 INFO [main] [] redolog - REDOING: txn 1379054272.931115 [CreateMessage] ver=1.31, tstamp=1379294847627, change=511368, mailbox=758, id=238682, rcpt=:API:, rcvDate=1379294847000, shared=false, blobDigest="CPmTYiKeoAjycFhpbuKAMpUReXw=", size=4962, dataLen=4962, folder=5, conv=238479, convFirstMsgId=-1, calItemPartStat=NE, noICal=true, flags=1, tags="", bodyType=1, path=/opt/zimbra/store/incoming/1379294619234-387.msg
    2013-09-16 08:27:47,290 INFO [main] [] index - Initialized Index for mailbox 758 directory: LuceneIndex{mbox=758, dir=org.apache.lucene.store.NIOFSDirectory@/opt/zimbra/index/0/758/index/0 lockFactory=org.apache.lucene.store.SingleInstance LockFactory@65988886} Analyzer=com.zimbra.cs.index.ZimbraAnalyzer@12b498 6
    2013-09-16 08:27:47,291 INFO [main] [] cache - initializing folder and tag caches for mailbox 758
    2013-09-16 08:27:47,352 INFO [main] [] mbxmgr - Mailbox 758 account d891f707-559c-41d0-b43c-5098c1f2f05c LOADED
    2013-09-16 08:27:47,901 INFO [main] [] cache - setting message cache size to 2000
    2013-09-16 08:27:48,151 INFO [main] [] zimlet - Loaded class com.zimbra.cs.zimlet.handler.NANPHandler
    2013-09-16 08:27:48,156 INFO [main] [] zimlet - Loaded class com.zimbra.cs.zimlet.handler.RegexHandler
    2013-09-16 08:27:48,832 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:yqbuagqaeqau
    2013-09-16 08:27:48,832 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:zgbyaguabgbn
    2013-09-16 08:27:48,901 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:ywbzadeamwaz
    2013-09-16 08:27:48,902 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:yqbtageababp
    2013-09-16 08:27:48,950 WARN [main] [] index - Unable to index: message-id:!&!aaaaaaaaaaayaaaa
    2013-09-16 08:27:49,020 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:zgbpagwaygbl
    2013-09-16 08:27:49,020 WARN [main] [] index - Unable to index: x-cr-hashedpuzzle:uablahiabqbv
    Version of zimbra is Release 7.2.0_GA_2669.RHEL6_64_20120410002025 RHEL6_64 FOSS edition

    Is there something need to configure with the java memory?

    [zimbra]$ ps -eaf | grep java
    root 2914 1 0 Jul03 ? 00:00:00 /opt/zimbra/libexec/zmmailboxdmgr start -Dfile.encoding=UTF-8 -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xss256k -Xms1971m -Xmx1971m -Xmn492m
    zimbra 25493 2914 99 08:49 ? 00:09:10 /opt/zimbra/java/bin/java -Dfile.encoding=UTF-8 -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xss256k -Xms1971m -Xmx1971m -Xmn492m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar /opt/zimbra/mailboxd/etc/jetty.properties /opt/zimbra/mailboxd/etc/jetty-setuid.xml /opt/zimbra/mailboxd/etc/jetty.xml
    zimbra 26133 2408 32 08:49 ? 00:00:54 /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner -Djava.io.tmpdir=/opt/zimbra/data/tmp -Dpython.cachedir=/opt/zimbra/data/tmp/jython-cachedir org.python.util.jython /opt/zimbra/libexec/zmconfigd
    zimbra 28487 28486 8 08:52 ? 00:00:02 /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l gs mailbox.domain.com
    This error doesn't happen everytime, it's only happen once a day.
    Sorry for my bad english.

    Regards,

  2. #2
    Join Date
    Jul 2007
    Location
    US
    Posts
    67
    Rep Power
    8

    Default

    It looks like you may need to tune your java options a bit. Here's a useful reference:
    Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki

    In particular, you probably want to be sure to increase -Xms/-Xmx (via mailboxd_java_heap_size for ZCS 7 within reason for your system) and then check other things like -XX:PermSize=196m -XX:MaxPermSize=350m as well.

  3. #3
    Join Date
    May 2007
    Location
    Zimbra
    Posts
    1,285
    Rep Power
    10

    Default

    I would also recommend upgrading to a current release, there are some known issues in 7.2.0 that would be good to resolve.
    Quanah Gibson-Mount
    Server Architect
    Zimbra, Inc
    --------------------
    Zimbra :: the leader in open source messaging and collaboration

Similar Threads

  1. Replies: 0
    Last Post: 02-16-2011, 06:44 PM
  2. [SOLVED] java.lang.OutOfMemoryError: Java heap space
    By frickson in forum Error Reports
    Replies: 3
    Last Post: 05-27-2009, 09:07 PM
  3. java.lang.OutOfMemoryError: Java heap space
    By mayk.backus in forum Error Reports
    Replies: 1
    Last Post: 08-12-2008, 09:07 AM
  4. Replies: 1
    Last Post: 06-06-2007, 07:21 AM
  5. java.lang.OutOfMemoryError: Java heap space
    By igeorg in forum Administrators
    Replies: 20
    Last Post: 08-25-2006, 06:53 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •