Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
vadonka
Posts: 31
Joined: Sat Sep 13, 2014 2:20 am

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby vadonka » Wed Jun 05, 2013 9:53 am

Hi!
Last night im upgraded our company zimbra server from 8.0.3 to 8.0.4.

The upgrade was completed without any error in no time.

However i notice that the 'java' process eat the cpu much more like in the previous version.

The zmlogger process also eat 100% cpu time after a while, plus the web UI noticable slower, for example after i click to a mail folder the browser unresponsive for a few seconds.

Im completely disabled the logger feature, this can help a little but the web UI still slower than the 8.0.3 version.

Also our users report that the new version slower than the previous.
Anyone have a new experience with this new zimbra version?

Thanks!


phoenix
Ambassador
Ambassador
Posts: 26522
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby phoenix » Wed Jun 05, 2013 10:21 am

[quote user="vadonka"]Anyone have a new experience with this new zimbra version?[/QUOTE]Yes, I installed this earlier this week without problems and it's noticeably faster (the web UI) and with no change in cpu usage or load on my system.
You haven't really given much information about your configuration nor the number of users and whether they're using the web UI or a fat client etc., etc. How about giving some useful diagnostic information about the problem?
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
vadonka
Posts: 31
Joined: Sat Sep 13, 2014 2:20 am

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby vadonka » Wed Jun 05, 2013 11:29 am

our server have ~2000 user, 300-400 active in the same time.

im using a dl370 g6 server with 2 xeon (hw virt supported) 3.2ghz processor (16 logical cpu), 96GB ram, and a couple of 1TB SAS hdd in raid5 array running with the ESXi 5.1

the zimbra virtual machine have 8 logical cpu and 8GB ram and 2TB vmdk, swappiness is 0.

the iotop said the IO stress is very low, but the cpu load high referring to the top output.

the filesystem is ext4 with noatime mount option.
this setup was working great with the zimbra 8.0.3.

im reverted to the 8.0.3 (from a VM snapshot), the webui faster again, then i try to upgrade again with the 8.0.4

the result is the same, the whole system slowed down.
currently im completely disabled the logger feature, and reduced the amavis max servers to 3.

this can help, its working properly with this modification. however the 8.0.3 version doesn't have such this issue.
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1668
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby quanah » Wed Jun 05, 2013 11:39 am

So what does it say is taking all the CPU?
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
vadonka
Posts: 31
Joined: Sat Sep 13, 2014 2:20 am

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby vadonka » Wed Jun 05, 2013 1:21 pm

[quote user="quanah"]So what does it say is taking all the CPU?[/QUOTE]
the 'java' process take a significant cpu time and raise the system load higher than 10.0
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1668
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby quanah » Thu Jun 06, 2013 10:55 am

ZCS has multiple Java processes. Saying "java" provides no useful information. Please provide the exact java command taking CPU.
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
gb3
Posts: 14
Joined: Sat Sep 13, 2014 12:46 am

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby gb3 » Fri Jun 07, 2013 3:53 am

I installed Zimbra 8.0.4 on Ubuntu 12.04 LTS (64) with 4GB RAM and zero (0) active users. I have experienced the very same issue. I've attempted reign in the memory loss by adjusting the java heap percentage using the zmlocalconfig tool and turned swappiness=0. At first glance these changes seemed to have been positive but it takes a few hours for the server to crash altogether.
Here is what I'm seeing:
20% of memory is crunching on "java", executing this command:
/opt/zimbra/java/bin/java -client -Xmx256m -Djava.net.preferIPv4Stack=true -D

zimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/e

xt:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner -Djava.io.tmpdir=/o

pt/zimbra/data/tmp -Dpython.cachedir=/opt/zimbra/data/tmp/jython-cachedir org.python.util.jython /opt/

zimbra/libexec/zmconfigd
Another 21% is crunching away at this command:
/opt/zimbra/java/bin/java -Dfile.encoding=UTF-8 -server -Djava.awt.headless=t

rue -Dsun.net.inetaddr.ttl=60 -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseConcMarkSweepGC

-XX:PermSize=128m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -

XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Djava.net.pre

ferIPv4Stack=true -Xss256k -Xms486m -Xmx486m -Xmn121m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djav

a.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.conf

ig=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc

/start.config -jar /opt/zimbra/mailboxd/start.jar OPTIONS=Server,jsp,jmx,resources,websocket,ext,jta,p

lus,rewrite,setuid /opt/zimbra/mailboxd/etc/jetty.properties /opt/zimbra/mailboxd/etc/jetty-setuid.xml

/opt/zimbra/mailboxd/etc/jetty.xml
An aggregate of 24% of memory is working on multiple threads like:
/opt/zimbra/amavisd/sbin/amavisd (master)

OR

/opt/zimbra/amavisd/sbin/amavisd (ch1-avail)

OR

/opt/zimbra/amavisd/sbin/amavisd (virgin child)


The only other software installed on this server is the zExtras suite.
Eventually the server will stop responding and you see various messages like this (OMMing):

Out of memory: Kill process 12222 (java) score 291 or sacrifice child

Killed process 12222 (java) total-vm:5441712kB, anon-rss:669904kB, file-rss:0kB

Out of memory: Kill process 22843 (java) score 148 or sacrifice child

Killed process 22843 (java) total-vm:2724916kB, anon-rss:341628kB, file-rss:0kB

Out of memory: Kill process 8108 (java) score 120 or sacrifice child

Killed process 8108 (java) total-vm:2127264kB, anon-rss:276580kB, file-rss:0kB

Out of memory: Kill process 8254 (java) score 121 or sacrifice child

INFO: rcu_sched self-detected stall on CPU { 0} (t=60000 jiffies g=321307 c=321306 q=5275)
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1668
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby quanah » Fri Jun 07, 2013 9:21 am

[quote user="gb3"]

The only other software installed on this server is the zExtras suite.

[/QUOTE]
Please confirm whether or not you see this issue if you do not install the zExtras suite. Past issues show that zExtras generally is incompatible with upgrades for some period of time, while they fix their suite to account for changes between releases. I can confirm I do not see any such issue like this on ZCS 8.0.4 without Zextras, running Ubuntu12.
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
gb3
Posts: 14
Joined: Sat Sep 13, 2014 12:46 am

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby gb3 » Fri Jun 07, 2013 2:26 pm

[quote user="quanah"]Please confirm whether or not you see this issue if you do not install the zExtras suite. [/QUOTE]

I can confirm that even before I installed the zExtras Core and Zimlet (1.8.7) the same issue occurred. However I did notice that when I ran:
zmlocalconfig mailboxd_java_heap_memory_percent 


it ruturned the following warning:

Warning: null valued key 'mailboxd_java_heap_memory_percent' 

After I ran:

zmlocalconfig -e mailboxd_java_heap_memory_percent=20


and

sudo sysctl vm.swappiness=0

This issue went away.

Do you know how Zimbra configures the mailboxd_java_heap_memory_percent by default?
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1668
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

Postby quanah » Fri Jun 07, 2013 2:43 pm

That isn't a valid local config key, thus the error.
The two localconfig errors that relate to the java heap are:


mailboxd_java_heap_new_size_percent = 25

mailboxd_java_heap_size = 742

--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 12 guests