Zimbra 8.0.4 processes consumes too much cpu time and web UI slower.

[vadonka]

Hi!
Last night im upgraded our company zimbra server from 8.0.3 to 8.0.4.

The upgrade was completed without any error in no time.

However i notice that the 'java' process eat the cpu much more like in the previous version.

The zmlogger process also eat 100% cpu time after a while, plus the web UI noticable slower, for example after i click to a mail folder the browser unresponsive for a few seconds.

Im completely disabled the logger feature, this can help a little but the web UI still slower than the 8.0.3 version.

Also our users report that the new version slower than the previous.
Anyone have a new experience with this new zimbra version?

Thanks!

[phoenix]

[quote user="vadonka"]Anyone have a new experience with this new zimbra version?[/QUOTE]Yes, I installed this earlier this week without problems and it's noticeably faster (the web UI) and with no change in cpu usage or load on my system.
You haven't really given much information about your configuration nor the number of users and whether they're using the web UI or a fat client etc., etc. How about giving some useful diagnostic information about the problem?

[vadonka]

our server have ~2000 user, 300-400 active in the same time.

im using a dl370 g6 server with 2 xeon (hw virt supported) 3.2ghz processor (16 logical cpu), 96GB ram, and a couple of 1TB SAS hdd in raid5 array running with the ESXi 5.1

the zimbra virtual machine have 8 logical cpu and 8GB ram and 2TB vmdk, swappiness is 0.

the iotop said the IO stress is very low, but the cpu load high referring to the top output.

the filesystem is ext4 with noatime mount option.
this setup was working great with the zimbra 8.0.3.

im reverted to the 8.0.3 (from a VM snapshot), the webui faster again, then i try to upgrade again with the 8.0.4

the result is the same, the whole system slowed down.
currently im completely disabled the logger feature, and reduced the amavis max servers to 3.

this can help, its working properly with this modification. however the 8.0.3 version doesn't have such this issue.

[quanah]

So what does it say is taking all the CPU?

[vadonka]

[quote user="quanah"]So what does it say is taking all the CPU?[/QUOTE]
the 'java' process take a significant cpu time and raise the system load higher than 10.0

[quanah]

ZCS has multiple Java processes. Saying "java" provides no useful information. Please provide the exact java command taking CPU.

[gb3]

I installed Zimbra 8.0.4 on Ubuntu 12.04 LTS (64) with 4GB RAM and zero (0) active users. I have experienced the very same issue. I've attempted reign in the memory loss by adjusting the java heap percentage using the zmlocalconfig tool and turned swappiness=0. At first glance these changes seemed to have been positive but it takes a few hours for the server to crash altogether.
Here is what I'm seeing:
20% of memory is crunching on "java", executing this command:
/opt/zimbra/java/bin/java -client -Xmx256m -Djava.net.preferIPv4Stack=true -D

zimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/e

xt:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner -Djava.io.tmpdir=/o

pt/zimbra/data/tmp -Dpython.cachedir=/opt/zimbra/data/tmp/jython-cachedir org.python.util.jython /opt/

zimbra/libexec/zmconfigd
Another 21% is crunching away at this command:
/opt/zimbra/java/bin/java -Dfile.encoding=UTF-8 -server -Djava.awt.headless=t

rue -Dsun.net.inetaddr.ttl=60 -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseConcMarkSweepGC

-XX:PermSize=128m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -

XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Djava.net.pre

ferIPv4Stack=true -Xss256k -Xms486m -Xmx486m -Xmn121m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djav

a.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.conf

ig=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc

/start.config -jar /opt/zimbra/mailboxd/start.jar OPTIONS=Server,jsp,jmx,resources,websocket,ext,jta,p

lus,rewrite,setuid /opt/zimbra/mailboxd/etc/jetty.properties /opt/zimbra/mailboxd/etc/jetty-setuid.xml

/opt/zimbra/mailboxd/etc/jetty.xml
An aggregate of 24% of memory is working on multiple threads like:
/opt/zimbra/amavisd/sbin/amavisd (master)

OR

/opt/zimbra/amavisd/sbin/amavisd (ch1-avail)

OR

/opt/zimbra/amavisd/sbin/amavisd (virgin child)


The only other software installed on this server is the zExtras suite.
Eventually the server will stop responding and you see various messages like this (OMMing):

Out of memory: Kill process 12222 (java) score 291 or sacrifice child

Killed process 12222 (java) total-vm:5441712kB, anon-rss:669904kB, file-rss:0kB

Out of memory: Kill process 22843 (java) score 148 or sacrifice child

Killed process 22843 (java) total-vm:2724916kB, anon-rss:341628kB, file-rss:0kB

Out of memory: Kill process 8108 (java) score 120 or sacrifice child

Killed process 8108 (java) total-vm:2127264kB, anon-rss:276580kB, file-rss:0kB

Out of memory: Kill process 8254 (java) score 121 or sacrifice child

INFO: rcu_sched self-detected stall on CPU { 0} (t=60000 jiffies g=321307 c=321306 q=5275)

[quanah]

[quote user="gb3"]

The only other software installed on this server is the zExtras suite.

[/QUOTE]
Please confirm whether or not you see this issue if you do not install the zExtras suite. Past issues show that zExtras generally is incompatible with upgrades for some period of time, while they fix their suite to account for changes between releases. I can confirm I do not see any such issue like this on ZCS 8.0.4 without Zextras, running Ubuntu12.

[gb3]

[quote user="quanah"]Please confirm whether or not you see this issue if you do not install the zExtras suite. [/QUOTE]

I can confirm that even before I installed the zExtras Core and Zimlet (1.8.7) the same issue occurred. However I did notice that when I ran:

zmlocalconfig mailboxd_java_heap_memory_percent 

it ruturned the following warning:

Warning: null valued key 'mailboxd_java_heap_memory_percent' 

After I ran:

zmlocalconfig -e mailboxd_java_heap_memory_percent=20

and

sudo sysctl vm.swappiness=0

This issue went away.

Do you know how Zimbra configures the mailboxd_java_heap_memory_percent by default?

[quanah]

That isn't a valid local config key, thus the error.
The two localconfig errors that relate to the java heap are:

mailboxd_java_heap_new_size_percent = 25

mailboxd_java_heap_size = 742