High cpu load for java process after upgrade to 8.8.6

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
sarge
Posts: 3
Joined: Sun Mar 18, 2018 3:44 pm
Location: Moscow

High cpu load for java process after upgrade to 8.8.6

Postby sarge » Sun Mar 18, 2018 4:17 pm

HI,
After upgrade from 8.6.0 to Zimbra 8.8.6_GA_1906(Community), there were problems with CPU.
My system:
VM Centos 6 x64 on VMWare esxi 5.5
4vCPU and 16gB RAM
2 domains, and 50 mailboxes

Problem:
top - 19:00:40 up 18:27, 1 user, load average: 0.25, 0.17, 0.11
Tasks: 202 total, 1 running, 201 sleeping, 0 stopped, 0 zombie
Cpu(s): 38.8%us, 0.2%sy, 0.0%ni, 81.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16334684k total, 7037212k used, 9297472k free, 225644k buffers
Swap: 18579448k total, 0k used, 18579448k free, 721476k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22238 zimbra 20 0 7942m 2.8g 21m S 95.1 18.0 45:00.50 java

This happens intermittently, but in the middle of the day, I have to reboot the server.

Config:
$ zmlocalconfig | grep java_heap
imapd_java_heap_new_size_percent = ${mailboxd_java_heap_new_size_percent}
imapd_java_heap_size =
mailboxd_java_heap_new_size_percent = 25
mailboxd_java_heap_size = 4096

I do not know what's going on :-(


User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 239
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: Release 8.7.11_GA_1854.RHEL6_64_201
Contact:

Re: High cpu load for java process after upgrade to 8.8.6

Postby JDunphy » Sun Mar 18, 2018 5:44 pm

A few other threads on this last month... Do any of these sound like what you have observed? https://forums.zimbra.org/viewtopic.php?f=15&t=63579&hilit=high+cpu&start=10

https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.6_HotFix Specifically this: Bug 108414. Double check the logs.
User avatar
sarge
Posts: 3
Joined: Sun Mar 18, 2018 3:44 pm
Location: Moscow

Re: High cpu load for java process after upgrade to 8.8.6

Postby sarge » Sun Mar 18, 2018 6:17 pm

Thank you,
Before I write here, I installed a patch - https://wiki.zimbra.com/wiki/Zimbra_Rel ... 8.6_HotFix

>Specifically this: Bug 108414. Double check the logs.
Checked the logs, did not see the error "mailbox - Failed to lock mailbox"

The CPU is still unevenly loaded :-((
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 239
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: Release 8.7.11_GA_1854.RHEL6_64_201
Contact:

Re: High cpu load for java process after upgrade to 8.8.6

Postby JDunphy » Sun Mar 18, 2018 7:42 pm

sarge wrote:Thank you,
Before I write here, I installed a patch - https://wiki.zimbra.com/wiki/Zimbra_Rel ... 8.6_HotFix

I figured as much but thought I would get that out of the way just in case.

You said you have to restart. Are you doing that because your users can't login or does the system become progressively unresponsive and slow? Is it all the clients or just the MUA's like outlook, thunderbird, etc. Does the web interface continue to function? You might also focus on the cmd line stuff (ie. stats) kicked off from cron to see if they are making the problem worse. Saw this unconfirmed Bug 108319 so keep that in the back of your mind if you see a lot of look ups or have some custom stuff running. These are just questions to think about as you zero in on the failing component.

Perhaps... gather some more information to see if you can narrow this down further. I am guessing you have been through the logs and found nothing and appear to be focused on the java components of Zimbra which seems like a good place to start.

Code: Select all

pidstat -l 1
iostat -xz 5
mpstat -P ALL 1

Watching pidstat for a while might give some additional clues once you lock onto the problem. I have been holding off moving to 8.8+ so am interested in what the resolution is myself. I would also be watching cpu affinity/pinning with some of these cpu bound tasks. I am out of ideas at this point and don't know either.
User avatar
sarge
Posts: 3
Joined: Sun Mar 18, 2018 3:44 pm
Location: Moscow

Re: High cpu load for java process after upgrade to 8.8.6

Postby sarge » Sun Mar 18, 2018 8:37 pm

At some point in the working day, the system(Web-interface users complain) become very slow.
CPU load for java, in this moment 100-150% constantly. But, server working.
After reload, the situation for some time is normalized.

While I was looking for a solution, I found this: https://forums.zimbra.org/viewtopic.php?f=15&t=58059&start=10
But I do not know, if this is right for me.
Now I'm thinking of increasing the number of processors, can this help? :-(

Tomorrow I will gather more information...

Thank you.
donguru
Posts: 3
Joined: Fri Mar 23, 2018 2:59 pm

Re: High cpu load for java process after upgrade to 8.8.6

Postby donguru » Fri Mar 23, 2018 3:08 pm

Same problem here! Zimbra 8.8.5, java 100% CPU minimun. My server has 16 cores. Only restart zmcontrol works fine!

How did you solve it?
donguru
Posts: 3
Joined: Fri Mar 23, 2018 2:59 pm

Re: High cpu load for java process after upgrade to 8.8.6

Postby donguru » Fri Mar 23, 2018 3:10 pm

sarge wrote:At some point in the working day, the system(Web-interface users complain) become very slow.
CPU load for java, in this moment 100-150% constantly. But, server working.
After reload, the situation for some time is normalized.

While I was looking for a solution, I found this: https://forums.zimbra.org/viewtopic.php?f=15&t=58059&start=10
But I do not know, if this is right for me.
Now I'm thinking of increasing the number of processors, can this help? :-(

Tomorrow I will gather more information...

Thank you.


Same problem here! Zimbra 8.8.5, java 100% CPU minimun. My server has 16 cores. Only restart zmcontrol works fine!

How did you solve it?
Last edited by donguru on Fri Mar 23, 2018 5:50 pm, edited 2 times in total.
User avatar
L. Mark Stone
Elite member
Elite member
Posts: 1780
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine
ZCS/ZD Version: 8.8.8 Patch 4 Network Edition
Contact:

Re: High cpu load for java process after upgrade to 8.8.6

Postby L. Mark Stone » Fri Mar 23, 2018 3:35 pm

Did you install the 8.8.6 Hotfix?
https://wiki.zimbra.com/wiki/Zimbra_Rel ... 8.6_HotFix

Or better yet upgrade to 8.8.7?
_____________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP and Consulting https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/
els2net
Posts: 6
Joined: Sat Sep 13, 2014 3:16 am

Re: High cpu load for java process after upgrade to 8.8.6

Postby els2net » Wed Apr 18, 2018 5:18 pm

L. Mark, i had the same problem.

You have found the solution ?

tks by help me.
donguru
Posts: 3
Joined: Fri Mar 23, 2018 2:59 pm

Re: High cpu load for java process after upgrade to 8.8.6

Postby donguru » Wed Apr 18, 2018 6:43 pm

L. Mark Stone wrote:Did you install the 8.8.6 Hotfix?
https://wiki.zimbra.com/wiki/Zimbra_Rel ... 8.6_HotFix

Or better yet upgrade to 8.8.7?



It's works fine! Upgrade to 8.8.6 + patch! Process java still < 100%

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 23 guests