Jetty High CPU load since July 1st 00:00:00 UTC

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
1048cvidal
Advanced member
Advanced member
Posts: 51
Joined: Fri Sep 12, 2014 10:02 pm

Jetty High CPU load since July 1st 00:00:00 UTC

Postby 1048cvidal » Sun Jul 01, 2012 3:48 am

I woke up this morning with 5 of my customers in 3 different continents showing high CPU loads for jetty since 00:00:00 UTC. 2 of them have only 5 accounts ! I restarted Zimbra but nothing changes, Jetty goes up again to the same load avg. This is what I see with top (process 10223 is jetty). The other 5 servers (and counting) with the same problem have similar output. The mailbox.log shows nothing abnormal. Any clues ???
All servers are running ZCS NE 7.1.4 on RHEL6.
top - 10:31:52 up 6 days, 49 min, 2 users, load average: 17.59, 18.02, 18.09

Tasks: 310 total, 2 running, 308 sleeping, 0 stopped, 0 zombie

Cpu0 : 26.4%us, 69.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 4.4%si, 0.0%st

Cpu1 : 28.0%us, 68.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.4%si, 0.0%st

Cpu2 : 28.5%us, 68.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.5%si, 0.0%st

Cpu3 : 42.5%us, 53.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 4.1%si, 0.0%st

Cpu4 : 27.7%us, 68.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.4%si, 0.0%st

Cpu5 : 35.4%us, 63.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.4%si, 0.0%st

Cpu6 : 30.4%us, 66.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.3%si, 0.0%st

Cpu7 : 33.0%us, 63.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.3%si, 0.0%st

Mem: 8176384k total, 4315580k used, 3860804k free, 677332k buffers

Swap: 3999736k total, 13888k used, 3985848k free, 1436244k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

10223 zimbra 20 0 6395m 502m 13m S 489.8 6.3 305:08.96 java

8811 zimbra 20 0 2217m 252m 12m S 117.4 3.2 82:38.36 java

13 root 20 0 0 0 0 S 13.6 0.0 36:30.47 ksoftirqd/3


MTHead
Posts: 7
Joined: Fri Sep 12, 2014 11:36 pm

Jetty High CPU load since July 1st 00:00:00 UTC

Postby MTHead » Sun Jul 01, 2012 7:01 am

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

29733 zimbra 20 0 4075m 1.1g 14m S 353 14.2 2252:36 java

28617 zimbra 20 0 820m 395m 12m S 171 4.9 914:14.79 java


same here
bku09
Posts: 13
Joined: Sat Sep 13, 2014 12:58 am

Jetty High CPU load since July 1st 00:00:00 UTC

Postby bku09 » Sun Jul 01, 2012 7:11 am

GOT THE SAME PROBLEM, NOTHING INTRESTING IN LOGFILES ...
SYSTEM: UBUNTU 10.04.4 LTS, ZCS 7.2.0, ZIMBRA RESTART DIDN'T SOLVE THE PROBLEM ...
STRACE ON 100% CPU JETTY THREADS:

----------------------------------

FUTEX(0X41F7B328, FUTEX_WAKE_PRIVATE, 1) = 0

FUTEX(0X7FED20AA7F24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 697245000}, FFFFFFFF) = -1 ETIMEDOUT (CONNECTION TIMED OUT)

FUTEX(0X41F7B328, FUTEX_WAKE_PRIVATE, 1) = 0

FUTEX(0X7FED20AA7F24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 697814000}, FFFFFFFF) = -1 ETIMEDOUT (CONNECTION TIMED OUT)

FUTEX(0X41F7B328, FUTEX_WAKE_PRIVATE, 1) = 0

FUTEX(0X7FED20AA7F24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 698305000}, FFFFFFFF) = -1 ETIMEDOUT (CONNECTION TIMED OUT)

FUTEX(0X41F7B328, FUTEX_WAKE_PRIVATE, 1) = 0

FUTEX(0X7FED20AA7F24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 698818000}, FFFFFFFF) = -1 ETIMEDOUT (CONNECTION TIMED OUT)

FUTEX(0X41F7B328, FUTEX_WAKE_PRIVATE, 1) = 0

FUTEX(0X7FED20AA7F24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 699337000}, FFFFFFFF) = -1 ETIMEDOUT (CONNECTION TIMED OUT)

FUTEX(0X41F7B328, FUTEX_WAKE_PRIVATE, 1) = 0

FUTEX(0X7FED20AA7F24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 699821000}, FFFFFFFF) = -1 ETIMEDOUT (CONNECTION TIMED OUT)

FUTEX(0X41F7B328, FUTEX_WAKE_PRIVATE, 1) = 0

FUTEX(0X7FED20AA7F24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 700337000}, FFFFFFFF) = -1 ETIMEDOUT (CONNECTION TIMED OUT)
GRAPH:

------


zimbra.jpg

MTHead
Posts: 7
Joined: Fri Sep 12, 2014 11:36 pm

Jetty High CPU load since July 1st 00:00:00 UTC

Postby MTHead » Sun Jul 01, 2012 7:27 am

Found this , trying it out
java leap second bug – 30 June / 1 July 2012 – fix » WPKG Blog » Blog Archive
bku09
Posts: 13
Joined: Sat Sep 13, 2014 12:58 am

Jetty High CPU load since July 1st 00:00:00 UTC

Postby bku09 » Sun Jul 01, 2012 7:44 am

thanks to user MTHead, here is the solution (tested and confirmed working):
-----

/etc/init.d/ntp stop

date `date +"%m%d%H%M%C%y.%S"`
You don’t have to restart your java applications (tomcat, wowza, or whatever using java); it should be enough to run the above commands.

You may want to enable ntp the next day (there were some reports that enabling it makes java misbehave on some installations).

-----

Source: java leap second bug – 30 June / 1 July 2012 – fix » WPKG Blog » Blog Archive
1048cvidal
Advanced member
Advanced member
Posts: 51
Joined: Fri Sep 12, 2014 10:02 pm

Jetty High CPU load since July 1st 00:00:00 UTC

Postby 1048cvidal » Sun Jul 01, 2012 7:53 am

It seems like a generic Java problem. One of the servers has a Java tool to monitor the UPS and shutdown if it runs out of battery. This process was also sucking a lot of CPU together with Jetty and zmconfigd. I did the Windows way (yes, mea culpa!), and rebooted the server (after 200 days uptime!), and so far the load averages are back to normal. What is this crap ?
dave_kempe
Outstanding Member
Outstanding Member
Posts: 200
Joined: Fri Sep 12, 2014 9:55 pm

Jetty High CPU load since July 1st 00:00:00 UTC

Postby dave_kempe » Sun Jul 01, 2012 8:41 am

You don't need to stop NTP.

My simple solution was this:

date -s "`date`"

and CPU usage returned to normal.

Also affected a few mysql intances on other apps, so watch out.
phoenix
Ambassador
Ambassador
Posts: 26677
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Jetty High CPU load since July 1st 00:00:00 UTC

Postby phoenix » Sun Jul 01, 2012 9:40 am

I guess someone will be filing a bug report about this problem and it's solution?
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
1048cvidal
Advanced member
Advanced member
Posts: 51
Joined: Fri Sep 12, 2014 10:02 pm

Jetty High CPU load since July 1st 00:00:00 UTC

Postby 1048cvidal » Sun Jul 01, 2012 10:15 am

Thank you very much everybody, everything is back to normal ;-)
bku09
Posts: 13
Joined: Sat Sep 13, 2014 12:58 am

Jetty High CPU load since July 1st 00:00:00 UTC

Postby bku09 » Sun Jul 01, 2012 11:11 am

the java "leap second bug" causes some serious problems all around the net:
'Leap Second' Bug Wreaks Havoc Across Web | Wired Enterprise | Wired.com

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 18 guests