I have a vm server running ZCS Release 8.8.10.GA.3039.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.10_P8 on Ubuntu 16.04.6 LTS with Z-PUSH 2.5.0+0-0 with zimbrabackend68. I have Apache running as the front door door on the server which either reverse proxies to Zimbra's proxy server or runs Z-Push depending on the request.
The ESXi server is a dual 8-core Xeon with 128gb and has 2 NUMA's. It is running the latest patched ESXi 6.5 and the VM is configured with 4-cores and 32gig of memory. The system/Zimbra had been running just fine for the last few months with nary an issue, but this week started exhibiting this bad behavior.
I'm experiencing a huge CPU load on both the primary java process and it's children as well as the mysqld process. Also, I've noticed a huge number of java exceptions in the zmmailboxd.out log file similar to:
Code: Select all
2019-05-25 00:04:38.754:INFO:oejm.ThreadMonitor:Thread-39: Thread 'qtp66233253-10172:https:https://smtp.XXXX.com/home/XXXX@XXXX.com/?fmt=tgz'[RUNNABLE,id:10172,cpu:90.26%] SPINNING
org.eclipse.jetty.monitor.thread.ThreadMonitorException: Thread 'qtp66233253-10172:https:https://smtp.XXXX.com/home/XXXX@XXXX.com/?fmt=tgz'[RUNNABLE,id:10172,cpu:90.26%] STACK TRACE
at java.util.zip.Deflater.deflateBytes(Native Method)
Code: Select all
2019-05-25 01:13:38.224:WARN:oejs.HttpChannel:qtp66233253-10695: https://smtp.XXXX.com:8443/service/soap/
During the day, I'm seeing load averages of 36.0 and ESXi's esxitop is showing this vm using 480% cpu with 0.9% idle. Something is definitely not right. From an ignorant, "I don't know really know what to do" approach, I'm trying a run of "zmdbintegrityreport -o -r -v", hoping that optimizing the database will help.
Any help would be greatly appreciated.