High CPU usage, Massive slow down
I've spent quite a bit of time searching the forums for an answer to our problems, but so far no luck. So it looks like it's time to post and see if someone might be able to help.
We are running 5.0.8NE on RHEL4.0. The machine has dual Xeon's with 6GB of RAM. This server supports 1500+ users with about 90-95% using the web interface.
The problem we are experiencing is a constant high CPU usage. Top shows 95%+ CPU usage with no idle. The main process swallowing up all this resource is java. The problem cropped up on Friday around 10:30am and we haven't been able to get it to go away. I used zmstat-chart to create graphs from the past few days. Graphs from earlier in the week show normal CPU usage, but on Friday there is a clear CPU ramp up on the graphs. This problem has continued, even through the weekend. Restarting, rebooting, even failing over to the second server doesn't resolve this problem. We've tried a few changes to the memory heap of java and mysql, but nothing seems to work.
Any ideas would be greatly appreciated. We might go to the backups to resolve the problem, but understanding what and why would help us sleep better at night.