Short story long: Inherited environment.
Wanting to upgrade from Open Source ZCS 4.5.7 to 5.0.21 on CentOS5.
Disk at 67% of 100GB used. Not enough space to perform Zimbra backup locally.
Logger had been disabled for some time.
Logger db was over 51 GB. Stats last updated in Aug 2009.
Not realizing that there had been an issue previously,
I enabled the logger via the GUI. Disk usage started increasing.
It was growing at about 200 MB per second. Root partition rapidly went from 67% usage to 85% and climbing fast. I tried to find which directory was being written into. I think it was related to btmp file, but not sure.
Best guess is that it was trying to build the missing statistics graphs, etc.
mysqld was running at 100% cpu. Load average was holding at 4.
Then, just as I thought I was going to run out of disk space, it stopped.
The disk space was returned, but now the load average went from 4 to 10.
I had top running in one console window, and another window was slow to respond.
Before I could kill the logger, the system froze. It had run out of swap space, and the load average went to 40 before it died. Had to reboot. Ugh.
Any ideas what happened?
I found a wiki article on how to shrink the logger db:
Logger (ZCS 5.0.x and earlier) - Zimbra :: Wiki
I followed those instructions and it reduced it from 51 GB down to 31 MB.
I was then able to run the backup script locally.
But I am still a little reluctant to re-enable the logger at this time.
I am going to proceed with the upgrade to 5.0.21 in hopes that whatever it was has been fixed. But it would still be nice to perform a post-mortem to understand what had happened.
Any thoughts would be most appreciated.