I have been running two separate, similar installs of ZCS. They are one-box installs. I started around 2012 with 7.x, then went successfully with all the upgrades to 8.0, 8.5, 8.6 and even to 8.7 after some struggling. And both ZCS installs were running quite smoothly, on CentOS 6 based VM under ESXi.
Last Sunday, I downloaded 8.8.3 from zimbra.org site. It was marked as GA then, now it has changed back to RC2. As usual, I took the steps to upgrade
- stopped the VM, made snapshot,
- updated ZCS,
- after a few tests everything was looking OK and I put the server online for users.
Unfortunately both installs become terribly slow. Users are complaining they cannot send / receive emails. Since the upgrade we have been observing very high cpu usage on java processes. Restarting with ‘zmcontrol restart’ helps for a while, but after some time it starts to be slow again. The problem exists during office hours. When the load is low, there is no indication of the problem.
Both installs are about 150 mailboxes, total storage is about 800-900 GB. When upgrading, I accidentally installed the new zimbra-imapd package on one of the hosts. It has not been configured and is not active. This host is performing terribly wrong. I omitted zimbra-imapd on the second one, which has been perfoming a bit better, but still I had to restart it on Tuesday morning (after 1,5 day since upgrade).
What can I do ? I understand I can
- wait for the stable version hoping it will fix it, or
- try to downgrade.
Option (1) - fix it. Is it anything we can check or change to investigate the problem? I definitely need some help here.
Option (2). I know downgrading is not possible with install process. However, I have snapshot of the whole drive containing /opt/zimbra taken before upgrade. For one of the hosts I have also snapshot of the whole CentOS VM taken before upgrade.
My thoughts and questions which make my sleepless nights now
- Is it possible to save the present state of mailbox store, restore the ZCS install from the state before upgrade and then make ZCS learn the new mailbox store? We cannot lose emails from the last 4 days.
- If the above is not so easy, I am thinking about saving all the mailboxes with zmmailbox (learning from https://wiki.zimbra.com/wiki/Zmmailbox) and then restoring them after restoring the previous install from snapshot. However it would probably take ages - am I right ? Is there a way to save only messages from the last 4 or 5 days and then import them to existing mailboxes? If that was possible, it would let me restore the previous install and make users happy.