one of my customers' zimbra server was restarted due to power loss recently. The machine runs zimbra "Release 7.1.0_GA_3140.UBUNTU8_64 UBUNTU8_64 FOSS edition." on ubuntu 8.04 LTS. The bad news: there is no backup and there is no UPS.
So much for the unchangeable facts.
Now the symptoms:
Starting zimbra fails at the attempt to start ldap. The zimbra.log says:
When I try to start slapd a bit more verbosely an irritating new issue arises:
Aug 10 16:01:31 zimbra slapd: @(#) $OpenLDAP: slapd 2.4.23 (Mar 4 2011 19:31:58) $ ^Iroot@zre-ubuntu8-64.eng.vmare.com:/home/build/p4/HELIX/ThirdParty/openldap/openldap-126.96.36.199z/servers/slapd
Aug 10 16:01:31 zimbra slapd: str2entry: entry -1 has no dn
Aug 10 16:01:31 zimbra slapd: slapd stopped.
Aug 10 16:01:31 zimbra slapd: connections_destroy: nothing to destroy.
Recipes that do *not* work for me:
zimbra@zimbra:~$ sudo /opt/zimbra/libexec/zmslapd -f /opt/zimbra/openldap/etc/openldap/slapd.conf -d 256
@(#) $OpenLDAP: slapd 2.4.23 (Mar 4 2011 19:31:58) $
Unrecognized database type (bdb)
/opt/zimbra/openldap/etc/openldap/slapd.conf: line 53: <database> failed init (bdb)
connections_destroy: nothing to destroy.
There are several threads around here, pointing out how to:
* recover from backups (obviously not an option)
* db_recover from zimbras packaged bdb tools happily returns "0" and changes not a single file, when run inside /opt/zimbra/data/ldap/hdb/db
* dumping and regenerating the bdb files with db_dump - db_load (as proposed on a bdb mailing list) seems to modify some files (they slightly shrink in size) but fails to make slapd happy too.
* of course slapd has bdb support _not_ compiled as a module (this is the standard reply from openldap folks) and the configuration is exactly the same as it is in a fresh zimbra installation.
I even tried to reinstall zimbra from the original tarball. The installer finds the existing ldap database, tries to recover, fails and then happily loops over the steps start ldap - fail - try to recover - ignore failure - start ldap - ...
No harm seems to be done to the mailstore. There are dozens of message within the 'store' directory and they seem to be intact. I could even write a script, collecting the files, sorting them by recipient and creating a maildir od maibox structure per user. This could be imported to a newly installed zimbra server. But users would loose their folders within the inbox, since that information is exclusively stored in ldap.
Is it true, that a corrupted bdb file prevents me from using this system ever again? There must be a way to fix that file, or at least extract the remaining data.
Any hints are welcome ...