Urgent Help Needed Please 8.0.6 OSE - Power Cut LDAP broken system will not start

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
realsparticle
Posts: 19
Joined: Sat Sep 13, 2014 3:29 am

Urgent Help Needed Please 8.0.6 OSE - Power Cut LDAP broken system will not start

Postby realsparticle » Tue Jan 14, 2014 7:08 pm

System is back up and running I will post a summary of the solution thanks to quanah once I get some sleep :)
Best regards

Stephen
OK here is the log of all the commands/actions that were performed on my server to get it back up again. I cannot stress how brilliant Quanah was in helping me to recover this situation. What an absolute superstar and very gracious in his explanations and insight. I would like to thank everyone who assisted me to get this back up. I hope the information is useful to others please feel free to PM me for any assistance.
At the start of this log the system was broken it would not start and if you have read my previous posts it was clear that the ldap system and possibly the config was broken. Partially by the power cut and probably by my running the upgrade script over itself a number of times when the LDAP system ws broken. That had in effect restored the ldap db to default thereby causing all sorts of issues with server not found 389 errors etc. The only thing i have changed are the password entries returned from the ldap lookups to confirm they were correct/existant etc. after restoring the 21st Dec ldap.bak DB.
In summary the procudure involved restoring a last known good ldap.bak file from 21st Dec which was created at upgrade time from 8.0.4. Then checking various permissions and the localconfig against the config.bak from the 21st Dec and making some changes to bring the schema upto 8.0.6.
The first red line below was after the first startup. The server was running but the Antivirus zmamavis was not running and would not start. The problem was essentially this one Upgrade Zimbra Proxy from 8.0.2 to 8.0.6 Breaks things after setting the zimbraAmavisLogLevel 1 and and checking and setting various keys to ensure the schema matched 8.0.6 not 8.0.4 and tuning the ldap db maxsize the system was restarted and came up perfectly the second red point. Then cold reboot to check if any issues and none were found. Then cold backup etc. see below. I will be looking for some help around automating the backup procedure :)


cd data/ldap/

ls -l

vi ldap.bak.20131221123538

ls

cd /tmp

lsred

rm q.*

cd

cd data/ldap/config/

ls

cd ..

ls

ls -l

grep zimbraThrottleWhitelist ldap.bak.20140113032714

grep zimbraDeviceCalendarSoftDeleteExcludePattern *

grep zimbraItemActionBatchSize *

zmprov gcf zimbraItemActionBatchSize

zmprov mcf zimbraItemActionBatchSize 1000

grep zimbraReverseProxyExternalRouteIncludeOriginalAuthusername *

zmprov mcf zimbraReverseProxyExternalRouteIncludeOriginalAuthusername FALSE

grep zimbraAntispamExtractionBatchSize *

zmprov mcf zimbraAntispamExtractionBatchSize 25

zmprov mcf zimbraAntispamExtractionBatchDelay 100

zmcontrol status

ps -eaf | grep zimbra

cd data/ldap/

ls

ls -l

ls -altr

ls /tmp

mv mdb mdb.broken

mkdir -p mdb/db

ls

cd config

ls

ls -altr

cd cn=config

ls

ls -altr

more olcDatabase={2}mdb.ldif

ls

cd ..

ls

cd ..

ls

ls -l

cd

./libexec/zmslapadd /opt/zibmra/data/ldap/ldap.bak.20131221123538

./libexec/zmslapadd /opt/zimbra/data/ldap/ldap.bak.20131221123538

cd

ldap start

zmlocalconfig | grep ldap | grep password

zmlocalconfig -s | grep ldap | grep password

ldapsearch -x -H ldapi:/// -D cn=config -w XXXXXXXX -b "" -s base

ldapsearch -LLL -x -H ldapi:/// -D cn=config -w XXXXXXXX -b "" -s base

ldapsearch -LLL -x -H ldapi:/// -D uid=zimbra,cn=admins,cn=zimbra -w XXXXXXXX -$

ldapsearch -LLL -x -H ldapi:/// -D uid=replica,cn=admins,cn=zimbra -w XXXXXXXX $

cd .saveconfig/

ls

cd 8.0.6.GA.5922.UBUNTU10.64/

ls -altr

cd 3

ls -altr

cat localconfig.xml

cd

zmlocalconfig

pwd

cd .saveconfig/

ls

cd 8.0.6.GA.5922.UBUNTU10.64/

ls

cd 0

ls

diff localconfig.xml /opt/zimbra/conf/localconfig.xml

cd

zmlocalconfig ssl_default_digest

cd -

ls -l

cd

zmcontrol start

zmcontrol status

zmamavisdctl start

cd conf

vi amavsid.conf.in

ls

vi amavisd.conf.in

cd

hostname

zmlocalconfig | grep host

zmprov gs zimbra.twhg.home zimbraAmavisLogLevel

cd data/ldap/

ls

ls -l

vi ldap.bak.20140113032714

zmprov mcf zimbraAmavisLogLevel 1

zmamavisdctl start

vi ldap.bak.20140113032714

zmprov mcf +zimbraStatThreadNamePrefix qtp

vi /tmp/q.1

cd /tmp

ls

mv 1.2 q.2

rm q.

diff -u q.1 q.2 | more

diff -u q.2 q.1 | more

diff -u q.2 q.1 | more

clear

cd /tmp

ls

mkdir config

cd config

ls

cd

clear

cd data/ldap/mdb/db

ls -l

du -c -h data.mdb

man rsync

ls

zmlocalconfig ldap_db_maxsize

zmlocalconfig -e ldap_db_maxsize=2147483648

ls -l

tail -f /var/log/zimbra.log

ls -l

cd

zmcontrol -v

zmprov help cos

zmprov gac

zmprov mc default zimbraMobileOutlookSyncEnabled TRUE

zmprov mc defaultExternal zimbraMobileOutlookSyncEnabled TRUE

zmprov mc default zimbraFeatureFromDisplayEnabled TRUE

zmprov mc defaultExternal zimbraFeatureFromDisplayEnabled TRUE

zmcontrol restart

zmcontrol status

zmcontrol stop

zmcontrol status

tail -f /var/log/zimbra.log

exit

sudo reboot now


After all was well I then took a full cold backup of the system, backing up the LDAP DB and config the correct way (thank you Quanah) to a backup directory in /opt/zimbra/data/ldap-backup/YYYYMMDDhhmmss. I then ran rsync outside of zimbra and created a full backup of the /opt/zimbra directory tree excluding the data/ldap/mdb/db and any previous mdb.prev directories. My system is approx. 11.3 GB and it took about half an hour. I am sure there is a better way of doing this but I did it long hand and cold to ensure I had at least a manual methodology that worked.
I would welcome any suggestions on the correct rsync options or pointers to 8.0.6 related backup scripts. We backup to a Raid 6 NAS server over the lan. I have seen many posts taking a hot rsync first and then a cold one to minimise down time. I am not quite there yet but I think I am heading in the right direction :)


sudo su - zimbra

zmcontrol stop

zmcontrol status

tail -f /var/log/zimbra.log

cd data/ldap

ls

rm -Rvf mdb.broken

mydir="/opt/zimbra/data/ldap-backup/`date +%Y%m%d%H%M%S`"

mkdir -p $mydir

cd data

ls -lash

cd ldap-backup

/opt/zimbra/libexec/zmslapcat $mydir

/opt/zimbra/libexec/zmslapcat -c $mydir

exit

sudo rsync -a -v -r -l -t -S -H --progress --exclude=data/ldap/mdb/db --exclude=data/ldap/mdb.prev* /opt/zimbra remote@remote:/remotezimbradir/

sudo su - zimbra

zmcontrol status

zmcontrol restart

zmcontrol status

tail -f /var/log/zimbra.log



realsparticle
Posts: 19
Joined: Sat Sep 13, 2014 3:29 am

Urgent Help Needed Please 8.0.6 OSE - Power Cut LDAP broken system will not start

Postby realsparticle » Wed Jan 15, 2014 3:56 pm

I have now documented the solution see above post.
Stephen

Return to “Administrators”

Who is online

Users browsing this forum: guitardood and 9 guests