Page 4 of 6 FirstFirst ... 23456 LastLast
Results 31 to 40 of 51

Thread: How to: cold standby server (no cluster)

  1. #31
    Join Date
    Feb 2007
    Location
    Minnesota
    Posts
    48
    Rep Power
    8

    Default

    I've been doing a live sync for quite sometime now using a modified version of the technique the Parent has posted.

    Basically, I start off like the Parent, with a server configured identically to my Live server.

    I then have a script that Rsyncs the store,index,zimlets-depoyed, and db/data directories on our Live running server.

    Now, this creates the issue of out-of-sync logs since there is a little bit of a time delay in backing up the database. I resolve this by running a full dump and restore of my Zimbra and Mailbox databases.

    The end result is a cold-standby that is updated hourly. I can easily test the server content by logging in using it's own IP address. I use a script similar to the Parent to switch from COLD to HOT in case of main server failure. When the Main server is repaired or replaced, I run the same procedure in reverse. It has saved me now on 3 separate occasions. The most data I risk losing is one hours worth of email (assuming the Main server dies just before the hourly script runs).

    I started doing this under Ubuntu 6.04x86 and Zimbra 6. I now do this with Ubuntu 8.04lts x64 and Zimbra 6.0.7NE.

    I of course have nightly backups being performed using Zimbra's built in functionality, but should something bad happen, the Cold Standby is an immediate and easier option to get back to operational status.
    Last edited by Harley77; 03-28-2011 at 09:57 AM.

  2. #32
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    9

    Default

    Quote Originally Posted by Harley77 View Post
    I resolve this by running a full dump and restore of my Zimbra and Mailbox databases.
    Pardon my ignorance, but how do you do this?

  3. #33
    Join Date
    Feb 2007
    Location
    Minnesota
    Posts
    48
    Rep Power
    8

    Default

    Quote Originally Posted by ewilen View Post
    Pardon my ignorance, but how do you do this?
    I'll post the Script I use. This is a modified version of a bash script I believe came from the forums here or in one of the Wiki's. It's been awhile so I don't quite remember.

    This is the last thing that runs in the mirroring process.
    I've been using it very successfully for almost two years now, so it's well tested as it runs every hour on the hour.

    It also works fine with Zimbra 7.
    Attached Files Attached Files

  4. #34
    Join Date
    Jun 2010
    Posts
    21
    Rep Power
    5

    Default How long does the restoration take on clod standby server?

    We have a zimbra NE 7.0.1 server with more than 100 users. I read this thread carefully, one real concern is that how long it will take for the cold standby server's restore process done. It seems to me that every time it does the restore on standby server, it will do the whole disaster recover procedure again.

    I am thinking to work this way:

    1. On live server, do weekly full backup, hourly incremental backup;

    2. Rsync full backup to standby server weekly; And after rsync done, run the restoration;

    3. Rsync incremental backup to standby server hourly; And after rsync, only

    restore the incremental backup (might have errors with new created account, but this way it should be fast enough );

    4. Once live server is down, startup the zimbra on standby server should be enough.

    Does anybody else think it is doable solution?

    I also noticed someone said to rsync "/opt/zimbra" whole directory with some mysql handling script to correct errors. I am afraid it will not work with zimbra next upgrade. So I prefer to use Zimbra native command if possible to implement my backup procedure.

    Thanks,
    Rocky


    Quote Originally Posted by fisch09 View Post
    Hi,

    Having a cold standby server in a different location would be nice. But at this point you can't use clustering anymore. So I spent some time and build a cold standby. Here is how I did it.

    System: ZCS 4.5.6 NE on Ubuntu 6.06.1

    Step one:
    Create a 1:1 copy of your server. You can use what ever you prefer LVM snapshot, physical hard drive copy, rsync. I used rsync as I don't have physical access to server. Important for this step: zimbra needs to be down to make sure its really 1:1.

    Step two:
    Adjusting DNS. As zimbra wont start if the DNS is not correct we have to fake a bit.
    Lets say the primary server is: zmail.mydomain.tld - add an additional DNS entry for this server zmail2.mydomain.tld
    Install a local DNS server on your cold standby server - I used dnsmasq.
    We need this DNS server so the cold standby server can use the FQDN of your primary server while having a different IP. To do this I added this line in /etc/dnsmasq.conf:
    Code:
    address=/zmail.mydomain.tld/192.168.1.100
    and changed /etc/hosts to:
    Code:
    127.0.0.1       localhost.localdomain localhost
    192.168.1.100  zmail.mydomain.tld zmail
    Now the cold standby server can use "zmail.mydomain.tld" to run a local zimbra configured for you primary server and still access the primary server using zmail2.mydomain.tld.

    Step three:
    Configure password less ssh using ssh keys - we need this to use rsync in a cron job.

    Step four:
    To be sure nothing is going wrong while syncing the backups I moved /opt/zimbra/backup to /zmailbackup.

    Step five:
    Create some scripts to control the sync / restore and if your server is "cold standby" or "active"

    /root/coldstandby is just a file. I use it to check if the server is in "cold standby" modus or live.

    /root/change.zimbra.status.sh is used to change the function from "cold standby" to "live". If the server is live you don't want to sync with your primary server anymore.....

    Code:
    #!/bin/bash
    case $1 in
    "status")
       if [ -f /root/coldstandby ]; then
       echo "server is in cold standby modus"
       else
       echo "server is LIVE"
       fi
       ;;
    "cold")
       echo "switching into cold modus"
       echo "... remove start scripts"
       update-rc.d -f zimbra remove
       echo "... activate sync, backup, restore"
       echo "if this file is missing server is live" > /root/coldstandby
       echo "... stop zimbra"
       /etc/init.d/zimbra stop
       echo "... done"
       ;;
    "hot")
       echo "switching into live modus"
       echo "... install start scripts"
       update-rc.d zimbra defaultis 99
       echo "... deactivate sync, backup, restore"
       rm /root/coldstandby
       echo "... check if a restore is running"
       RESTORE=`ps fax | grep -i java | grep -i restore | wc -l`
       if [ $RESTORE -gt 0 ]; then
         while [ $RESTORE -gt 0 ]; do
           echo "!!! FOUND ACTIVE RESTORE PROCESS PLEASE WAIT UNTIL FINISHED !!!"
           echo "... waiting for 5 min, and check again"
           sleep 3000
           RESTORE=`ps fax | grep -i java | grep -i restore | wc -l`
         done
       fi
       echo "... no restore runnin anymore ... going live now"
       /etc/init.d/zimbra stop
       /etc/init.d/zimbra start
       ;;
    *)
       echo "help..."
       echo "switch to live modus: ./change.zimbra.status hot"
       echo "switch to cold modus: ./change.zimbra.status cold" 
       echo "query status: ./change.zimbra.status status"
       ;;
    esac
    /root/sync.live.server.sh syncs the backup folder with the primary server and starts the restore.
    Code:
    #!/bin/bash
    if [ -f /root/coldstandby ];
    then
      echo "`date`: start syncing backups" >> /var/log/zimbra.cold.log
      rsync -a root@zmail2.mydomain.tld:/opt/zimbra/backup/* /zmailbackup/
      echo "`date`: start restoring backups" >> /var/log/zimbra.cold.log
      su - zimbra /opt/zimbra/cold.restore.sh
      echo "`date`: restore finished"
    else
      echo "`date`: no sync/restore done server considered to be LIVE" >> /var/log/zimbra.cold.log
    fi
    /opt/zimbra/cold.restore.sh restores the backup
    Code:
    #!/bin/bash
    zmcontrol stop
    LABEL=`zmrestoreldap -lbs -t /zmailbackup | sed -n 1p`
    zmrestoreldap -lb $LABEL -t /zmailbackup
    zmmailboxctl start
    zmrestore -a "all" --ignoreRedoErrors -t /zmailbackup
    zmcontrol stop
    I had trouble with RedoErrors and the only way I could get it to work was using the "--ignoreRedoErrors" option. I also tried to use zmrestoreoffline - but this did not work at all for me. The "zmcontrol stop" at the beginning and the end are just for safety.

    The only thing that is left is a cron job. I have this line in my /etc/crontab:
    Code:
    55 */2  * * *   root   /root/sync.live.server.sh
    For me this is 30min after the server did it's backup - which works fine for me.

    I know that the "cold standby" server is never 100% up2date but this is ok for me.

    Cheers
    Andre

  5. #35
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    9

    Default

    Quote Originally Posted by Rocky View Post
    We have a zimbra NE 7.0.1 server with more than 100 users. I read this thread carefully, one real concern is that how long it will take for the cold standby server's restore process done. It seems to me that every time it does the restore on standby server, it will do the whole disaster recover procedure again.

    I am thinking to work this way:

    1. On live server, do weekly full backup, hourly incremental backup;

    2. Rsync full backup to standby server weekly; And after rsync done, run the restoration;

    3. Rsync incremental backup to standby server hourly; And after rsync, only

    restore the incremental backup (might have errors with new created account, but this way it should be fast enough );

    4. Once live server is down, startup the zimbra on standby server should be enough.

    Does anybody else think it is doable solution?
    It sounds doable; the main concern as you note is accounts that have been created since the last weekly full backup. I have a note to myself that doing zmrestoreldap may be sufficient to create the missing accounts before having their data populated by zmplayredo, but I haven't actually tested that.

    This may also be of interest: Bug 33606 – Improve zmrestore & zmrestoreoffline performance (Note the comments.)

  6. #36
    Join Date
    Jun 2010
    Posts
    21
    Rep Power
    5

    Default

    Thanks Elliot. After thinking about it for a while and make it easier to implement, I decided to on standby server run zimbra restore daily (e.g. 2am) including ldap and mailbox; And hourly rsync the incremental backup to standby server. In case of live server failure, one need manually login standby server to do following steps:

    1. If no users was created that day, just simply use "zmplayredo" to apply all redo logs after the last restoration;

    2. If new users were created, first run " zmrestoreldap -lb ", then play the redo log files;

    Either way, I'll assume the restoration will be quite fast. Then start zimbra service on standby server. Thus we got the least downtime (which should include the zimbra DNS TTL time). I'll give that a try.

    Thanks,
    Rocky


    Quote Originally Posted by ewilen View Post
    It sounds doable; the main concern as you note is accounts that have been created since the last weekly full backup. I have a note to myself that doing zmrestoreldap may be sufficient to create the missing accounts before having their data populated by zmplayredo, but I haven't actually tested that.

    This may also be of interest: Bug 33606 – Improve zmrestore & zmrestoreoffline performance (Note the comments.)

  7. #37
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    9

    Default

    You're welcome. Incidentally, you might also want to look at the links this post: http://www.zimbra.com/forums/adminis...tml#post113555

  8. #38
    Join Date
    Jun 2010
    Posts
    21
    Rep Power
    5

    Default

    I tested the script on first page and Harley77-Backup - Zimbra :: Wiki (Method 1), none of them work for me (Zimbra NE 7.0.1 on CentOS 5.6). I have to exactly follow the restore procedure here Network Edition Disaster Recovery - Zimbra :: Wiki to restore my zimbra mailbox. I'll have to put it in shell script to make it work.

    I have not tried the redo log play part. Will definitely give it a try. Since we have huge of user data (200 ~ 300GB), rsyncing one full backup from live server to standby server took 18 ~ 24 hours. I'll see if i can restore the whole server daily (which will depends how long it took for restoration process); I might end up with only keep last week's data imported, and manually play redo log one by one (or possibly just rsync live server:/opt/zimbra/redo to live server for log replay later).

  9. #39
    Join Date
    Jun 2010
    Posts
    21
    Rep Power
    5

    Default

    I got a very clear idea regarding our zimbra live standby server now:

    1. Weekly full backup on live server, sync to standby server and restore it right away;

    2. Every day incremental backup, sync to standby server, apply the ldap db and redo log after that;

    3. Hourly sync the redolog (/opt/zimbra/redologs) from live server to standby server;

    In case of live server down, one just need manually login standby server, apply the latest redo log got from step 3, then start up the zimbra service. This is by far the fastest method to make standby server online with the at most lost of one hour data, plus we only use zimbra native commands and no tricks needed.

  10. #40
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    9

    Default

    Sounds good.

    Note: depending on a variety of factors, the form that your full backup takes will affect the speed of your rsync. If you don't use the --noZip option, then I doubt there's any benefit to having rsync do a compare before copy. You'll also be pumping quite a bit of data through. On the other hand if you do use --noZip (has to be applied manually in crontab, under ZCS 6), then you can structure your rsync so that it'll only copy the files that differ from the previous full backups, creating local hardlinks for the rest. However, my experience is that you then spend a fair amount of time on the compare.

Similar Threads

  1. initializing ldap...FAILED(256)ERROR
    By manjunath in forum Installation
    Replies: 39
    Last Post: 06-07-2013, 10:27 AM
  2. Zimbra fails after working for 2 weeks
    By Linsys in forum Administrators
    Replies: 10
    Last Post: 10-07-2008, 12:42 AM
  3. need advice on configuring zimbra to work with fax server
    By pheonix1t in forum Administrators
    Replies: 0
    Last Post: 07-11-2007, 07:46 PM
  4. Error 256 on Installation
    By RuinExplorer in forum Installation
    Replies: 5
    Last Post: 10-19-2006, 09:19 AM
  5. Getting problems in FC4 while instalation
    By kitty_bhoo in forum Installation
    Replies: 13
    Last Post: 09-12-2006, 10:34 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •