Page 1 of 11 123 ... LastLast
Results 1 to 10 of 659

Thread: [SOLVED] Yet Another Backup Script for Community Version

Hybrid View

  1. #1
    Join Date
    Jan 2008
    Location
    Germany
    Posts
    83
    Rep Power
    7

    Default [SOLVED] Yet Another Backup Script for Community Version

    Hi All

    I have written the following script to suit my backup needs (it is actually quite a simple script...). I want to publish it here so you can have a good and maybe help improve on it and add functionality to it.

    I admit some of the ideas come from existing scripts and ideas from forums.

    I will maintain a current working version of the script here: Home*-*Open Source Office Lösungen

    The Script Download now available again!

    Script features:
    • Backups with or without strong encryption!
    • Optional Off-site copying of archives after creation
    • MD5 checksums for integrity checks of archives
    • Weekly backup rotation - 1 Full & 6 Diff's per rotation
    • Email report on Full backup
    • Email notifications on errors
    • Backup file lists (attached to weekly full backup report)
    • Installer & Setup option for quick deployment (install needed software and setup env e.g. ssh pki auth and cronjob helper)

    Here is the code up to date code:
    Code:
    #!/bin/bash
    ## *** Info ***
    # USAGE:    -h or --help for help & usage.
    #           -f or --full for Full backup.
    #           -d or --diff for Diff backup.
    #           -V or --version for version info.
    #           --INSTALL	 for script install and setup.
    #
    # This is a backup script for the FOSS version of Zimbra mail server.
    # The script is free and open source and for use by anyone who can find a use for it.
    #
    # THIS SCRIPT IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
    # A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT
    # HOLDERS AND/OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
    # INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
    # BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
    # LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
    # AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
    # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
    # THE USE OF THIS DOCUMENT, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    #
    # What this script does:
    # 1. Makes daily off-line backups, at a service downtime of ~ < 2 min.
    # 2. Weekly backup cycle - 1 full backup & 6 diffs.
    # 3. Predefined archive sizes, for writing backups to CD or DVD media...
    # 4. Backup archive compression.
    # 5. Backup archive encryption.
    # 6. Backup archive integrity checks and md5 checksums creation.
    # 7. Automated DR - Off-site copy of backup archives via ssh.
    # 8. Install and setup function for needed sofrware (Ubuntu Systems only)
    # 9. Weekly eMail report & eMail on error - including CC address.
    #
    # This script makes use of following tools:
    # apt-get, cron, dar, dpkg, mailx, md5sum, rsync, ssh, uuencode, wget, zimbra mta.
    #
    # We have opted to use a pre-sync directory to save on "down time", but this 
    # causes one to have huge additional space usage.
    # But hard drives are cheep today!
    #
    # What is still to come or needs work on:
    # 1. Recover option
    # 2. Better documentation
    #
    #------- CONFIG -------#
    # Edit this part of the script to fit your needs.
    #
    
    #--- Directories ---#
    # Please add the trailing "/" to directories!
    ZM_HOME=/opt/zimbra/
    SYNC_DIR=/srv/backup/sync/
    ARCHIVEDIR=/srv/backup/dars/
    TOO_MEDIA_DIR=/srv/backup/burn/
    
    #
    #--- PROGRAM OPTIONS ---# 
    RSYNC_OPTS="-aHK --delete --exclude=*.pid"
    
    #
    #--- ARCHIVE NAMES ---#
    BACKUPNAME="Zimbra_Backup"
    BACKUPTYPE_F="FULL"	# name prefix to sort between full and diff backups
    BACKUPTYPE_D="DIFF"
    BACKUPDATE=`date +&#37;d-%B-%Y`
    
    #
    #--- ARCHIVE SIZE ---#
    # storage media size
    ARCHIVESIZE="4395M"
    COMPRESS="9"		# valid answers are 1 - 9 ( 9 = best )
    
    CRYPT="yes"		# valid answers are "yes" or "no"
    PASSDIR=/etc/zmbac/
    PASSFILE="noread"
    
    #
    #--- EMAIL ADDRESS ---#
    EMAIL="admin@localhost"
    EMAILCC="admin@remotehost"
    LOG="/var/log/zim_backup.log"
    
    #--- SSH REMOTE DR COPY ---#
    # This option will secure copy you archives to a remote server via 'scp'
    DRCP="yes" 		# valid answers are "yes" or "no" 
    SSHUSER="heinzg"
    REMOTEHOST="172.16.184.1"
    REMOTEDIR="/tmp/"
    
    
    #--- USE HACKS !?! ---#
    # Built in hacks to fix common problems
    #Hack to start Stats, even run zmlogprocess if needed
    STATHACK="yes" 		# valid answers are "yes" or "no"
    
    
    ## ~~~~~!!!! SCRIPT RUNTIME !!!!!~~~~~ ##
    # Best you don't change anything from here on, 
    # ONLY EDIT IF YOU KNOW WHAT YOU ARE DOING
    #
    #
    
    !!! the script is to long to fit into the post!! To view the full script visit my development site link is above.
    Here is a sample print out of the log file after a full & diff backup:

    Full Backup started at: 23:21

    No old backups found

    Setting date & version Marker into /opt/zimbra/DATE_VERSION.txt

    Doing a hotsync of /opt/zimbra/ to /srv/backup/fakebackup/
    sync went ok

    Stopping the Zimbra server

    Deactivated zimbra's cronjobs

    Host mail.igrill.local
    Stopping stats...Done
    Stopping mta...Done
    Stopping spell...Done
    Stopping snmp...Done
    Stopping archiving...Done
    Stopping antivirus...Done
    Stopping antispam...Done
    Stopping imapproxy...Done
    Stopping mailbox...Done
    Stopping logger...Done
    Stopping ldap...Done

    Zimbra server has stopped

    kill any left over zimbra procs
    Nothing to kill

    Doing a fast cold sync...
    sync went ok

    Reinstated zimbra's cronjobs

    Host mail.igrill.local
    Starting ldap...Done.
    Starting logger...Done.
    Starting mailbox...Done.
    Starting antispam...Done.
    Starting antivirus...Done.
    Starting snmp...Done.
    Starting spell...Done.
    Starting mta...Done.
    Starting stats...Done.

    Zimbra has started

    Service down time was - Hr:0 Min:2 Sec:16

    Running A hack... This one to check and start Stats subsystem
    Hack not needed Stats seems to be running fine...

    All services are running


    Writing a full backup called:
    12_Zimbra_Backup_30-March-2008_FULL
    in: /srv/backup/zimbra_dars/ with file sizes of max: 4395M
    Archive is Encrypted


    --------------------------------------------
    19271 inode(s) saved
    with 1115 hard link(s) recorded
    0 inode(s) changed at the moment of the backup
    0 inode(s) not saved (no inode/file change)
    0 inode(s) failed to save (filesystem error)
    0 inode(s) ignored (excluded by filters)
    0 inode(s) recorded as deleted from reference backup
    --------------------------------------------
    Total number of inode considered: 19271
    --------------------------------------------
    EA saved for 0 inode(s)
    --------------------------------------------
    Archive Created successfully

    Create MD5 Checksum for 12_Zimbra_Backup_30-March-2008_FULL.1.dar

    MD5 Checksum Created successfully

    copy archive to 172.16.184.1 remote directory /tmp/

    Create file listing from archive

    :::::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::
    Full Zimbra Backup ended at: 23:42
    Backup took Hr:0 Min:21 Sec:27 to complete
    :::::::::::::::::Cheers Osoffice for the script:::::::::::::::::::::::


    Diff Backup started at: 23:46

    Doing a hotsync of /opt/zimbra/ to /srv/backup/fakebackup/

    sync went ok

    Stopping the Zimbra server

    Deactivated zimbra's cronjobs

    Host mail.igrill.local
    Stopping stats...Done
    Stopping mta...Done
    Stopping spell...Done
    Stopping snmp...Done
    Stopping archiving...Done
    Stopping antivirus...Done
    Stopping antispam...Done
    Stopping imapproxy...Done
    Stopping mailbox...Done
    Stopping logger...Done
    Stopping ldap...Done

    Zimbra server has stopped

    kill any left over zimbra procs
    Nothing to kill

    Doing a fast cold sync...
    sync went ok

    Reinstated zimbra's cronjobs

    Host mail.igrill.local
    Starting ldap...Done.
    Starting logger...Done.
    Starting mailbox...Done.
    Starting antispam...Done.
    Starting antivirus...Done.
    Starting snmp...Done.
    Starting spell...Done.
    Starting mta...Done.
    Starting stats...Done.

    Zimbra has started

    Service down time was - Hr:0 Min:2 Sec:15

    Running A hack... This one to check and start Stats subsystem
    Hack not needed Stats seems to be running fine...

    All services are running


    Writing a diff backup called:
    12_Zimbra_Backup_30-March-2008_DIFF
    in: /srv/backup/zimbra_dars/ with file sizes of max: 4395M
    Archive is Encrypted
    Warning, the archive 12_Zimbra_Backup_30-March-2008_FULL has been encrypted. A wrong key is not possible to detect, it would cause DAR to report the archive as corrupted


    --------------------------------------------
    154 inode(s) saved
    with 1115 hard link(s) recorded
    0 inode(s) changed at the moment of the backup
    19118 inode(s) not saved (no inode/file change)
    0 inode(s) failed to save (filesystem error)
    0 inode(s) ignored (excluded by filters)
    5 inode(s) recorded as deleted from reference backup
    --------------------------------------------
    Total number of inode considered: 19277
    --------------------------------------------
    EA saved for 0 inode(s)
    --------------------------------------------
    Archive Created successfully

    Create MD5 Checksum for 12_Zimbra_Backup_30-March-2008_DIFF.1.dar

    MD5 Checksum Created successfully

    copy archive to 172.16.184.1 remote directory /tmp/

    Create file listing from archive

    :::::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::
    Diff Zimbra Backup ended at: 23:50
    Backup took Hr:0 Min:4 Sec:26 to complete
    :::::::::::::::::::Cheers Osoffice the script:::::::::::::::::::::::::

    I hope this helps someone and have fun!
    ( Note the service downtime was due to a very weak vmware guest running on my laptop.... So it is faster in real life )

    Cheers
    Heinzg
    Last edited by heinzg; 08-05-2009 at 12:37 PM. Reason: Script download now available again

  2. #2
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Awesome, be sure to get that into Open Source Edition Backup Procedure - Zimbra :: Wiki
    Think under the snapshots section might be appropriate as you have both a full & differential method in there
    Thanks for sharing!

  3. #3
    Join Date
    Feb 2008
    Location
    Norway
    Posts
    59
    Rep Power
    7

    Default

    Mailing the log is disabled in the script right? to enable it, remove # in front of $MAILX everywhere and crontab 30 4 * * * /root/zmbac.sh > /tmp/zim_backup_report.log

    or?

  4. #4
    Join Date
    Jan 2008
    Location
    Germany
    Posts
    83
    Rep Power
    7

    Default

    Hi osiris!

    Yes the sending mails part has been disabled for the moment.
    As you have mentioned you will get it working again. This is the part of the script I still want to refine in the coming days!

    My cronjob for this scrip looks like this:
    Code:
    0 3 * * 1    /bin/sh   /root/zmbac.sh -f > /tmp/zim_backup_report.log 2>&1
    0 3 * * 2-7 /bin/sh   /root/zmbac.sh -d >> /tmp/zim_backup_report.log 2>&1
    I will post an update when I have sorted out the mail reports part.

    Cheers
    Heinzg

  5. #5
    Join Date
    Feb 2008
    Location
    Norway
    Posts
    59
    Rep Power
    7

    Default

    guess it should be 0 for full and 1-6 for diff, thaught it went from 0 to 6 not 1 to 7.. Anyhow, I see in the script something that makes me think you have or have had the same issues as me now.

    After a backup is done, every service run except the zmstat service.. I have tried to uncomment the 2 lines in your script regarding this:

    Code:
            # quick hack to start the stats component
            # echo
            # sudo -u $ZM_USER "$ZM_HOME"bin/zmstatctl start
    Also tried with changing sudo to use the su -c way, last night I even tried to set up a crontab for zimbra user like:

    Code:
    30 4 * * * /opt/zimbra/bin/zmstatctl stop && /opt/zimbra/bin/zmstatctl start
    It simply refuses to work, I have to manually via shell as the zimbra user

    Code:
    /opt/zimbra/bin/zmstatctl stop && /opt/zimbra/bin/zmstatctl start
    to get things going again... Found any solution to that one? or someone else with an idea for what to try?

  6. #6
    Join Date
    Feb 2008
    Location
    Norway
    Posts
    59
    Rep Power
    7

    Default

    Ahh nevermind, got it, I changed every(!) instance of sudo to su - zimbra -c way in the script, now it works fine...

  7. #7
    Join Date
    Jul 2007
    Location
    San Jose, CA
    Posts
    1,027
    Rep Power
    10

    Exclamation Vote on FOSS hot backups

    You guys who care about backup, here is a thread initiated by Zimbra that you really ought to vote on:

    http://www.zimbra.com/forums/announc...kups-foss.html

    Dan

  8. #8
    Join Date
    Sep 2008
    Posts
    11
    Rep Power
    7

    Default

    Hello,
    My thanks to the author for sharing this script with us, is very handy.
    I have installed it in ubuntu 8.04 LTS after executing the command
    ./zmbac.sh -f
    Everything goes fine except for the last part when stuck in
    Archive 10_Zimbra_Backup_11-March-2009_FULL requires a password:

    Any ideias?
    Thanks for your help.

  9. #9
    Join Date
    Jan 2008
    Location
    Germany
    Posts
    83
    Rep Power
    7

    Default

    Quote Originally Posted by 163828045 View Post
    Hello,
    My thanks to the author for sharing this script with us, is very handy.
    I have installed it in ubuntu 8.04 LTS after executing the command
    ./zmbac.sh -f
    Everything goes fine except for the last part when stuck in
    Archive 10_Zimbra_Backup_11-March-2009_FULL requires a password:

    Any ideias?
    Thanks for your help.

    Hi,

    Can you post the output of the script, I might spot something...
    did you run the INSTALL option? this looks like you are wanting to crypt the archive but don't have a passwd file for it yet.

    cheers
    heinzg
    Motions of a day at work -

  10. #10
    Join Date
    Jan 2008
    Location
    Germany
    Posts
    83
    Rep Power
    7

    Wink Call For Enhancement Requests

    I see there are quite a few people still using this script, and there are some bugs that need killing, and the code need a lot of cleaning, and spring is in the air (well almost anyway...) !!

    So here it is - CALL FOR ENHANCEMENT REQUESTS !!

    Anyone need something that the script don't do at the moment?

    Anyone going to share there fixes, to rid the script of a bug or two?

    Cheers
    heinzg
    Motions of a day at work -

Similar Threads

  1. Zimbra Backup Script ... Small Problem
    By frankb in forum Administrators
    Replies: 12
    Last Post: 07-14-2008, 08:40 AM
  2. Debian - Upgrade from 4.5.6
    By magikman in forum Installation
    Replies: 1
    Last Post: 01-19-2008, 11:59 AM
  3. Backup script issues
    By SSS in forum Administrators
    Replies: 18
    Last Post: 12-03-2007, 08:56 AM
  4. 4.0.4 to 4.5.6 upgrade failed in network edition
    By chenthil in forum Administrators
    Replies: 1
    Last Post: 08-27-2007, 09:36 AM
  5. [SOLVED] Simple backup question...
    By dameron in forum Administrators
    Replies: 3
    Last Post: 08-25-2007, 09:36 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •