Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: Remote mirroring of /opt/zimbra/backup dir using rsync over low-bandwidth link

  1. #1
    Join Date
    Apr 2010
    Posts
    14
    Rep Power
    5

    Default [SOLVED] Remote mirroring of /opt/zimbra/backup dir using rsync over low-bandwidth li

    Hello all,
    I've just recently changed up my Zimbra backup strategy. Previously, I was just mirroring the entire /opt/zimbra directory and had disabled the backup mechanism that is included with NE. However, I found that this can lead to inconsistencies within accounts when a backup is restored, so I've now re-enabled the native NE backup to /opt/zimbra/backup and am trying to mirror this directory using rsync.

    My problem is that the sheer volume of data created by the NE backup process is overwhelming my low-bandwidth rsync setup. The rsync goes thru a VPN to an off-site location, but since the locating housing our Zimbra server only has ~2.5 MBit upstream, the transfer speed is very limited.

    Each full backup is about 65G, which would take around ~25 days to transfer over the given bandwidth. Daily incrementals are closer to ~200MB, which would not be an issue to transfer daily.

    My question is: What are some procedures being used to mirror the /opt/zimbra/backup directory to an offsite location? We're already using the compression built in to rsync, but this provides little relief. While it would be technically possible to have one full backup and daily incrementals indefinitely, I find this a very disturbing solution.
    Is there any solution to help rsync in dealing with full backups? As noted by the size of our incrementals, the daily changes are ~200 MB. If we could somehow have rsync only transfer the *changes* between the full backups every weekend, while doing daily transfers of the incrementals, this would be an optimal solution.

    Has anyone else had to deal with mirroring of /opt/zimbra/backup over low-bandwidth connections?
    Last edited by Apple_Eater; 11-21-2011 at 11:13 AM.

  2. #2
    Join Date
    May 2010
    Posts
    272
    Rep Power
    5

    Default

    Hello,

    Uhm i know your Pain. I actually dropped the NE Hotbackup feaute at all cause i still dont believe in its implementation.

    So first instead of rsync maybe a clusterfilesystem would work better ... anyway

    i went now to a dirtyhot / cleancold backup mix of simply the hole zimbra directory.

    so what i do is start with the first d.hotbackup at for exampel 10am - maxruntime 5 hours
    at 3:30pm - second run d.hotbackup - max runtime 1 hour
    at 5pm - shutdown - coldbackup

    (ok its in the night but the time in between is correct)
    i use the same backupset for that and my own onlinebackup solution and save incremental and encrypted

    to determine when todo a cold backup i use a simple shellscript as a prebackup /postbackup command
    it does nothing else than look at the clock and make a shutdown at >=5pm and <6 and an startup at >=5pm and <11pm

    that way ive only to backup a minimum at coldbackup so only a downtime of 3 minutes and a minimum of datatransfer
    also i do not need twice the amount of data on the zimbra server

    My backup is compressed encrypted less that 50% of total zimbra usage so its the absolute minumum resources usuage i could do without compromising security

    since we saw thats not a bad thing to reboot the zimbra frequently and 3 minutes in the night is something everyone can life with its the best solution atm at least for me

  3. #3
    Join Date
    Apr 2010
    Posts
    14
    Rep Power
    5

    Default

    Have you done any test restores using this method? I unfortunately know very little about the inner workings of Zimbra, so I am not sure how it might handle any inconsistencies, i.e. with the dirtyhot backup. Previously I had just been using rsync to mirror the /opt/zimbra directory with the system online every 12 hours. It worked well, data transfer was small, etc. But I was worried about integrity.

    However I like your idea of a short downtime and doing a cold backup. So perhaps I could shutdown all zimbra services for ~10 minutes, create a tgz backup or something, and use rsync's difference transferring algorithms to do the transfer?

    I would really like to stick with NE's native backup methods if possible though. While I don't mind cooking up my own solution, I like to go with the "tried and true" tested, supported solutions when available. We paid a lot of money for the Network Edition, so I hate to waste one of it's more useful features!

  4. #4
    Join Date
    May 2010
    Posts
    272
    Rep Power
    5

    Default

    No look,

    You cannot restore the hotbackups - i mean you could but trouble will be comming for shure

    The Hotbackups are only made to reduce the downtime during the coldbackup
    so i make 2 hotbackups to collect all the fixed data - which is not chaging till cold
    then when the coldbackup starts theres only a little data to collect - usually a few hundred megs - its done within 8 min max including shutdown and startup zimbra - of course how long it takes for you its a mtter of your line


    the real backup IS ONLY the cold one

    - you cannot rsync one tar file
    however theres a backupscript for rsync rsyncing the hole backup for you
    thers also a script called ftplicity which could do that - that script actually
    make an incremental copy - split up in junks - encrypt em and upload em


    however i do not recommend those scripted options
    i really dont - no script backup is really relyable if you do not watch it every day

    mayn things can go wrong and you get no information/no logs no overview
    so id recommend using a prof. backupsolution if you do what i do
    otherwise stick with ne backup

    but take a look at the OSS backup solutions by script
    theres also one which mirror the zimbra dir on harddrive
    it backups hot first - then shutdown zimbra - rsync the differences between hot and shutdown state - restart zimbra

    now you got a zimbradir to comfortly backup offsite by rsync with a minimum downtime

    however as i stated more than once - keep watching every day
    i often run into troubles with such scripted solutions - and very often i had to purge everything and start over



    btw my backup are 100% relyable - i had a few restores - about 1 in every 60 days because a user killed his mailbox

    i restore em to a virtual host - which is setup in a way i can restore the last backup and instantly start zimbra to backup that mailbox and restore it to production

    how ever my luck is our backup server uses also an server - server replication
    so the hole backup server is mirrored and one backupmirror hangs here in my company
    so the restore of that backup not long over gigabit ethernet


    i know ne backup can restore a single mailbox - but i dont want to use both
    (to much resources /diskspace bandwith cpu) for little benefit
    because ne backup is nothing more than a scripted solution and does not have features a real backup solution have


    i also tell you something from 15 years expierence
    script may work - may work relyable for a time but it complicate things
    after a few years your Infrastructure endup in a very complicated setup where little can break down much

    keep it simple keep it clean - it will get complicated by itself over the time


    i use a lot of time to replace still working solutions with newer simplier once to reduce the amount of complexity frequently

    also virtualisation dont make things better it makes it worse (on the complexitiy level)

  5. #5
    Join Date
    Apr 2010
    Posts
    14
    Rep Power
    5

    Default

    Bump -- still looking for an rsync solution that leverages NE's native backup functionality. I'm sure I can't be the only person that uses rsync to move the NE backups offsite?

  6. #6
    Join Date
    May 2010
    Posts
    272
    Rep Power
    5

    Default

    Im sorry but im not shure you really should do that since your knowlege lacks a bit in that topic und such solutions arent easy to maintain.

    however youre not looking for rsync - rsync can only sync files - not binary deltas - you need rdiff or dulicity or something like that

    those are rsync variants who can create delta file.
    deltas are differential files between file A and B
    so you have to initially transfer those files and backup alter only binary differentials


    and still this is a problem.
    ne backupfiles ae new every day - for the bandwith you describd to much
    rdiff backup on zimbrafolder itself = need coldbackup

    only solution - i also ready told you - take a look on those backupscripts for OSS

    they create a local copy with rsync while coldbackup to reduce downtime and then sync it up the differentials - still those files can be huge so im not shure if they finish in time

    even worse - if you get a shedule problem (sync take to long) youre get easy into troubles

    but really i dont get it - you spend a decnet amount on the NE version but wanna save money on backup?

    why dont you simply rent a cheap dedicaded server somewhere for a few bucks and park data there or you go professional and take a decent backupsoftware you can save your trouble (or a decent cloudbackup provider)
    mostly cheaper than a regular tapebbackup cost (you have to change one tape a month minimum with a 5+1 rotation so cost are min 100$ a months for tapebackup plus drive and software)

    a cloudbackup gonna cost you around 80-120 bucks including space a month ...
    way better than never be shre if your ass is save

  7. #7
    Join Date
    Apr 2010
    Posts
    14
    Rep Power
    5

    Default

    This backup isn't about saving money, this is the only way I am aware to do it. We have to get this backed up offsite somehow, and our internet connection (the fastest available at our location) is only 2 Mbit upstream. For this same reason we couldn't have the email system hosted offsite because our employees will frequently exchange large (50-100mb) emails with eachother internally, that would take a very long time to send to a remote server just to download it again.

    If I am missing something obvious please let me know, but the way I see it, the data has to get offsite "somehow". I already backup our 2TB+ datastore with unison to the same offsite location, but because of how the NE backups work, this doesn't really work for them (since they have a different name/path each time).

  8. #8
    Join Date
    May 2010
    Posts
    272
    Rep Power
    5

    Default

    Yes youre missing something obiviouse
    My posts

    ive already written so one more time ill try it again....

    First you cant rsync zimbra backupfiles - reason we both know
    so you got 2 possible solutions

    #1 the good one
    Using a professional backup which can do incremental and infile delta backups

    how does that work - first you make a incremental delta of the zimbra directory while zimbra is running

    maybe you wanna do this twice or 3 times

    then you make a copy wihting THE SAME backupset of the zimbra directory
    but this time you use a prebackup command to shutdown zimbra


    that way - the cold backup (when zimbra is down) only transfer the infile delta difference between the last hot backup and the state when its down

    you can do this 2 way - one your backup software can do multiple shedules with individual preback post backup commands
    #2 in case your software cannot do individual post / pre backup commands
    you use my little shellscripts


    why?
    that way you make for example 3 times an incremental backup
    1am - 4am - 5am
    but the script will be set to 5 am shutdown
    which create you automatically 2 times hotbackup (to collect data)
    1 time cold backup - to make the real consistent backup



    solution#2 - not so good but should work
    use one of those OSS backup script in the zimbra wiki
    it does the following

    it will hotback zimbradir to another dir (to reduce the amount of data)
    then shutdown zimbra
    rsync the diff to the another dir (now you got a local backup for real)
    startup zimbra (now we got a lot of time)

    rsync offsite



    ----------
    however in both cases of course the total anmoutn of data generated each day should not bigger than what you can transfer (obivisouly)
    second method can have a lot of drawbacks you never reckognize

    also consider a solution which CAN reduce the bandwidth backup uses
    otherwise you may block your line totally


    im using a solution can do both - but of course i got a big upstream so i dont have to bother about how long the hotbackup will take

    but in case somethigng goes wrong i make the following to reduce the downtime

    1 make 1 big hotbackup - sheduled for max runtime 5 hours
    2 hours after the max runtime is ended another hotbackup - max runtime one hour (this should backup all new files some crazy users left behind)

    3rd now we make the coldbackup - but just in case max runtime to 1 hour
    in case that it will take longer i got here the fallback so zimbra is down max 1 hour in worst case szenario (which should NEVER happen but who knows)

    worst case will be one backup not fully finished but zimbra is up and still some data to recover in case by the hotbackups


    i hope i could describe understanbale what / how and why im doing and how this works


    so now we calculate your data/hour trougput
    lets say we make a max of 1 mbit for backups
    means 8 seconds / MB gives us 7.5 MB / minute and 450MB per hour

    that should be sufficent
    so if you stick to the almost ident backupplan as i do you should be able to backup alsmost 3 gig during hotbackup and the rest during cold


    so again in short

    one backupset for all 3 backups below

    Backup #1 - hotbackup (precollect data) 1am runtime 5 hours (till 6am)
    Backup #2 - hotbackup (precollect data2) 6:30am runtime 1 hour (till 7:30am)
    Backup #3 - coldbackup (our real backup 8am - our script above will now shutdown zimbra because its 8am - max runtime 1 hour just in case its to much we sacrifice a backup for uptime)


    amount of data being backed up - 3.07 gigabyte


    pps: why im doing the second hotbackup and not one with 6 hours -
    it could be that the first hotback end after 30 minutes
    but 2 hours later some big junks comming in
    then the coldbackup would take to long - so the second hotbackup is another failsafe)
    Last edited by bofh; 10-27-2011 at 04:43 AM.

  9. #9
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    9

    Default

    Quote Originally Posted by Apple_Eater View Post
    Bump -- still looking for an rsync solution that leverages NE's native backup functionality. I'm sure I can't be the only person that uses rsync to move the NE backups offsite?
    It would be helpful to post what your actual backup command is in your zimbra crontab, and what your rsync command is.

    If you use the --noZip option, then the fulls will be stored as uncompressed file/directory hierarchies, which allows hardlinking of duplicate data across backups. While this does mean that rsync has more things to compare, it also means that less data ultimately has to be sent over the wire between full backups.

    Make sure you use the -H (or --hard-links) option in rsync so that hard links are preserved.

  10. #10
    Join Date
    Apr 2010
    Posts
    14
    Rep Power
    5

    Default

    From crontab -l as zimbra user:

    # BACKUP BEGIN
    0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all --mail-report
    0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i --mail-report
    0 0 * * * /opt/zimbra/bin/zmbackup -del 1m --mail-report
    # BACKUP END

    The --noZip option sounds perfect, is there a way to implement it via the GUI? Or at least set a persistent setting so that it will survive upgrades?

    Also the current behavior (with the commands listed above) is already to store the full directory structure in an unzipped format, however no hardlinks were immediately evident in the directory structure.

    Also:
    zmcontrol -v

    Release 6.0.10_GA_2692.UBUNTU8_64 UBUNTU8_64 NETWORK edition.

    We haven't made the jump to the 7.x series yet because we haven't seen any seriously compelling improvements to warrant the switch -- has the backup system had significant changes in the new series that might be useful to our situation?

    Thanks for the help!

Similar Threads

  1. [SOLVED] rsync permision denied
    By get2guy in forum Administrators
    Replies: 4
    Last Post: 10-22-2010, 01:32 AM
  2. 4.5 /opt/zimbra/backup dir
    By slally in forum Installation
    Replies: 8
    Last Post: 02-06-2007, 04:14 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •