Results 1 to 10 of 11

Thread: ZCS NE Backup - Archiving To Tape Is To Slow

Hybrid View

  1. #1
    Join Date
    Sep 2006
    Location
    Illinois
    Posts
    374
    Rep Power
    9

    Question ZCS NE Backup - Archiving To Tape Is To Slow

    Hi,

    I've been perusing all the backup documents in the wiki and have searched through a lot of the forum discussions regarding backup. I haven't seen any one who indicates they are having the problem we are having though.

    We would like to get tape backups of the Zimbra backups (/opt/zimbra/backup), but in it's native state it is just too slow. We think the problem is that the Zimbra backup is producing thousands (probably tens of thousands or more) of small files....and it is negatively impacting the performance of our tape backups.

    We currently have a Sun Library with 2 LTO3 drives in it, and this is sitting on 4GB fiber channel. The speed of the hardware is not the issue. We can tar a single large file off to the tape drives at 50-80MB/s. Trying to do the same with a very large subdirectory of Zimbra backups gets us results in the 10-15MB/s range initially, but gradually dropping as time goes on. At times we can see it get well below 1MB/s.

    We have 3 mailstores, each with about 250GB of backups on it. Later this Spring we will be adding 2 more mailstores, 13,000 or so users, and probably almost double our backup data(1.5-2TB). We can't realistically back this up at those slow speeds. But that's not the worst part...restores go so much slower than backups that it would probably take weeks to restore these systems if we had to come back from a major disaster.

    Our current workaround is to tar /opt/zimbra/backup to another LUN on our SAN, and then backup that single tar file to tape(one for each mailstore)... which goes much faster. This however requires us to have double the storage requirements to hold all the backups and the nightly tars.

    So...what is the answer? Has anyone else in a large enterprise environment run into this and found a workaround? I saw somewhere that we could use the '-z' parameter on zmbackup, to zip up the blobs. Will that reduce the number of really small files that get created on a zimbra backup? If so that might solve the problem for us....but I'm not sure. Anyone have any thoughts or discovered a good (very fast) way to archive your Zimbra backups off to tape?

    Thanks,
    Matt

  2. #2
    Join Date
    Jul 2007
    Posts
    11
    Rep Power
    8

    Default

    We were running into backup issue with the Veritas Netbackup solution we have. Our backup infrastructure is pretty robust but it was having trouble backing up the /opt/zimbackups directory. What we did was create separate policies for each /opt/zimbackups/backup/sessions/full*, /opt/zimbackups/backup/sessions/incr*, and then all the rest of the /opt/zimbra directory. This seems to have worked out for us but we are still looking for cleaner ways to do this. One idea we are still looking at is possibly doing snapshots of the backup directory through the SAN instead of the netbackup agent.

    I also would be very interested in finding out if the "-z" option would make things better for backups.

    Vince

  3. #3
    Join Date
    Mar 2006
    Location
    Beaucaire, France
    Posts
    2,322
    Rep Power
    13

    Default

    AFAIK, the "-z" option compresses blobs while it seems your backup problems are related to the number of tiny files to backup...

  4. #4
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    719
    Rep Power
    9

    Default

    I see limited advantage to snapshots in the SAN. The Zimbra backup directory is essentially append-only, so except for a modicum of protection against file system corruption and insider vandals (and that's assuming the vandals can't also kill the snapshots), SAN snapshots are little different than simply extending the retention beyond the default 30 days.

    If you need tape for longer-term archiving, look for a backup product that behaves more like "dump" than like "tar." It's usually an expensive add-on, unfortunately. Veritas seems to call theirs "Block-Level Incremental Backup (BLIB)."

    If you think you need tape for offsite DR, consider putting your backup directories offsite instead. My /opt/zimbra/backup is SAN-mounted 5km away. I have not done it yet, but what I plan to experiment with is two /opt/zimbra/backup directories on different hardware in different physical locations. All incrementals would need to be synced to both, but the destination of the full backups would rotate.

  5. #5
    Join Date
    Oct 2007
    Location
    New York City - Hell's Kitchen
    Posts
    17
    Rep Power
    8

    Default backup solution

    look into bakbone software...as i was running into similar issues.
    i move about 2TB a day.

  6. #6
    Join Date
    Sep 2006
    Location
    Illinois
    Posts
    374
    Rep Power
    9

    Default

    Bakbone=Netvault Backup?

    We're running an enterprise solution now, Syncsort's Backup Express, but disk to tape speeds won't go any faster than what raw tar can do. Maybe we need to look at a disk-to-disk solution (which Backup Express has but we haven't really used), and forget about the tapes?

    They also have the option to do block level backups, which might work? But they are only supporting Solaris and Windows currently with that feature...and have said Linux support is coming this spring...but I'm not holding my breath. Plus it's an expensive addon.

    What is everyone else doing?

    Matt

  7. #7
    Join Date
    Oct 2007
    Location
    New York City - Hell's Kitchen
    Posts
    17
    Rep Power
    8

    Default netvault

    correct bakbone=netvault.

    typically I run a cronjob nightly on the /opt/zimbra/backup directory syncing the folder to another disk.
    for live backups you have me there.

  8. #8
    Join Date
    Jul 2007
    Posts
    6
    Rep Power
    8

    Default Archiving to tape: Compressing the session folders

    I've added a bug/feature request that seems to address this thread.
    The feature I would love would be for each session folder in the backup folder to be gziped or tarball compressed. This way, archiving to tape would be MUCH more efficient. It would be great if both the Zimbra backup and restore features were 'compression' aware, so you could restore from a compressed session folder.

    Here's the bug/feature I added. Please comment and vote if appropriate.
    Bug 31748 – Backup Session Compression

    Thanks!

    Jonathan Perel

Similar Threads

  1. Replies: 210
    Last Post: 01-17-2012, 01:19 AM
  2. Trouble Sending mail - All Messages deferred!
    By SiteDiscovery in forum Administrators
    Replies: 7
    Last Post: 09-03-2009, 05:52 AM
  3. Replies: 41
    Last Post: 10-29-2007, 03:36 PM
  4. FYI: ZCS NE backup to fuse/sshfs mount, worked.
    By jagipson in forum Administrators
    Replies: 0
    Last Post: 09-28-2007, 07:37 AM
  5. ZCS 3.2 Beta Available
    By KevinH in forum Announcements
    Replies: 31
    Last Post: 07-07-2006, 04:46 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •