How to properly backup Zimbra?

[paul]

I looked at snapshots, but saw a few comments that
snapshots were not a suitable long term backup mechanism.

Probably when they say it's not a long term solution, they mean live copy-on-write snapshots, and retaining them. That's true, because all write operations then have to be duplicated.

My system is basically what you have, but then without downtime, and more efficient because it doesn't store unused blocks from the file system. I use Xen and my disks are on LVM (logical volume management). With LVM, you can make in instant copy-on-write frozen snapshot of any volume. So, my scripts, in essence:

* Make a snapshot
* Take the first 100 MB and dd it: that will contain partition table and boot loaders.
* use losetup to scan for partitions on it.
* Do a filesystem repair, because it 'looks' like a uncleanly shutdown file system
* Use partclone on a designated partition in the snapshot, to only copy the used blocks to a gzip file
* Delete the snapshot

Your system looks pretty slick, although what's the restore operation look like? At the end of it... do you have
to piece together the GZIP files which contain sets of used blocks?

I normally use LVM, but for some reason this particular server does not have LVM. Not sure why.

Your comment about the snapshots looking unclean is something I ran across. I initially paused
the VM before copying, and of course that always needed a repair before restore. I was worried about
whether those repairs were guaranteed to succeed, so I elected to switch to shutdown where I had a
guaranteed clean FS. Sadly - I now get occasional restart issues .

Thanks for the insights.
Paul.

[halfgaar]

Moreover, the .tgz files produced by the scripts always have a reasonable size, there are no 10MB backup files of a 1GB mailbox.

You mean that data is not dropped when exporting; the size is the expected size? I'm not sure, because it was hard to piece together people's reports, but I've seen it said that it's the importer that drops the data.

It is still better than only making/restoring snapshots, that are a quite long task to perform...

The duration isn't really the issue. It's more that it's an all or nothing restore. I could spin up a restored snapshot without network and then export some elements I need, I guess. I never had to, but in a pinch, I think I could manage.

Your system looks pretty slick, although what's the restore operation look like? At the end of it... do you have to piece together the GZIP files which contain sets of used blocks?

I work as a devop, so what some consider user unfriendly, is second nature to me , but I have to: make a logical volume of the same size, write that first 100 MB into it with dd, so that I have the partition table and boot loader. (my backup routine also dumps fdisk output, so that I can confirm it's correct). Then I have to use 'losetup --find --show --partscan /dev/vgroup/newvolume'. This will create loop devices. Then I have to do (for example): 'cat vmsnapshot_partclone_loop0p1.gz | gunzip | partclone.ext4 --restore --source - --output /dev/loop3p1' or something to that effect.

The method was inspired by how Clonezilla works. That also writes dd of the first MB or so, output of fdisk, and partclone output.

Your comment about the snapshots looking unclean is something I ran across. I initially paused
the VM before copying, and of course that always needed a repair before restore. I was worried about
whether those repairs were guaranteed to succeed, so I elected to switch to shutdown where I had a
guaranteed clean FS. Sadly - I now get occasional restart issues .

Because the file system is journalled, the damage is minimal. What I do in my script, is actually first mount the snapshot (this replays the journal, better than fsck does), then unmount, then fsck, and if there are no errors in that, make the backup using partclone.

[paul]

Just in case anyone is interested in a simple outside-the-vm backup for VMs (like Zimbra), here is a copy of my script:
https://github.com/tussock/VirtualBoxBackup

It has the following advantages:
- VERY simple - it doesn't require @halfgaar's mad devops skills
- Restore is trivial - grab the last version and unzip/copy back
- Backs up all VMs, and keeps <n> copies.
- Zip and Encrypt of each VM
- Runs happily as a normal user via cron, if that user has access to the VMs and dest folder.
- Mails results.

And the following downsides:
- Not space efficient (a number of full zipped copies of the VM is kept)
- shuts each VM down for a few minutes to ensure clean file systems.

If you have any suggestions to improve this - they would be very welcome.
[One idea that occurs to me from talking with Halfgaar is to switch it back to "Pause"
rather than "Shutdown", and run an FSCK after copying.]

[halfgaar]

I feel that if you can write that script, you can also do what I typed, or automate that

Apparently, VirtualBox can use raw disk back-ends, so it should be easy enough for you to migrate to LVM using dd, except that your current disk layout may not support it, so you may need to take it offline and re-partition disks.

Anyway, one tip is that you can instead of use zip, use gzip, and then use the --rsyncable flag. You can then keep overwriting one file, which in turn you backup using something like rdiff-backup, which only stores the increments.

[paul]

I feel that if you can write that script, you can also do what I typed, or automate that

Yeah... old linux programmer here.
Not devops though... I look for simplicity.

Apparently, VirtualBox can use raw disk back-ends, so it should be easy enough for you to migrate to LVM using dd, except that your current disk layout may not support it, so you may need to take it offline and re-partition disks.

Yeah... I really should migrate this server to LVM. Its near the bottom of my backlog.

Anyway, one tip is that you can instead of use zip, use gzip, and then use the --rsyncable flag. You can then keep overwriting one file, which in turn you backup using something like rdiff-backup, which only stores the increments.

I use pigz for performance at the moment... but I like where you are going and pigz supports rsyncable. I have an extra requirement of encryption, but I can mount encfs over sshfs and backup to that.

[ibertrix]

This thread is a bit old, but I think it deserves a continuation.

The Zmbackup tool (actually written in Bash) uses the REST interface for account export, and that has been broken and will remain broken, it seems. See this bug report: REST interface has critical flaws. I wrote a short script using that too, but I recently found it that is useless.

I've been Googling for backup strategies/scripts, and I find it all very rudimentary, and when it comes to disaster recovery, mostly inferior to my virtual machine snapshots.

I'm left with the conclusion that there is no per-account, or even per message, backup method? For instance, if someone deletes some e-mail accidentally, and needs to perform a partial recovery, you just can't?

Since there are no alternatives at the moment, I'd say that a strategy composed by VM or LVM snapshot + Zmbackup/REST scripts is still an "ok" backup strategy: yes, it is not guaranteed that 100% of mailbox elements are backupped by the scripts, but in my experience I did not have bad experiences using the scripts. During the years I (luckily) did only a small amount of restores and users never complained about missing elements...Moreover, the .tgz files produced by the scripts always have a reasonable size, there are no 10MB backup files of a 1GB mailbox. It is still better than only making/restoring snapshots, that are a quite long task to perform...

I am landing to this thread because I used to use zimbashckup.sh and when I tried to restore data I got a bunch of errors and was not able to repair a broken mailbox of an account.

What LVM strategy you use? Snapshotting /opt/zimbra/store ?

Can you use zmbackup in cronjobs?

[halfgaar]

What LVM strategy you use? Snapshotting /opt/zimbra/store ?

I outlined it some post back, but in short: the entire virtual machine is on an logical volume, which I snapshot and backup. But, your issue emphasis the important of being able to restore one account, which is kind of tricky with the FOSS version, because the account exporter (or the importer) is broken.

[paul]

Sorry to rekindle a dead thread, but I wanted to let anyone interested know that there is a second
version of the "outside the VM" backup script - this one pauses the VirtualBox VMs (rather than a full shutdown)
as it backs up.

Pausing has several advantages:

• less downtime (no need to shutdown and startup).
• you dont need to implement acpipowerbutton support in the VM
• I found the daily shutdown of Zimbra to be unreliable - occasionally there would be a restart issue. Pause seems to be more reliable.

It does need to perform an fsck on the VDI while backing up, but so far that has proven reliable.

You can find it here:
https://github.com/tussock/VirtualBoxBackup

Comments welcome.

[10424bofh]

there basically 2 ways

way1: in case your zm shutdown still works (many people have the issue that it hangs for a long time on ther webapp part)

i use my own online backupsolution for this, rsync script would work too, but id rather have notifications if a backup took place or if not.

make a hotbackup before backup time, or maybe ven several times over the day.
at backuptime - i set my max backuptime to 5 minutes no matter what
shutdown zimbra
make a incremental backup (should be very quick as most data is already here thanks to the hotbackups from before)
restart zimbra

downsides:
-only full server restore. to restore an individual mailbox or email it would require a full restore. you may have a box on standby todo such things.
-so in essence only disaster recover
-unreliable zimbra shutdown and start so server may be down in the morning

way2: VM snapshots
i highly recommend to either use lvm or even much better zfs for that matter
simply snapshot with freeing the vm. over 1gbps 100gbyte fullbackup should take only 15 min. incremental ofc even less
with zfs you can do much much more, like livesync that thing all the time to a different box. also id recommend zfs because its self healing, dataintegrity and buildin raid funcitonality completly outperforms mdma raids and lvms in every regard. also its handling is excelent for kvm virtualisation.

downside: also only a disaster recover.
nothing else