ZCS-NE : rsync /opt/zimbra/backup

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
breverend
Posts: 21
Joined: Sat Sep 13, 2014 12:29 am

ZCS-NE : rsync /opt/zimbra/backup

Postby breverend » Fri Dec 11, 2009 4:19 am

Hello,
I need to keep offsite copies of my zimbra full and incremental backups with no time limitation.
For now, I use a cronjob every night calling a script :
/usr/bin/rsync -aHvz myzimbra.mycompany.com:/opt/zimbra/backup /local/dir/
It does work but it's eating my disk space, as each new full backup takes 18 GB instead of less than 2 on the server.
It's not keeping the hardlinks. I was thinking about use the --link-dest option of rsync, and filter on full backups only, but the "small full backups" (made when you create a new account) kill this solution.
So, anyone have experience with offsite copies of /opt/zimbra/backup ?
Thanks


dougz
Posts: 31
Joined: Fri Sep 12, 2014 10:43 pm

ZCS-NE : rsync /opt/zimbra/backup

Postby dougz » Fri Dec 11, 2009 7:04 am

i use rsync+tar/gz+sshfs/fuse to backup to another server.

sshfs act like nfs, but over ssh.
CAUTION: sshfs !! not shfs. shfs is very slow for huge files.

You can use tar to insert the data on stdin and redirect for a ssh connection if you dont want to compile a kernel module like sshfs.
Useful link:

SSH Filesystem
sample usage:

#create a local dir to map the remote dir

mkdir /var/tmp/tmprfs
#mount the remote dir

$BPATH/sshfs $REMOTESERVER:$REMOTEDIR /var/tmp/tmprfs -o reconnect
#use tar and gzip to compress and redirect to remote dir:

$TAR -zcvf /var/tmp/tmprfs/$BACKUPFILE -T $SELECTION --exclude-from=$EXCLUSION > /var/tmp/tmprfs/rel.ok 2>rel.err
#the errors are redirected to rel.err file, and you can check the rel.ok to view all files on backup
ewilen
Elite member
Elite member
Posts: 1429
Joined: Fri Sep 12, 2014 11:34 pm

ZCS-NE : rsync /opt/zimbra/backup

Postby ewilen » Fri Dec 18, 2009 5:27 pm

[quote user="breverend"]Hello,
I need to keep offsite copies of my zimbra full and incremental backups with no time limitation.
For now, I use a cronjob every night calling a script :
/usr/bin/rsync -aHvz myzimbra.mycompany.com:/opt/zimbra/backup /local/dir/[/quote]Is /local/dir/ always the same? If not then I think you will be copying all of /opt/zimbra/backup every time that you use a new /local/dir/
Also, your local machine is Unix/Linux, right? I'm not sure if rsync on Windows will handle hardlinks properly.
This might interest you: rsnapshot
[quote
, anyone have experience with offsite copies of /opt/zimbra/backup ?[/QUOTE]Sort of...I've used rsync in the past to test various options for maintaining a cold standby (which I hope to implement before year's end). It seemed to work fine but I wasn't dealing with a great deal of data.
jsturtevant
Posts: 10
Joined: Fri Sep 12, 2014 10:22 pm

ZCS-NE : rsync /opt/zimbra/backup

Postby jsturtevant » Fri Dec 25, 2009 3:51 pm

this would be useful to add to the wiki as storing the backups on another machine is desirable.
i'm using: rsync -avzp --delete /opt/zimbra backup-01:/usr/local/backup/mail-02
any comments as to whether these are the best rsync options to mirror the /opt/zimbra/backup on another machine?
mlanner
Advanced member
Advanced member
Posts: 111
Joined: Fri Sep 12, 2014 11:13 pm

ZCS-NE : rsync /opt/zimbra/backup

Postby mlanner » Sat Jan 09, 2010 10:51 pm

I was wondering the same. What about tossing an -H in there for preserving hardlinks? Like:
rsync -avzpH --delete /opt/zimbra backup-01:/usr/local/backup/mail-02

I noticed they were using that here:
Moving ZCS to Another Server » Zimbra :: Blog
gdr777
Posts: 8
Joined: Sat Sep 13, 2014 12:36 am

ZCS-NE : rsync /opt/zimbra/backup

Postby gdr777 » Thu Jan 14, 2010 8:02 am

Rather than copying/rsyncing the backup files, why not just create a remote mounted directory where all of the backups are initially created. sshfs could be used if the remote host is across an untrusted network. To keep zimbra from deleting backups older than 1 month, edit the crontab that controls backup, and comment out the 3rd line in the backup section.
uxbod
Ambassador
Ambassador
Posts: 7811
Joined: Fri Sep 12, 2014 10:21 pm

ZCS-NE : rsync /opt/zimbra/backup

Postby uxbod » Thu Jan 14, 2010 8:07 am

The problem with that is it would not keep any manually edited configuration files eg. spamassassin which is why people rsync against /opt/zimbra.
gdr777
Posts: 8
Joined: Sat Sep 13, 2014 12:36 am

ZCS-NE : rsync /opt/zimbra/backup

Postby gdr777 » Thu Jan 14, 2010 8:23 am

True, but the original poster was having issues w/ preserving hard links in /opt/zimbra/backup. Relocate the default backup location /opt/zimbra/backup outside of /opt/zimbra. /opt/zimbra can then be rsynced/backed up however one wants w/o backup tree getting in the way.
User avatar
L. Mark Stone
Elite member
Elite member
Posts: 2028
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine
ZCS/ZD Version: 8.8.12 Network Edition
Contact:

ZCS-NE : rsync /opt/zimbra/backup

Postby L. Mark Stone » Thu Jan 14, 2010 9:22 am

rsync also has an --exclude switch, which can be used multiple times in a commandline.
Our best practice is to have /opt/zimbra/backup be on a separate ext3 formatted JBOD. That way, if (sorry, when -- all hardware eventually fails) the Zimbra server blows up, you can simply attach the JBOD to the replacement server.
If you run a cron job to rsync things like /opt/zimbra/conf to, say, /opt/zimbra/backup/zcs-conf periodically, you are covered.
So, if you keep a spare chassis handy with an operating system already installed and your production Zimbra server blows up, you can be back up and running very, very quickly.
Hope that helps,

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/
breverend
Posts: 21
Joined: Sat Sep 13, 2014 12:29 am

ZCS-NE : rsync /opt/zimbra/backup

Postby breverend » Thu Jul 01, 2010 9:18 am

Thanks all for your answers.

I'm getting back on this issue. Let me be more specific :
I have 1 remote server which is my production Zimbra. Let's call it "server"

The install is standard, and I'm running ZCS 6.0.6 at the moment. I have around 30 accounts.

I also have an "office" server, which I can access physically. Let's call it "backup"
What I want to do is just replicate the content of

server:/opt/zimbra/backup

to

backup:/backups/zimbra/backup
So inside of those dir, I would have the same tree :

accounts.xml sessions tmp
And inside backup:/backups/zimbra/sessions, I want to have all full and incr backups, i.e. those on server:/opt/zimbra/backup/sessions + those which have been deleted over time.
Currently, I'm using :

on "backup" :

rsync -aHvz server:/opt/zimbra/backup /backups/zimbra
Now, maybe I'm just reading wrong info from df and du commands, but those backups are really taking a lot of diskspace :


root@server:/opt/zimbra/backup/sessions# du -hs *

22G full-20100604.230010.263

612K full-20100610.230034.511

23G full-20100611.230008.043

23G full-20100618.230010.257

620K full-20100623.230039.677

23G full-20100625.230010.855

89M incr-20100531.230011.174

58M incr-20100601.230010.711

109M incr-20100602.230012.958

63M incr-20100603.230009.654

62M incr-20100605.230011.670

8.0M incr-20100606.230011.065

55M incr-20100607.230012.156

111M incr-20100608.230011.642

78M incr-20100609.230009.626

64M incr-20100610.230010.017

51M incr-20100612.230017.684

25M incr-20100613.230010.515

106M incr-20100614.230011.702

54M incr-20100615.230017.320

94M incr-20100616.230015.921

113M incr-20100617.230010.885

218M incr-20100619.230014.408

15M incr-20100620.230010.343

119M incr-20100621.230014.579

85M incr-20100622.230016.570

81M incr-20100623.230012.822

198M incr-20100624.230010.893

108M incr-20100626.230014.666

9.8M incr-20100627.230010.251

221M incr-20100628.230011.867

182M incr-20100629.230009.865

298M incr-20100630.230011.408
root@server:/opt/zimbra/backup/sessions# df -kh

Filesystem Size Used Avail Use% Mounted on

[...]

/dev/md1 412G 124G 268G 32% /opt
root@server:/opt/zimbra/backup/sessions# du -hs /opt/zimbra

123G /opt/zimbra


There's almost only zimbra in /opt (rest is less than 1GB).

Why are those backups taking so much space ?

Over a year, it will eat 50*20GB, more than 1 TB of backups, just for less than 30 ppl ? Sounds a bit much.

I also recall seeing some 1 or 2 GB full backups a few month ago. Maybe using the wrong command ?
Anyway, as you can see, I'm loosing control over the situation (as I nearly have no more space left on my "backup" machine), so if you think I did something wrong, just tell me. For the record, I've got less than 1 year of experience on Zimbra, and my time is not fully going to it :)
Thanks !

Return to “Administrators”

Who is online

Users browsing this forum: MSN [Bot] and 7 guests