[SOLVED] Yet Another Backup Script for Community Version

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Postby LaFong » Thu Jun 02, 2011 3:16 pm

[quote user="7225123456789"]Thank you for your time.

At this moment nothing happed yet. Thanks God. But, just in case of ... you know ...

[/QUOTE]
Ah, good. Yes, you have to have been running the DB add-on to the main script to allow restoring of users or folders. (Though theoretically you could configure a temp server with a backup copy of all of zimbra, run the DB script on that to recover SQL, LDAP, messages, indexes, then take those and restore on your production server. That would be a pain.)
I have changed the link in the instructions again. It's hard to find a free, anonymous place that will store your files indefinitely. Here is the link so you don't have to search in the original instructions:

zmDBbac.zip


7225123456789
Posts: 3
Joined: Sat Sep 13, 2014 2:03 am

[SOLVED] Yet Another Backup Script for Community Version

Postby 7225123456789 » Fri Jun 03, 2011 9:43 am

[quote user="liverpoolfcfan"]While my zimbra server has lots of space for keeping multiple weeks of backups, the remote server I use for offloading a copy does not. Twice I have ended up filling the hard drive on the remote system - causing problems on that server. So, I decided to see if I could figure out a way to make sure it doesn't happen again. This works for me, hopefully it might be of use to some of you too. It uses awk, and will only work if the remote system is also a linux system (I use it on CentOS)
At the top of zmbak.sh - find the section

#--- SSH REMOTE DR COPY ---#

# This option will secure copy your archives to a remote server via 'scp'

DRCP="yes" # valid answers are "yes" or "no"



and add two more lines at the bottom of the section

REMOTEVOL="VolGroup00-LogVol00"     # volume on remote host where backups will be saved 

REMOTEMINFREE=200000 # Required minimum 1K disks blocks free after copy (Approx 200MB in this case)


Then, search to near the bottom of the script where you will find

	# DRCP Section. To scp newly created archives to a remote system

if [ "$DRCP" = "yes" ]

then



After the "then", add the following code
		echo "Checking disk space on volume [$REMOTEVOL] on host [$REMOTEHOST]"

needed=$(find $ARCHIVEDIR -name "$ARCHIVENAME*" -exec ls -sk {} ; | awk '{ s+=$1 } END { print s }')

avail=$(ssh $SSHUSER@$REMOTEHOST df -P -k | grep -i $REMOTEVOL | awk '{ print $4 }')

surplus=$(expr $avail - $needed)

if [ $surplus -ge $REMOTEMINFREE ]; then

echo "Avail:"$avail "- Needed:"$needed "- Surplus:"$surplus "- MinFree:"$REMOTEMINFREE "- OK TO COPY"

echo "Begin copy ..."

fi

if [ $surplus -lt $REMOTEMINFREE ]; then

echo "Avail:"$avail "- Needed:"$needed "- Surplus:"$surplus "- MinFree:"$REMOTEMINFREE "- CANNOT COPY"

echo "Error NOT ENOUGH SPACE AVAILABLE to copy backup files to $REMOTEHOST"

exit 1

fi




This code will get the total number of 1K blocks needed to copy the current backup, and then the total number of 1K blocks available on the specified volume on the remote host. It then subtracts the needed from the available to give a surplus (which could be negative of course) and checks to see if that surplus is larger than the configured value for the minimum number of free blocks on the remote volume.
NOTE: REMOTEVOL="VolGroup00-LogVol00"

This does not need to be the complete volume name as listed by df (in my case that is /dev/mapper/VolGroup00-LogVol00) - it just needs to be enough to uniquely identify the volume from the df output.
TODO: If anyone knows a way to find the remote volume name using the remote directory name - the configuration could be simplified further to just adding the Minimum Free Blocks setting. - Please reply back if there is way to do it.[/QUOTE]
Do you use filesystem quota ? You should ... anyway you can run this script


:

# @(#) -- show the filesystems short of space - JMS 03/06/2011 v1.0

#
#!/bin/sh
# #########################

# # Check Disk Space Utilization #

# # if >= 80% then alert sysadmin #

# #########################
# NOTE
# at the prompt of your tty type

# the command what

# and the name of this script whithout the extention .sh

# exemple -> $ what alertfreespace

# That's why the first 5 lines are so important


LOGDIR=`pwd`

LOGFILE=${LOGDIR}"/alertfreespace.log"

#LOGFILE1=${LOGDIR}"/tmp.log"
STDOUT=${LOGDIR}"/tmp1.log"

MAINTLOG=${LOGDIR}"/tmp2.log"
LOGFILEALL=${LOGDIR}"/resumo_all.log"

DATA=`date +%d%b%Y_%Hh%M`

COMANDOSDIR="/bin/"

UPTIME=`/usr/bin/uptime | awk '{print $3}'`
# ACTION: Check Disk Space Utilization

# Notify administrator if disk space utilization on any filesystem >= 80%

#
echo "" > ${LOGFILE}
echo "*** custom alert ***" > ${MAINTLOG}

echo "====================" >> ${MAINTLOG}
${COMANDOSDIR}df -t ext3 | awk '{if (int($5)>=80) {print $0}}' > ${STDOUT}
if [ -s $STDOUT ] ; then
echo "filesystems have >= 80% used :" >> ${MAINTLOG}

echo "--------------------------------------------------" >> ${MAINTLOG}

${COMANDOSDIR}df -t ext3 | head -1 >> ${MAINTLOG}

cat $STDOUT >> ${MAINTLOG}
fi
# ACTION: Check Inode Utilization

# Notify administrator if inode utilization on any filesystem >= 80%

#
${COMANDOSDIR}df -i -t ext3 | awk '{if (int($5)>=80) {print $0}}' > ${STDOUT}
if [ -s $STDOUT ] ; then
echo "filesystems have >= 80% inode used :" >> ${MAINTLOG}

echo "--------------------------------------------------" >> ${MAINTLOG}

${COMANDOSDIR}df -i -t ext3 | head -1 >> ${MAINTLOG}

cat $STDOUT >> $MAINTLOG
fi
echo "====================" >> ${MAINTLOG}

cat $MAINTLOG >> ${LOGFILE}
demanl
Advanced member
Advanced member
Posts: 72
Joined: Fri Sep 12, 2014 10:05 pm

[SOLVED] Yet Another Backup Script for Community Version

Postby demanl » Sat Jun 04, 2011 10:31 am

Bravo Mr Heinzg and other script contributors.....very nice script indeed.
But simple.....no.....very nice though.
I am a linux novice.......and since coming from a windows world I am having trouble wrapping my mind around the ease at which you can do things in linux ...lol....so please...if my question in very noobish...plz consider the source...LOL...
In the backup portion of the script...u have a hot sync rsync and a separate cold sync rsync portion....I understand the reasoning behind the COLD sync..and applaud it.....but what is the reasoning/purpose of the HOT sync...???
Again kudos/applause for the very nice script and explanation of how it works..!!
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Postby LaFong » Sat Jun 04, 2011 12:12 pm

[quote user="demanl"]

In the backup portion of the script...u have a hot sync rsync and a separate cold sync rsync portion....I understand the reasoning behind the COLD sync..and applaud it.....but what is the reasoning/purpose of the HOT sync...???

[/QUOTE]

You do the hot sync so that the cold sync goes much faster. Users can continue to use the server while it is being hot-synced. The hot sync will often sync 90% or more of the data. When you then shut down Zimbra services to do the cold sync, there isn't much data left to sync, so downtime is much less than if you do only a cold sync.
demanl
Advanced member
Advanced member
Posts: 72
Joined: Fri Sep 12, 2014 10:05 pm

[SOLVED] Yet Another Backup Script for Community Version

Postby demanl » Sat Jun 04, 2011 12:46 pm

[quote user="LaFong"]You do the hot sync so that the cold sync goes much faster. Users can continue to use the server while it is being hot-synced. The hot sync will often sync 90% or more of the data. When you then shut down Zimbra services to do the cold sync, there isn't much data left to sync, so downtime is much less than if you do only a cold sync.[/QUOTE]
Awesome...pretty incredible.....tyvm for a quick replay......:)
ciclonite
Posts: 1
Joined: Sat Sep 13, 2014 2:03 am

[SOLVED] Yet Another Backup Script for Community Version

Postby ciclonite » Wed Jun 08, 2011 4:47 am

Hi,

i'm a new user of Zimbra Community Edition v. 7.1.1 on Ubuntu 10.04 LTS 64Bit.

Now i want to setup zmbak ver 0.8 to backup the ZCS.

When i try to install the script (after configure it) with ./zmbak_v.0.8.sh --INSTALL

it need to install mailx. On Ubuntu 10.04 the installation fail with this error:


./zmbak_v.0.8.sh

cat: /opt/zimbra/backup.week: No such file or directory
'mailx' is not installed!
For a Howto install mailx without postfix please visit the link below


Should">http://www.zimbra.com/forums/administrators/13528-sending-email-comand-line-logwatch-ubuntu-6-06-a.html#post70636
Should
I try install this for you?
!! Only say yes if you are running Ubuntu!!
install y or n: y
Downloading mta-dummy package to /tmp

--2011-06-08 11:45:24-- http://ubuntu.lnix.net/misc/mta-dummy/mta-dummy_1.0_all.deb

Resolving ubuntu.lnix.net... 81.26.52.3

Connecting to ubuntu.lnix.net|81.26.52.3|:80... connected.

HTTP request sent, awaiting response...

HTTP/1.1 416 Requested Range Not Satisfiable

Date: Wed, 08 Jun 2011 09:45:25 GMT

Server: Apache

Connection: close

Content-Type: text/html; charset=iso-8859-1
The file is already fully retrieved; nothing to do.


(Reading database ... 92434 files and directories currently installed.)

Preparing to replace mta-dummy 1.0 (using mta-dummy_1.0_all.deb) ...

Unpacking replacement mta-dummy ...

Setting up mta-dummy (1.0) ...
Reading package lists... Done

Building dependency tree

Reading state information... Done

Package mailx is a virtual package provided by:

mailutils 1:2.1+dfsg1-4ubuntu1

heirloom-mailx 12.4-1.1

bsd-mailx 8.1.2-0.20090911cvs-2ubuntu1

You should explicitly select one to install.

E: Package mailx has no installation candidate

Had a problem installing mailx, please ask for help in the forums


I have not try to install one of this packages manually because im worry about the postfix installation.
Any suggestion is appreciate.

Thanks in advice.
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Postby LaFong » Fri Aug 26, 2011 4:27 pm

I HAVE UPDATED MY ZMDBBAC.SH SCRIPT, "ADDING PER USER BACKUP AND RESTORE, WHICH ALLOWS PER-USER BACKUP. THE NEW VERSION ALLOWS FOR MULTIPLE USERS PER DATABASE. ZIMBRA CREATES A MAXIMUM OF 100 USER DATABASES. USER IDS ABOVE 100 SHARE A DATABASE WITH OTHER USERS. FOR EXAMPLE, USER ID 112 WOULD SHARE THE MBOXGROUP12 DATABASE WITH USER ID 12. IT IS RECOMMENDED THAT:




  • ANYONE USING THE SCRIPT SHOULD DOWNLOAD AND UPDATE TO THIS NEWER VERSION. OTHER THAN SWAPPING OUT THE ZMDBBAC.SH SCRIPT, THE CHANGES ARE ENTIRELY ON THE RESTORE END.

  • ANY OLD BACKUPS MADE BY THE PREVIOUS VERSION OF THIS SCRIPT SHOULD BE MODIFIED FIRST BEFORE BEING USED FOR A RESTORE. THIS CAN BE DONE WITH SIMPLE TEXT EDITS. PM ME IF THE NEED EVER ARISES.

  • I HAVE NOT TESTED THIS SCRIPT WITH ZCS V7.X YET. I PLAN ON UPDATING SHORTLY. IT IS ADVISABLE NOT TO USE A V6 BACKUP TO RESTORE TO A V7 SERVER, SINCE ZIMBRA MAY HAVE MADE SOME SLIGHT DB ALTERATIONS. THESE ARE UNLIKELY TO AFFECT THE BACKUP PORTION OF THE SCRIPTS.

  • DO TEST RESTORES ON A TEST SERVER. I HAVE DONE SEVERAL SUCCESSFULLY, BUT WOULD APPRECIATE IT IF OTHERS WOULD TEST AND GIVE FEEDBACK. ESPECIALLY IF YOU'RE USING ZCS 7.X.

  • I HAVE UPDATED THE INSTRUCTIONS IN THE ORIGINAL POST, ALMOST ENTIRELY CONCERNING RESTORES.


CHANGES IN THIS 0.4 ALPHA VERSION ARE:



  • REMOVED DELETION OF EXISTING DB, SINCE A DB CAN HAVE MULTIPLE USERS

  • CREATED SKELETON DUMP OF JUST DB TABLE STRUCTURE, FOR RESTORES TO OTHER SERVERS WHERE THE DB DOES NOT EXIST

  • DELETE AND RE-INSERT USER'S TABLE ROWS, THEN OPTIMIZE DB, INSTEAD OF DROPPING AND RECREATING DBS

  • CHANGE EMAIL OUTPUT TO SHOW BOTH USER ID AND DB NAME

  • CREATED RESTORE_USER.SH SCRIPT TO AUTOMATE RESTORE PROCESS

LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Postby LaFong » Tue Sep 06, 2011 2:41 pm

I have now tested a few user restores with v7.1.2, and they appear to have worked fine.


  • I have placed an updated version of the zip file containing the scripts at the same link, zmDBbac.zip.

  • This update contains mainly cosmetic changes to the restore_user.sh script. The zmDBbac.sh script was not changed.

  • The changes to restore_user.sh make things clearer when restoring, hopefully, so you should download the new version.

  • When restoring, you will get messages like

    Table does not support optimize, doing recreate + analyze instead

    This is because Zimbra 7.x has changed its version of MySQL. These messages may be ignored.

kazooless
Advanced member
Advanced member
Posts: 91
Joined: Sat Sep 13, 2014 12:08 am

[SOLVED] Yet Another Backup Script for Community Version

Postby kazooless » Thu Sep 22, 2011 5:47 pm

Hello,
I have been backing up with this script (zmbak_v.0.8.sh). After I upgraded to 7.1.2 I lost my spam configs, even though the file said that it was the right place to make changes leading me to believe that it would overwrite my edits.
Anyway, I just need to recover one file and the time since the backup has been a while so I only have the files in the "burn" directory. How do I access these files to retrieve one file? (:/opt/zimbra/conf/spamassassin/local.cf)
Thanks,
kazoo
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Postby LaFong » Thu Sep 22, 2011 9:48 pm

[quote user="kazooless"]Hello,
I have been backing up with this script (zmbak_v.0.8.sh). After I upgraded to 7.1.2 I lost my spam configs, even though the file said that it was the right place to make changes leading me to believe that it would overwrite my edits.
Anyway, I just need to recover one file and the time since the backup has been a while so I only have the files in the "burn" directory. How do I access these files to retrieve one file? (:/opt/zimbra/conf/spamassassin/local.cf)

[/QUOTE]

So, you have the dar files in the burn directory? Here is a tutorial on dar:

DAR - Tutorial
You would likely get the local.cf file from the FULL dar file. If the file was named, for example, 37_Zimbra_Backup_12-September-2011_FULL.1.dar, you could extract local.cf with this command:

cd /tmp

dar -R / -x /Backup/burn/37_Zimbra_Backup_12-September-2011_FULL -g conf/spamassassin/local.cf
Notice that you leave out the ".1.dar" part of the filename. You will now have the restored file in /tmp/conf/spamassassin/local.cf

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 10 guests