Results 1 to 10 of 34

Thread: Per User Backup for FOSS edition

Threaded View

  1. #1
    Join Date
    May 2007
    Location
    England
    Posts
    13
    Rep Power
    8

    Default Per User Backup for FOSS edition

    Hi all,

    I have decided to have a go at creating a per user backup solution for the FOSS edition. Here comes a brain dump...

    Currently I use the LVM backup technique to create a nightly backup of '/opt/zimbra' as a disaster recovery option. However, more of my users are using webmail and sooner or later someone is going to do something really silly and I will need a way to restore the data for an individual. Restoring the whole server for one users cock up is not really an option

    Disclaimer

    The scripts attached to this post are experimental and only suitable for testing purposes only on non-production servers. If you use them on a production server and it breaks, you get to keep the pieces!

    Scope

    Basically I want a hot/warm-ish backup solution which will give me a 24 hourly restore point for individual user accounts. If someone does something silly, then loosing 24 hours of email is fair penalty in my book If I can get reliable per user backup/restore working then I will continue to do my '/opt/zimbra' DR backup just once a week, and user backups every night.

    How it works

    In a nut shell...

    • Dumps the 'zimbra' database.
    • Get a list of all users
    • Loops through each user
      • Figures out what volumes the user data is stored in
      • Makes a "hot" sync of the users index/store
      • Puts the account in "maintenance" mode
      • Dumps the user DB and rows from associated tables in the 'zimbra' DB
      • Re-syncs the users index/store
      • Puts the account in "active" mode
      • Creates a tarball of the synced index/store data
      • Rotates the backups


    It's a bash script. I have tried to make sure there is no hard coded paths, accounts, passwords, etc. All but two configuration parameters are taken directly from Zimbra. Everything is logged to the console and log file.

    The directory each account rsyncs it index/store to (the cache) is preserved between backups, so subsequent syncs are relatively quick. However, the cache must be local or mounted file system. In addition to the cache, a daily backup directory is created containing a tarball of that days index/store data plus the dumped SQL. Therefore, you need extra disk space. You can do the maths

    Not sure how well this will scale, but for my 15 users it should be fine. I am planning to keep the cache on local disk (for speed) and the backup directory on NFS.

    Anyone for testing?

    I have created a vmware box running a snapshot of my live server. It runs ZCS 5.0.4 FOSS on Debian 4.0 32bit. I have not tested on 4.5.x (yet). Is there anyone out there who might be interested in testing on non-production servers to help iron out the wrinkles?

    The Script

    v0.1 - 11th April 2008
    • Initial release.
    • Scripts to backup individual user or all users.
    • No restore features as yet.
    • No real documentation.


    I haven't started on a restore script yet, but getting the basics going shouldn't be to time consuming. I will be working on the restore scripts early next week.

    Download the attached tarball and extract on your non-production server. 'zimbra_functions.sh' has a couple of parameters you can configure at the top to choose the cache and backup locations.

    To back up a single user run the following on your Zimbra server.

    Code:
    ./zimbra_backup_user.sh user1@example.org
    To backup all users run to following on your Zimbra server.

    Code:
    ./zimbra_backup_all_users.sh
    The Logs

    Below is example log output to show what it does so far.

    Code:
    Thu Apr 10 12:51:05 EDT 2008 : Dumping zimbra --single-transaction > /opt/zimbra_backup/zimbra.sql
    Thu Apr 10 12:51:05 EDT 2008 : Getting accounts
    Thu Apr 10 12:51:07 EDT 2008 : Backing up accounts
    Thu Apr 10 12:51:07 EDT 2008 : Processing user1@example.org
    Thu Apr 10 12:51:11 EDT 2008 : Getting group_id and index_volume_id
    Thu Apr 10 12:51:11 EDT 2008 : Id      : 10
    Thu Apr 10 12:51:11 EDT 2008 : Group   : 10
    Thu Apr 10 12:51:11 EDT 2008 : Mailbox : mboxgroup10
    Thu Apr 10 12:51:11 EDT 2008 : Getting message_volume_id and index_volume_id
    Thu Apr 10 12:51:11 EDT 2008 : Getting index volume path
    Thu Apr 10 12:51:11 EDT 2008 : Getting message volume path
    Thu Apr 10 12:51:12 EDT 2008 : index1 path   : /opt/zimbra/index
    Thu Apr 10 12:51:12 EDT 2008 : message1 path : /opt/zimbra/store
    Thu Apr 10 12:51:12 EDT 2008 : Syncing /opt/zimbra/index/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:12 EDT 2008 : Size : 3.4M
    Thu Apr 10 12:51:12 EDT 2008 : Syncing /opt/zimbra/store/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:12 EDT 2008 : Size : 144K
    Thu Apr 10 12:51:16 EDT 2008 : user1@example.org is now in maintenance mode.
    Thu Apr 10 12:51:16 EDT 2008 : Dumping mboxgroup10 > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mboxgroup10.sql
    Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra mailbox --no-create-info --where "id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mailbox.sql
    Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra mailbox_metadata --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mailbox_metadata.sql
    Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra out_of_office --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/out_of_office.sql
    Thu Apr 10 12:51:17 EDT 2008 : Dumping zimbra scheduled_task --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/scheduled_task.sql
    Thu Apr 10 12:51:17 EDT 2008 : Dumping zimbra table_maintenance --no-create-info --where "database_name='mboxgroup10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/table_maintenance.sql
    Thu Apr 10 12:51:17 EDT 2008 : Syncing /opt/zimbra/index/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:17 EDT 2008 : Size : 3.4M
    Thu Apr 10 12:51:17 EDT 2008 : Syncing /opt/zimbra/store/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:17 EDT 2008 : Size : 144K
    Thu Apr 10 12:51:21 EDT 2008 : user1@example.org is now in active mode.
    Thu Apr 10 12:51:21 EDT 2008 : Creating index_n_store tarball from /opt/zimbra_cache/user1@example.org
    Thu Apr 10 12:51:26 EDT 2008 : Rotating backups
    Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1251
    Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1233
    Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1209
    Thu Apr 10 12:51:26 EDT 2008 : Deleting 2008-04-10-1149
    
    Thu Apr 10 12:51:26 EDT 2008 : Processing user2@example.org
    Thu Apr 10 12:51:29 EDT 2008 : Getting group_id and index_volume_id
    Thu Apr 10 12:51:29 EDT 2008 : Id      : 8
    Thu Apr 10 12:51:29 EDT 2008 : Group   : 8
    Thu Apr 10 12:51:29 EDT 2008 : Mailbox : mboxgroup8
    Thu Apr 10 12:51:29 EDT 2008 : Getting message_volume_id and index_volume_id
    Thu Apr 10 12:51:29 EDT 2008 : Getting index volume path
    Thu Apr 10 12:51:29 EDT 2008 : Getting message volume path
    Thu Apr 10 12:51:29 EDT 2008 : index1 path   : /opt/zimbra/index
    Thu Apr 10 12:51:29 EDT 2008 : message1 path : /opt/zimbra/store
    Thu Apr 10 12:51:29 EDT 2008 : Syncing /opt/zimbra/index/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:29 EDT 2008 : Size : 104M
    Thu Apr 10 12:51:29 EDT 2008 : Syncing /opt/zimbra/store/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:30 EDT 2008 : Size : 231M
    Thu Apr 10 12:51:33 EDT 2008 : user2@example.org is now in maintenance mode.
    Thu Apr 10 12:51:33 EDT 2008 : Dumping mboxgroup8 > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mboxgroup8.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra mailbox --no-create-info --where "id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mailbox.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra mailbox_metadata --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mailbox_metadata.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra out_of_office --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/out_of_office.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra scheduled_task --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/scheduled_task.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra table_maintenance --no-create-info --where "database_name='mboxgroup8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/table_maintenance.sql
    Thu Apr 10 12:51:33 EDT 2008 : Syncing /opt/zimbra/index/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:33 EDT 2008 : Size : 104M
    Thu Apr 10 12:51:33 EDT 2008 : Syncing /opt/zimbra/store/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:34 EDT 2008 : Size : 231M
    Thu Apr 10 12:51:37 EDT 2008 : user2@example.org is now in active mode.
    Thu Apr 10 12:51:37 EDT 2008 : Creating index_n_store tarball from /opt/zimbra_cache/user2@example.org
    Thu Apr 10 12:59:08 EDT 2008 : Rotating backups
    Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1251
    Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1233
    Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1210
    Thu Apr 10 12:59:09 EDT 2008 : Deleting 2008-04-10-1150
    Thu Apr 10 12:59:10 EDT 2008 : Deleting 2008-04-10-1148
    The Cache

    Example cache directory.

    Code:
    user1@example.org/
    `-- opt
        `-- zimbra
            |-- index
            |   `-- 0
            |       `-- 10
            |           `-- index
            |               `-- 0
            |                   |-- _6ip.cfs
            |                   |-- _6ip.del
            |                   |-- _8on.cfs
            |                   |-- _8on.del
            |                   |-- _9em.cfs
            |                   |-- _9em.del
            |                   |-- _gkx.cfs
            |                   |-- _gkx.del
            |                   |-- _gyu.cfs
            |                   |-- _gyu.del
            |                   |-- _h38.cfs
            |                   |-- _h3d.cfs
            |                   |-- deletable
            |                   `-- segments
            `-- store
                `-- 0
                    `-- 10
                        `-- msg
                            |-- 0
                            |-- 1
                            `-- 2
                                |-- 10023-44804.msg
                                |-- 10062-45306.msg
                                |-- 10063-45307.msg
                                |-- 10064-45308.msg
                                |-- 10082-45404.msg
                                |-- 10083-45406.msg
                                |-- 10185-46210.msg
                                |-- 10191-46216.msg
                                |-- 10201-46401.msg
                                |-- 10203-46405.msg
                                |-- 10267-46714.msg
                                |-- 10280-46800.msg
                                |-- 10300-47000.msg
                                |-- 10301-47001.msg
                                |-- 8406-37009.msg
                                |-- 9865-44013.msg
                                |-- 9920-44304.msg
                                |-- 9951-44425.msg
                                `-- 9964-44510.msg
    The Backups

    Example backup directory.

    Code:
    user1@example.org/
    |-- 2008-04-10-1209
    |   |-- index_n_store.tar.bz2
    |   |-- mailbox.sql
    |   |-- mailbox_metadata.sql
    |   |-- mboxgroup10.sql
    |   |-- out_of_office.sql
    |   |-- scheduled_task.sql
    |   `-- table_maintenance.sql
    |-- 2008-04-10-1233
    |   |-- calendar.ics
    |   |-- contacts.csv
    |   |-- mailbox.sql
    |   |-- mailbox_metadata.sql
    |   |-- mboxgroup10.sql
    |   |-- out_of_office.sql
    |   |-- scheduled_task.sql
    |   `-- table_maintenance.sql
    `-- 2008-04-10-1251
        |-- index_n_store.tar.bz2
        |-- mailbox.sql
        |-- mailbox_metadata.sql
        |-- mboxgroup10.sql
        |-- out_of_office.sql
        |-- scheduled_task.sql
        `-- table_maintenance.sql
    Thoughts, comments, suggestions?

    Regards, Martin.
    Attached Files Attached Files
    Last edited by fdsadmin; 04-11-2008 at 07:08 AM. Reason: v0.1 Released

Similar Threads

  1. Replies: 21
    Last Post: 02-04-2010, 10:06 AM
  2. Post instsallation problems
    By Assaf in forum Installation
    Replies: 14
    Last Post: 01-29-2007, 11:38 AM
  3. Services stopped working
    By lilwong in forum Administrators
    Replies: 4
    Last Post: 08-15-2006, 10:19 AM
  4. Fedora Core 3, Clean Install - Not working!
    By pcjackson in forum Installation
    Replies: 17
    Last Post: 03-05-2006, 07:38 PM
  5. Network edition - strange behavior
    By goetzi in forum Installation
    Replies: 6
    Last Post: 11-16-2005, 03:08 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •