Page 1 of 3 123 LastLast
Results 1 to 10 of 34

Thread: Per User Backup for FOSS edition

Hybrid View

  1. #1
    Join Date
    May 2007
    Location
    England
    Posts
    13
    Rep Power
    8

    Default Per User Backup for FOSS edition

    Hi all,

    I have decided to have a go at creating a per user backup solution for the FOSS edition. Here comes a brain dump...

    Currently I use the LVM backup technique to create a nightly backup of '/opt/zimbra' as a disaster recovery option. However, more of my users are using webmail and sooner or later someone is going to do something really silly and I will need a way to restore the data for an individual. Restoring the whole server for one users cock up is not really an option

    Disclaimer

    The scripts attached to this post are experimental and only suitable for testing purposes only on non-production servers. If you use them on a production server and it breaks, you get to keep the pieces!

    Scope

    Basically I want a hot/warm-ish backup solution which will give me a 24 hourly restore point for individual user accounts. If someone does something silly, then loosing 24 hours of email is fair penalty in my book If I can get reliable per user backup/restore working then I will continue to do my '/opt/zimbra' DR backup just once a week, and user backups every night.

    How it works

    In a nut shell...

    • Dumps the 'zimbra' database.
    • Get a list of all users
    • Loops through each user
      • Figures out what volumes the user data is stored in
      • Makes a "hot" sync of the users index/store
      • Puts the account in "maintenance" mode
      • Dumps the user DB and rows from associated tables in the 'zimbra' DB
      • Re-syncs the users index/store
      • Puts the account in "active" mode
      • Creates a tarball of the synced index/store data
      • Rotates the backups


    It's a bash script. I have tried to make sure there is no hard coded paths, accounts, passwords, etc. All but two configuration parameters are taken directly from Zimbra. Everything is logged to the console and log file.

    The directory each account rsyncs it index/store to (the cache) is preserved between backups, so subsequent syncs are relatively quick. However, the cache must be local or mounted file system. In addition to the cache, a daily backup directory is created containing a tarball of that days index/store data plus the dumped SQL. Therefore, you need extra disk space. You can do the maths

    Not sure how well this will scale, but for my 15 users it should be fine. I am planning to keep the cache on local disk (for speed) and the backup directory on NFS.

    Anyone for testing?

    I have created a vmware box running a snapshot of my live server. It runs ZCS 5.0.4 FOSS on Debian 4.0 32bit. I have not tested on 4.5.x (yet). Is there anyone out there who might be interested in testing on non-production servers to help iron out the wrinkles?

    The Script

    v0.1 - 11th April 2008
    • Initial release.
    • Scripts to backup individual user or all users.
    • No restore features as yet.
    • No real documentation.


    I haven't started on a restore script yet, but getting the basics going shouldn't be to time consuming. I will be working on the restore scripts early next week.

    Download the attached tarball and extract on your non-production server. 'zimbra_functions.sh' has a couple of parameters you can configure at the top to choose the cache and backup locations.

    To back up a single user run the following on your Zimbra server.

    Code:
    ./zimbra_backup_user.sh user1@example.org
    To backup all users run to following on your Zimbra server.

    Code:
    ./zimbra_backup_all_users.sh
    The Logs

    Below is example log output to show what it does so far.

    Code:
    Thu Apr 10 12:51:05 EDT 2008 : Dumping zimbra --single-transaction > /opt/zimbra_backup/zimbra.sql
    Thu Apr 10 12:51:05 EDT 2008 : Getting accounts
    Thu Apr 10 12:51:07 EDT 2008 : Backing up accounts
    Thu Apr 10 12:51:07 EDT 2008 : Processing user1@example.org
    Thu Apr 10 12:51:11 EDT 2008 : Getting group_id and index_volume_id
    Thu Apr 10 12:51:11 EDT 2008 : Id      : 10
    Thu Apr 10 12:51:11 EDT 2008 : Group   : 10
    Thu Apr 10 12:51:11 EDT 2008 : Mailbox : mboxgroup10
    Thu Apr 10 12:51:11 EDT 2008 : Getting message_volume_id and index_volume_id
    Thu Apr 10 12:51:11 EDT 2008 : Getting index volume path
    Thu Apr 10 12:51:11 EDT 2008 : Getting message volume path
    Thu Apr 10 12:51:12 EDT 2008 : index1 path   : /opt/zimbra/index
    Thu Apr 10 12:51:12 EDT 2008 : message1 path : /opt/zimbra/store
    Thu Apr 10 12:51:12 EDT 2008 : Syncing /opt/zimbra/index/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:12 EDT 2008 : Size : 3.4M
    Thu Apr 10 12:51:12 EDT 2008 : Syncing /opt/zimbra/store/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:12 EDT 2008 : Size : 144K
    Thu Apr 10 12:51:16 EDT 2008 : user1@example.org is now in maintenance mode.
    Thu Apr 10 12:51:16 EDT 2008 : Dumping mboxgroup10 > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mboxgroup10.sql
    Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra mailbox --no-create-info --where "id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mailbox.sql
    Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra mailbox_metadata --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mailbox_metadata.sql
    Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra out_of_office --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/out_of_office.sql
    Thu Apr 10 12:51:17 EDT 2008 : Dumping zimbra scheduled_task --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/scheduled_task.sql
    Thu Apr 10 12:51:17 EDT 2008 : Dumping zimbra table_maintenance --no-create-info --where "database_name='mboxgroup10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/table_maintenance.sql
    Thu Apr 10 12:51:17 EDT 2008 : Syncing /opt/zimbra/index/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:17 EDT 2008 : Size : 3.4M
    Thu Apr 10 12:51:17 EDT 2008 : Syncing /opt/zimbra/store/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:17 EDT 2008 : Size : 144K
    Thu Apr 10 12:51:21 EDT 2008 : user1@example.org is now in active mode.
    Thu Apr 10 12:51:21 EDT 2008 : Creating index_n_store tarball from /opt/zimbra_cache/user1@example.org
    Thu Apr 10 12:51:26 EDT 2008 : Rotating backups
    Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1251
    Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1233
    Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1209
    Thu Apr 10 12:51:26 EDT 2008 : Deleting 2008-04-10-1149
    
    Thu Apr 10 12:51:26 EDT 2008 : Processing user2@example.org
    Thu Apr 10 12:51:29 EDT 2008 : Getting group_id and index_volume_id
    Thu Apr 10 12:51:29 EDT 2008 : Id      : 8
    Thu Apr 10 12:51:29 EDT 2008 : Group   : 8
    Thu Apr 10 12:51:29 EDT 2008 : Mailbox : mboxgroup8
    Thu Apr 10 12:51:29 EDT 2008 : Getting message_volume_id and index_volume_id
    Thu Apr 10 12:51:29 EDT 2008 : Getting index volume path
    Thu Apr 10 12:51:29 EDT 2008 : Getting message volume path
    Thu Apr 10 12:51:29 EDT 2008 : index1 path   : /opt/zimbra/index
    Thu Apr 10 12:51:29 EDT 2008 : message1 path : /opt/zimbra/store
    Thu Apr 10 12:51:29 EDT 2008 : Syncing /opt/zimbra/index/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:29 EDT 2008 : Size : 104M
    Thu Apr 10 12:51:29 EDT 2008 : Syncing /opt/zimbra/store/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:30 EDT 2008 : Size : 231M
    Thu Apr 10 12:51:33 EDT 2008 : user2@example.org is now in maintenance mode.
    Thu Apr 10 12:51:33 EDT 2008 : Dumping mboxgroup8 > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mboxgroup8.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra mailbox --no-create-info --where "id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mailbox.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra mailbox_metadata --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mailbox_metadata.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra out_of_office --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/out_of_office.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra scheduled_task --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/scheduled_task.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra table_maintenance --no-create-info --where "database_name='mboxgroup8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/table_maintenance.sql
    Thu Apr 10 12:51:33 EDT 2008 : Syncing /opt/zimbra/index/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:33 EDT 2008 : Size : 104M
    Thu Apr 10 12:51:33 EDT 2008 : Syncing /opt/zimbra/store/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:34 EDT 2008 : Size : 231M
    Thu Apr 10 12:51:37 EDT 2008 : user2@example.org is now in active mode.
    Thu Apr 10 12:51:37 EDT 2008 : Creating index_n_store tarball from /opt/zimbra_cache/user2@example.org
    Thu Apr 10 12:59:08 EDT 2008 : Rotating backups
    Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1251
    Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1233
    Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1210
    Thu Apr 10 12:59:09 EDT 2008 : Deleting 2008-04-10-1150
    Thu Apr 10 12:59:10 EDT 2008 : Deleting 2008-04-10-1148
    The Cache

    Example cache directory.

    Code:
    user1@example.org/
    `-- opt
        `-- zimbra
            |-- index
            |   `-- 0
            |       `-- 10
            |           `-- index
            |               `-- 0
            |                   |-- _6ip.cfs
            |                   |-- _6ip.del
            |                   |-- _8on.cfs
            |                   |-- _8on.del
            |                   |-- _9em.cfs
            |                   |-- _9em.del
            |                   |-- _gkx.cfs
            |                   |-- _gkx.del
            |                   |-- _gyu.cfs
            |                   |-- _gyu.del
            |                   |-- _h38.cfs
            |                   |-- _h3d.cfs
            |                   |-- deletable
            |                   `-- segments
            `-- store
                `-- 0
                    `-- 10
                        `-- msg
                            |-- 0
                            |-- 1
                            `-- 2
                                |-- 10023-44804.msg
                                |-- 10062-45306.msg
                                |-- 10063-45307.msg
                                |-- 10064-45308.msg
                                |-- 10082-45404.msg
                                |-- 10083-45406.msg
                                |-- 10185-46210.msg
                                |-- 10191-46216.msg
                                |-- 10201-46401.msg
                                |-- 10203-46405.msg
                                |-- 10267-46714.msg
                                |-- 10280-46800.msg
                                |-- 10300-47000.msg
                                |-- 10301-47001.msg
                                |-- 8406-37009.msg
                                |-- 9865-44013.msg
                                |-- 9920-44304.msg
                                |-- 9951-44425.msg
                                `-- 9964-44510.msg
    The Backups

    Example backup directory.

    Code:
    user1@example.org/
    |-- 2008-04-10-1209
    |   |-- index_n_store.tar.bz2
    |   |-- mailbox.sql
    |   |-- mailbox_metadata.sql
    |   |-- mboxgroup10.sql
    |   |-- out_of_office.sql
    |   |-- scheduled_task.sql
    |   `-- table_maintenance.sql
    |-- 2008-04-10-1233
    |   |-- calendar.ics
    |   |-- contacts.csv
    |   |-- mailbox.sql
    |   |-- mailbox_metadata.sql
    |   |-- mboxgroup10.sql
    |   |-- out_of_office.sql
    |   |-- scheduled_task.sql
    |   `-- table_maintenance.sql
    `-- 2008-04-10-1251
        |-- index_n_store.tar.bz2
        |-- mailbox.sql
        |-- mailbox_metadata.sql
        |-- mboxgroup10.sql
        |-- out_of_office.sql
        |-- scheduled_task.sql
        `-- table_maintenance.sql
    Thoughts, comments, suggestions?

    Regards, Martin.
    Attached Files Attached Files
    Last edited by fdsadmin; 04-11-2008 at 07:08 AM. Reason: v0.1 Released

  2. #2
    Join Date
    Oct 2007
    Location
    Brazil
    Posts
    88
    Rep Power
    8

    Default

    Hi Martin

    Looks promising, I would be happy to do some testing, I have a semi-production server with a few users running UBUNTU FOSS where
    I can run some tests.

    Let me know

    good luck

    []s

  3. #3
    Join Date
    May 2007
    Location
    England
    Posts
    13
    Rep Power
    8

    Default v0.1 Released

    See the first post, I have attached v0.1 of the scripts and added some basic usage instructions.

  4. #4
    Join Date
    Mar 2008
    Posts
    4
    Rep Power
    7

    Default

    I'm really not an expert, but I launched the script and everything looks good.

    Tested on Ubuntu 6.06.2 - Zimbra 5.0.4 FOSS.
    I only have 12 users and a few mails. I'm going to import my 800Mo .pst on Zimbra.

    To test it more seriously, can you give us a method for restoring 1 account ?

  5. #5
    Join Date
    May 2006
    Location
    England.
    Posts
    927
    Rep Power
    10

    Default

    Looks very interesting, once you have enough details & scripts together, consider adding the info to the wiki where it can be maintained easily.

    I'll have a good look through the sourcecode shortly, but it's always good to see this kind of effort

  6. #6
    Join Date
    May 2007
    Location
    England
    Posts
    13
    Rep Power
    8

    Default

    2l2n, I will post an updated script in the new few days which includes restore capability.

    Dirk, I will be more than happy to maintain a wiki page for the script once I have got the basic backup/restore stuff working :-) For now, I will keep the first post in the topic up to date.

  7. #7
    Join Date
    Apr 2008
    Posts
    3
    Rep Power
    7

    Default

    Would like to help here as well. Will be doing the test on our 5.0.2 on Dapper 6.06. Have about 70 users and more being added over the next few weeks. Currently, I am using rdiff-backup for dailies and full over the weekends but no per user backup and restore capabilites....this would do it. Will let you know...

  8. #8
    Join Date
    Oct 2007
    Location
    Pescara, Italy
    Posts
    5
    Rep Power
    8

    Default

    Hi All! Thank you for your work, it's great!

    I have a zimbra 4.5.7 on a Debian Etch and i have a problem whit the script "zimbra_backup_user.sh":

    - at the line 48 :
    Code:
    current_account_zimbraId=`zimbra_run "${zimbra_home}/bin/zmprov ga ${current_account} zimbraId | grep zimbraId | cut -d' ' -f2"`
    Whit this version of Zimbra, the zmprov ga don't accept the "zimbraId" parameter. I have modified the script in this:

    - at line 48 :
    Code:
    current_account_zimbraId=`zimbra_run "${zimbra_home}/bin/zmprov ga ${current_account} |  grep zimbraId | cut -d' ' -f2 | sed -n 1p"`

    I am posting this in the case could serve to someone.

    Sorry for my english, but i am italian.

  9. #9
    Join Date
    Oct 2008
    Location
    Sweden
    Posts
    3
    Rep Power
    7

    Default Questions and a thread bump

    Hi!

    First of all, i like the script a lot I't looks like it does what's needed. However; has anyone tried to restore an account backed up by this script (and how did you do it - i have an idea but if someone already tried that would be nice)?

    Second, has the project moved to a wiki-page or did you just run out of time? If it has moved somewhere else, a link would be nice for newbs like me

    Again, thanks for the effort.

  10. #10
    Join Date
    Feb 2009
    Posts
    11
    Rep Power
    6

    Default

    when use the script backup the user mails finished , how to restore the mail ?
    thx

Similar Threads

  1. Replies: 21
    Last Post: 02-04-2010, 10:06 AM
  2. Post instsallation problems
    By Assaf in forum Installation
    Replies: 14
    Last Post: 01-29-2007, 11:38 AM
  3. Services stopped working
    By lilwong in forum Administrators
    Replies: 4
    Last Post: 08-15-2006, 10:19 AM
  4. Fedora Core 3, Clean Install - Not working!
    By pcjackson in forum Installation
    Replies: 17
    Last Post: 03-05-2006, 07:38 PM
  5. Network edition - strange behavior
    By goetzi in forum Installation
    Replies: 6
    Last Post: 11-16-2005, 03:08 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •