Results 1 to 10 of 11

Thread: [SOLVED] Massive migration using imapsync, don't know users passwords

Hybrid View

  1. #1
    Join Date
    Nov 2007
    Location
    Atlanta, GA
    Posts
    5
    Rep Power
    8

    Default [SOLVED] Massive migration using imapsync, don't know users passwords

    I've got about 2500 users from about 40 UNIX mail servers to be migrated to a couple of beefy Dell Servers running RHEL5. It is not feasible to reset their passwords. We are scripting the moving process. For the migration, we've got boxes set aside that will do all of the crunching. The Zimbra servers just have to accept the incoming imap connections from imapsync.

    On the UNIX boxes, we are manipulating the shadow file, so we can get in there and get the mail, no problem. The new Zimbra boxes are authenticating off of our AD. We have to do our migration in stages (just because of the sheer numbers here, I'm talking TB, not GB of mail between the new servers). After the first migration, the first box will be live with users while we import in more (I know, crazy, but we are under a gun here). We have to minimalize the impact to the users.

    I can not manipulate the authentication once Box1 goes live, so we can't just turn off the AD authentication to can use local authentication. As we know, Zimbra will not allow for multiple authentication sources (local and AD, etc.)

    Because we have a mixed environment, our users can, and usually have, separate passwords for the UNIX world and the Windows world, so shadowing their login's on the UNIX boxes won't necessarily work. (this will be coming to an end soon too)

    Here's what I have discovered:
    If I set an account to be an administrator, it can use both local and AD Auth, so I know it has kept the password that I set when I batch created the accounts using zmprov.

    Here's what I want to know, can I:
    Batch change accounts to be administrator accounts (dangerous, but needed) and then batch change them back. I haven't found this documented anywhere.
    or
    On the command line force Zimbra to look at either the local password store or AD Auth until our migration is over and then we will turn it back to AD only.

    Our only other option is to sit at the admin web gui and hit clicky boxes on every account before and after the migration.

    Please help me Obi Wan Kenobi, you are my only hope. (yeah, I know, we are probably screwed)

    Thanks in advance!

  2. #2
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Welcome to the forums,

    Don't worry, you definitely got other methods than making people admins!

    If supported on both ends you should be able to use imapsync plain auth.

    So what your seeing is that the admin accounts in the most recent versions automatically have fallback auth set incase your external AD auth is unavailable or configured improperly.
    If you've already set up this domain against the AD for auth you could alternatively enable fallback auth for everyone on it:
    Code:
     zmprov md migration.domain.com zimbraAuthFallbackToLocal TRUE
    Then 'cut' your connection to the AD box.

    Some create a domain with local auth just for migration - you can do this from the admin gui of course, but the commands are:
    Code:
     zmprov cd migration.domain.com zimbraAuthMech zimbra
    When you provision the accounts don't make the password too easy, because even though you're going to use the AD later (&/or change fallbackauth back) it's still not a good idea to have simple pass (you would use ‘’ for like a null if your just going to use AD the whole time)
    Code:
    zmprov ca user@migration.domain.com password
    Obviously you will have more on these lines for account names etc, see:
    Zmprov - Zimbra :: Wiki
    Zmprov Examples - Zimbra :: Wiki
    Bulk Provisioning - Zimbra :: Wiki

    So now you have both the shadow accounts & zimbra accounts with known passwords.
    Then imapsync from unix boxes (user@domain.com) > zimbra boxes (user@migration.domain.com)

    And script a rename of the accounts to the main domain that uses external auth (& no fallback):
    Code:
    zmprov ra user@migration.domain.com user@domain.com
    When your all done just delete that migration.domain.com

    ---
    Note if you are on the Network Edition:
    After the account renames they might not make it in-to incremental backups until the next system full backup; either automatic or if you manually start one.
    I can't imagine we're talking 2.5K users with more than 10TB, but if you can't do a full system wide backup too often because of disk or tape capacity storage reasons etc; and this is going to be a long migration timeline, you can do them individually:
    zmbackup -f -s server.domain.com -a user@domain.com
    (server being the mailstore they happen to be on)
    Last edited by mmorse; 11-05-2007 at 11:08 PM.

  3. #3
    Join Date
    Aug 2005
    Posts
    1,433
    Rep Power
    12

    Default

    Zimbra does support AUTH=PLAIN. You should be able to authenticate as a target user using only an admin username/password and the target username -- no user passwords are referenced in this case.
    Bugzilla - Wiki - Downloads - Before posting... Search!

  4. #4
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    sry started elaborating on the falback auth method because you mentioned it

    so you'd have like:
    imapsync --host1 source.server.com --user1 username --authuser1 adminusername --password1 adminpassword --ssl1 --port1 993 --host2 destination.server.com --user2 username --password2 password --authmech2 PLAIN --ssl2 --port2 993 --syncinternaldates --subscribe --nosyncacls

    or

    imapsync --syncinternaldates --subscribed --host1 sourceserver --host2 destinationserver --user1 userAcctSrc --authuser1 adminUserSrc --password1 adminPassDest --user2 useracctondest --authuser2 adminUserDest --password2 adminPassDest --ssl1 --port1 num --ssl2 --port2

    or you could still manipulate shadow files on the source servers if you want

    be sure zimbraImapSSLServerEnabled is set to TRUE

    AUTH=PLAIN requires TLS encoding, so an IMAPs or STARTTLS command on a normal IMAP connection

  5. #5
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    719
    Rep Power
    9

    Default

    I did 2500 users in 36 hours last July with essentially no user-visible downtime, though the total volume of mail was only 500GB.

    I used a set of queue/lockfile directories to keep 4 simultaneous imapsync jobs going.

    Of the 3 boxes involved, the imapsync host in the middle needs to be the most powerful.

    For the two weeks before the migration, I ran imapsyncs in a loop (one simultaneous job during peak hours, three simultaneous jobs off hours), copying mail from user@carleton.edu on the Cyrus server to user@migrate.carleton.edu in the Zimbra server. The per-user final sync process was:

    - Pick the next available user from the "todo" queue; move lockfile into "doing" queue.

    - Incremental imapsync, AUTH PLAIN as the cyrus admin user on the originating end, and as a Zimbra admin user on the destination. Here's my command line. ldap2migrate.pl populates the Zimbra directory from our enterprise directory (trivial script, but probably only locally useful), and all the regtrans2's are necessary to work around folder names that conflict with default calendars/addressbooks/etc and the fact that ":" seems to be illegal in Zimbra folder names.

    Code:
    #!/bin/sh
    ssh zimbra@mail2 "(ldap2migrate.pl carlnetid=$* | zmprov)"
    imapsync --host1 original-imap.carleton.edu --host2 new-zimbra.carleton.edu\
     --buffersize 8192000 \
     --user1 $* --user2 $*@migrate.carleton.edu --nosyncacls --noauthmd5 --ssl1 --ssl2 --sep1 /\
     --exclude '^Trash-not-migrated$|^Trash$|^trash$|^Deleted Messages$' \
     --syncinternaldates --authuser1 cyrus --authuser2 zimbra-superuser@carleton.edu\
     --useheader Message-ID --useheader Date --skipsize --subscribe --prefix1 ''\
     --expunge2 --passfile1 /root/cyrus-pass --passfile2\
     /root/zimbra-pass  --authmech1 PLAIN --authmech2 PLAIN --delete2\
     --expunge1 --regextrans2 's/^Calendar$/Calendar (old)/' \
     --regextrans2 's/^CALENDAR/CALENDAR (old)/' \
     --regextrans2 's/^Contacts$/Contacts (old)/'\
     --regextrans2 's/^Notes$/Notes (old)/'\
     --regextrans2 's/^calendar$/calendar (old)/'\
     --regextrans2 's/^contacts$/contacts (old)/'\
     --regextrans2 's/^notes$/notes (old)/' --regextrans2 's/: / /g'\
     --regextrans2 's/://' --regextrans2 's/^Contacts\//Contacts (old)\//i'\
     --regextrans2 's/^Calendar\//Calendar\//i'\
     --regextrans2 's/^Notes\//Notes (old)\//i' 2>&1\
     >> /var/log/migration/$*-cyrus-zimbra-migrate 2>> /var/log/migration-err/$* ||
    mail -s "imapsync errors $*" admin-account@carleton.edu < /var/log/migration-err/$*
    - When that incremental imapsync is done, rename the Cyrus account to user.migrate (which has the effect of locking out the user); set forwarding for user@carleton.ed to a mailer that simply queues; create a new Cyrus account "user" with a couple placeholder messages to the effect "your mail is being synchronized to the new server, please try Zimbra in 15 minutes"; and run another incremental imapsync

    - When the second imapsync is done (typically only a minute, though multi-gigabyte mailboxes took as long as 30 minutes simply to list), rename the Zimbra account from user@migrate.carleton.edu to user@carleton.edu, then set forwarding on the original system to deliver to Zimbra

    - Move the lockfile for user to the "done" directory

    If your source server is something other than Cyrus imapd, then renaming accounts might be a little less trivial, but you ought to be able to do something similar.

  6. #6
    Join Date
    Nov 2007
    Location
    Atlanta, GA
    Posts
    5
    Rep Power
    8

    Default

    O.K., I'm trying to do your suggestion mmorse with the migration domain. Oh and thanks for the dkarp. After we had created our working string of imapsync commands, I had forgotten about the authuser command.

    We have figured that it is best to go with the migration domain since we will have some users who will try to log in and mess with e-mail in the middle of the migration. This way we can control it better. However, I'm having problems with the second domain.

    I created it and have a couple accounts that are part of that domain (in our case, it is called mm2). Even though there is a local password for say audit@mm2, I can not log in. However, if I change my domain to the real working domain, I can log in.

    I get a 2 NO AUTHENTICATE failed error message when part of the mm2 domain.

    You did have a typo with the account provisioning, correct? That should be zmprov ca user@migration.domain.com password, right?

    So, what am I missing. We are supposed to start this up tomorrow. :/

    Thanks!

  7. #7
    Join Date
    Oct 2009
    Posts
    1
    Rep Power
    6

    Default

    Quote Originally Posted by dkarp View Post
    Zimbra does support AUTH=PLAIN. You should be able to authenticate as a target user using only an admin username/password and the target username -- no user passwords are referenced in this case.
    Hello ...
    I try to synchronize 2 servers (QMail --> Zimbra CS) with imapsync... and I have a problem with imapsync AUTH PLAIN...
    Here's my command :

    Code:
    imapsync --syncinternaldates --host1 myhostSrc --host2 myhostDest 
    --user1 userLoginSrc --authuser1 adminLoginHostSrc --password1 
    passwordAdminHostSrc --user2 userLoginDest --authuser2 
    adminLoginHostDest --password2 passwordAdminHostDest
    With the option authuser1, only the mails of the "admin" (authuser1) are synchronized and not those of the user "user1"...

    Don't understand why... have you got any idea to resolve this problem ?

    Thank you.

  8. #8
    Join Date
    Jan 2009
    Posts
    369
    Rep Power
    6

    Default

    did you try with "--authmech1 PLAIN" and "--authmech2 PLAIN" options?

Similar Threads

  1. Migrating users with imapsync... without passwords?
    By misleb in forum Installation
    Replies: 6
    Last Post: 08-12-2007, 09:03 AM
  2. migration without passwords
    By alam in forum Migration
    Replies: 4
    Last Post: 07-31-2007, 11:15 AM
  3. 4.0 RC1 imapsync with admin???
    By kirme3 in forum Administrators
    Replies: 37
    Last Post: 07-19-2007, 10:52 AM
  4. need advice on configuring zimbra to work with fax server
    By pheonix1t in forum Administrators
    Replies: 0
    Last Post: 07-11-2007, 08:46 PM
  5. Dump users & passwords
    By avisser in forum Administrators
    Replies: 1
    Last Post: 02-17-2006, 03:17 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •