Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: [SOLVED] Massive migration using imapsync, don't know users passwords

  1. #1
    Join Date
    Nov 2007
    Location
    Atlanta, GA
    Posts
    5
    Rep Power
    8

    Default [SOLVED] Massive migration using imapsync, don't know users passwords

    I've got about 2500 users from about 40 UNIX mail servers to be migrated to a couple of beefy Dell Servers running RHEL5. It is not feasible to reset their passwords. We are scripting the moving process. For the migration, we've got boxes set aside that will do all of the crunching. The Zimbra servers just have to accept the incoming imap connections from imapsync.

    On the UNIX boxes, we are manipulating the shadow file, so we can get in there and get the mail, no problem. The new Zimbra boxes are authenticating off of our AD. We have to do our migration in stages (just because of the sheer numbers here, I'm talking TB, not GB of mail between the new servers). After the first migration, the first box will be live with users while we import in more (I know, crazy, but we are under a gun here). We have to minimalize the impact to the users.

    I can not manipulate the authentication once Box1 goes live, so we can't just turn off the AD authentication to can use local authentication. As we know, Zimbra will not allow for multiple authentication sources (local and AD, etc.)

    Because we have a mixed environment, our users can, and usually have, separate passwords for the UNIX world and the Windows world, so shadowing their login's on the UNIX boxes won't necessarily work. (this will be coming to an end soon too)

    Here's what I have discovered:
    If I set an account to be an administrator, it can use both local and AD Auth, so I know it has kept the password that I set when I batch created the accounts using zmprov.

    Here's what I want to know, can I:
    Batch change accounts to be administrator accounts (dangerous, but needed) and then batch change them back. I haven't found this documented anywhere.
    or
    On the command line force Zimbra to look at either the local password store or AD Auth until our migration is over and then we will turn it back to AD only.

    Our only other option is to sit at the admin web gui and hit clicky boxes on every account before and after the migration.

    Please help me Obi Wan Kenobi, you are my only hope. (yeah, I know, we are probably screwed)

    Thanks in advance!

  2. #2
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Welcome to the forums,

    Don't worry, you definitely got other methods than making people admins!

    If supported on both ends you should be able to use imapsync plain auth.

    So what your seeing is that the admin accounts in the most recent versions automatically have fallback auth set incase your external AD auth is unavailable or configured improperly.
    If you've already set up this domain against the AD for auth you could alternatively enable fallback auth for everyone on it:
    Code:
     zmprov md migration.domain.com zimbraAuthFallbackToLocal TRUE
    Then 'cut' your connection to the AD box.

    Some create a domain with local auth just for migration - you can do this from the admin gui of course, but the commands are:
    Code:
     zmprov cd migration.domain.com zimbraAuthMech zimbra
    When you provision the accounts don't make the password too easy, because even though you're going to use the AD later (&/or change fallbackauth back) it's still not a good idea to have simple pass (you would use ‘’ for like a null if your just going to use AD the whole time)
    Code:
    zmprov ca user@migration.domain.com password
    Obviously you will have more on these lines for account names etc, see:
    Zmprov - Zimbra :: Wiki
    Zmprov Examples - Zimbra :: Wiki
    Bulk Provisioning - Zimbra :: Wiki

    So now you have both the shadow accounts & zimbra accounts with known passwords.
    Then imapsync from unix boxes (user@domain.com) > zimbra boxes (user@migration.domain.com)

    And script a rename of the accounts to the main domain that uses external auth (& no fallback):
    Code:
    zmprov ra user@migration.domain.com user@domain.com
    When your all done just delete that migration.domain.com

    ---
    Note if you are on the Network Edition:
    After the account renames they might not make it in-to incremental backups until the next system full backup; either automatic or if you manually start one.
    I can't imagine we're talking 2.5K users with more than 10TB, but if you can't do a full system wide backup too often because of disk or tape capacity storage reasons etc; and this is going to be a long migration timeline, you can do them individually:
    zmbackup -f -s server.domain.com -a user@domain.com
    (server being the mailstore they happen to be on)
    Last edited by mmorse; 11-05-2007 at 11:08 PM.

  3. #3
    Join Date
    Aug 2005
    Posts
    1,433
    Rep Power
    12

    Default

    Zimbra does support AUTH=PLAIN. You should be able to authenticate as a target user using only an admin username/password and the target username -- no user passwords are referenced in this case.
    Bugzilla - Wiki - Downloads - Before posting... Search!

  4. #4
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    sry started elaborating on the falback auth method because you mentioned it

    so you'd have like:
    imapsync --host1 source.server.com --user1 username --authuser1 adminusername --password1 adminpassword --ssl1 --port1 993 --host2 destination.server.com --user2 username --password2 password --authmech2 PLAIN --ssl2 --port2 993 --syncinternaldates --subscribe --nosyncacls

    or

    imapsync --syncinternaldates --subscribed --host1 sourceserver --host2 destinationserver --user1 userAcctSrc --authuser1 adminUserSrc --password1 adminPassDest --user2 useracctondest --authuser2 adminUserDest --password2 adminPassDest --ssl1 --port1 num --ssl2 --port2

    or you could still manipulate shadow files on the source servers if you want

    be sure zimbraImapSSLServerEnabled is set to TRUE

    AUTH=PLAIN requires TLS encoding, so an IMAPs or STARTTLS command on a normal IMAP connection

  5. #5
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    719
    Rep Power
    9

    Default

    I did 2500 users in 36 hours last July with essentially no user-visible downtime, though the total volume of mail was only 500GB.

    I used a set of queue/lockfile directories to keep 4 simultaneous imapsync jobs going.

    Of the 3 boxes involved, the imapsync host in the middle needs to be the most powerful.

    For the two weeks before the migration, I ran imapsyncs in a loop (one simultaneous job during peak hours, three simultaneous jobs off hours), copying mail from user@carleton.edu on the Cyrus server to user@migrate.carleton.edu in the Zimbra server. The per-user final sync process was:

    - Pick the next available user from the "todo" queue; move lockfile into "doing" queue.

    - Incremental imapsync, AUTH PLAIN as the cyrus admin user on the originating end, and as a Zimbra admin user on the destination. Here's my command line. ldap2migrate.pl populates the Zimbra directory from our enterprise directory (trivial script, but probably only locally useful), and all the regtrans2's are necessary to work around folder names that conflict with default calendars/addressbooks/etc and the fact that ":" seems to be illegal in Zimbra folder names.

    Code:
    #!/bin/sh
    ssh zimbra@mail2 "(ldap2migrate.pl carlnetid=$* | zmprov)"
    imapsync --host1 original-imap.carleton.edu --host2 new-zimbra.carleton.edu\
     --buffersize 8192000 \
     --user1 $* --user2 $*@migrate.carleton.edu --nosyncacls --noauthmd5 --ssl1 --ssl2 --sep1 /\
     --exclude '^Trash-not-migrated$|^Trash$|^trash$|^Deleted Messages$' \
     --syncinternaldates --authuser1 cyrus --authuser2 zimbra-superuser@carleton.edu\
     --useheader Message-ID --useheader Date --skipsize --subscribe --prefix1 ''\
     --expunge2 --passfile1 /root/cyrus-pass --passfile2\
     /root/zimbra-pass  --authmech1 PLAIN --authmech2 PLAIN --delete2\
     --expunge1 --regextrans2 's/^Calendar$/Calendar (old)/' \
     --regextrans2 's/^CALENDAR/CALENDAR (old)/' \
     --regextrans2 's/^Contacts$/Contacts (old)/'\
     --regextrans2 's/^Notes$/Notes (old)/'\
     --regextrans2 's/^calendar$/calendar (old)/'\
     --regextrans2 's/^contacts$/contacts (old)/'\
     --regextrans2 's/^notes$/notes (old)/' --regextrans2 's/: / /g'\
     --regextrans2 's/://' --regextrans2 's/^Contacts\//Contacts (old)\//i'\
     --regextrans2 's/^Calendar\//Calendar\//i'\
     --regextrans2 's/^Notes\//Notes (old)\//i' 2>&1\
     >> /var/log/migration/$*-cyrus-zimbra-migrate 2>> /var/log/migration-err/$* ||
    mail -s "imapsync errors $*" admin-account@carleton.edu < /var/log/migration-err/$*
    - When that incremental imapsync is done, rename the Cyrus account to user.migrate (which has the effect of locking out the user); set forwarding for user@carleton.ed to a mailer that simply queues; create a new Cyrus account "user" with a couple placeholder messages to the effect "your mail is being synchronized to the new server, please try Zimbra in 15 minutes"; and run another incremental imapsync

    - When the second imapsync is done (typically only a minute, though multi-gigabyte mailboxes took as long as 30 minutes simply to list), rename the Zimbra account from user@migrate.carleton.edu to user@carleton.edu, then set forwarding on the original system to deliver to Zimbra

    - Move the lockfile for user to the "done" directory

    If your source server is something other than Cyrus imapd, then renaming accounts might be a little less trivial, but you ought to be able to do something similar.

  6. #6
    Join Date
    Nov 2007
    Location
    Atlanta, GA
    Posts
    5
    Rep Power
    8

    Default

    O.K., I'm trying to do your suggestion mmorse with the migration domain. Oh and thanks for the dkarp. After we had created our working string of imapsync commands, I had forgotten about the authuser command.

    We have figured that it is best to go with the migration domain since we will have some users who will try to log in and mess with e-mail in the middle of the migration. This way we can control it better. However, I'm having problems with the second domain.

    I created it and have a couple accounts that are part of that domain (in our case, it is called mm2). Even though there is a local password for say audit@mm2, I can not log in. However, if I change my domain to the real working domain, I can log in.

    I get a 2 NO AUTHENTICATE failed error message when part of the mm2 domain.

    You did have a typo with the account provisioning, correct? That should be zmprov ca user@migration.domain.com password, right?

    So, what am I missing. We are supposed to start this up tomorrow. :/

    Thanks!

  7. #7
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    This in the web-client/did you type in the full audit@mm2.domain.com?

    So it sound's like your going to combine techniques, which is perfectly fine.

    So as your going to use auth plain for the imapsync, you could simply use:
    zmprov ca user@migration.domain.com ‘’

    Quote Originally Posted by Rich Graves View Post
    -When that incremental imapsync is done, rename the Cyrus account to user.migrate (which has the effect of locking out the user); set forwarding for user@carleton.ed to a mailer that simply queues; create a new Cyrus account "user" with a couple placeholder messages to the effect "your mail is being synchronized to the new server, please try Zimbra in 15 minutes"; and run another incremental imapsync

    - When the second imapsync is done (typically only a minute, though multi-gigabyte mailboxes took as long as 30 minutes simply to list), rename the Zimbra account from user@migrate.carleton.edu to user@carleton.edu, then set forwarding on the original system to deliver to Zimbra
    Not to add 'one more alternative' to your list, but if your worried about them messing around during migration, you could also provision the accounts then set:
    zmprov ma user@domain.com zimbraAccountStatus locked

    Locked means that login is disabled, but mail is still delivered to the account.
    (You don't want it but there's also maintenance which is also login disabled, BUT mail is queued at the MTA. An account can be set to maintenance mode for backing up or restoring the mailbox.)

    Of course set whatever you need to on the other end to prevent usage (lock/rename/Rich's above method, etc)

    Side note: Judging by your email (college of computing at gatech) etc, I'll wager you're on the Network Edition...you do know you could request assistance/professional services for your setup/migration.
    Last edited by mmorse; 11-06-2007 at 07:03 PM.

  8. #8
    Join Date
    Nov 2007
    Location
    Atlanta, GA
    Posts
    5
    Rep Power
    8

    Default

    Well, we are into our migration now and we have successfully migrated two mail servers that lived on old SunBlade 100's, about 600 or so users. It took a bit longer than planned because imapsync just beat the hell out of the processors on the box. We could only move about 8 users at a time from each box.

    We did a replace of the shadow password with a known to login to the source box. We created a migration domain on the Zimbra server and a migration admin account. Once the e-mail was migrated, we switched the users to the correct domain. It worked great as far as that is concerned. Just this morning, I finally found this wiki page Split Domain - Zimbra :: Wiki that is helpful for others.

    The login info you gave me about using the full name and migration domain to login to the web portal worked. Thanks. Didn't think about that at the time. I was running on too little sleep by then.

    We have refined our process. Due to users being able manage their mail how they want, we could not account for all of the combinations that we would find. So instead of trying to get all of their mail, we are only getting their inboxes and providing instructions to the users on how to use a thick client to migrate their own mail. It greatly sped up what we are doing and removed any guess work on whether or not we got everything. It puts that responsibility on the user who should be savvy enough to figure it out.

    We are running two servers, one is open source for a particular group of users and the other is the Network Edition for a different group of users. We started migrating to the open source first. Thanks for the info to help get us going. Now if only the faculty would get out of our way, we can continue the migrations.

  9. #9
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    That is an interesting approach - wish I had the liberty to skip stuff in migration like that! Hm, it sure would reduce storage on the SAN's...as I know some users would never bother to import their old stuff...

    -I'm going to mark this particular thread solved - glad you're on your way!

  10. #10
    Join Date
    Oct 2009
    Posts
    1
    Rep Power
    6

    Default

    Quote Originally Posted by dkarp View Post
    Zimbra does support AUTH=PLAIN. You should be able to authenticate as a target user using only an admin username/password and the target username -- no user passwords are referenced in this case.
    Hello ...
    I try to synchronize 2 servers (QMail --> Zimbra CS) with imapsync... and I have a problem with imapsync AUTH PLAIN...
    Here's my command :

    Code:
    imapsync --syncinternaldates --host1 myhostSrc --host2 myhostDest 
    --user1 userLoginSrc --authuser1 adminLoginHostSrc --password1 
    passwordAdminHostSrc --user2 userLoginDest --authuser2 
    adminLoginHostDest --password2 passwordAdminHostDest
    With the option authuser1, only the mails of the "admin" (authuser1) are synchronized and not those of the user "user1"...

    Don't understand why... have you got any idea to resolve this problem ?

    Thank you.

Similar Threads

  1. Migrating users with imapsync... without passwords?
    By misleb in forum Installation
    Replies: 6
    Last Post: 08-12-2007, 09:03 AM
  2. migration without passwords
    By alam in forum Migration
    Replies: 4
    Last Post: 07-31-2007, 11:15 AM
  3. 4.0 RC1 imapsync with admin???
    By kirme3 in forum Administrators
    Replies: 37
    Last Post: 07-19-2007, 10:52 AM
  4. need advice on configuring zimbra to work with fax server
    By pheonix1t in forum Administrators
    Replies: 0
    Last Post: 07-11-2007, 08:46 PM
  5. Dump users & passwords
    By avisser in forum Administrators
    Replies: 1
    Last Post: 02-17-2006, 03:17 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •