Results 1 to 6 of 6

Thread: From POP3 and 100 scattered mailboxes to Zimbra

  1. #1
    Join Date
    Jul 2006
    Posts
    3
    Rep Power
    9

    Default From POP3 and 100 scattered mailboxes to Zimbra

    Hello,

    This is my implementation plan for a major upgrade I'm doing to a company's e-mail server. I haven't gone through with it yet, and plan on doing it tomorrow, but I'd like to know what you think. If it works out well, and contains best practices, I'll see if I can post this somewhere in more detail, maybe on a wiki, for anyone who ever has to go through a scenario anything like this.

    Anyways, I've inherited a situation with a company with a POP3 server used for mail, apparently set up on qpopper and running what seems to be inherently open relays, running on an old FreeBSD box, with Vexira doing anti-virus work.

    Since this is POP3 only, the user's mailboxes were not centralized -- they're scattered all over the place, unsychronized, on mailboxes on the network, on the workstations, on laptops, at people's houses, and so forth.

    Oddly enough for the situation, this place has about 100 users and has somehow managed to work ok, minus numerous inconveniences, drag on the network, and massive use of storage space. After all, a 10MB file, base64 encoded to 20MB, and sent out to 100 different mailboxes, does take up 2GB under this sort of system.

    So, I have been assigned to recreate the mail server there to solve these problems and centralize their mail. I was provided a Dell Poweredge SC430 with a Pentium D 2.8, 2GB DDR533, and 2x320GB 7200 16MB cache hard drives that I set up in RAID1.

    I was planning on a typical old IMAP/Squirrelmail server, but however, looking around the net for what was new with open source mail, I found Zimbra. It floored me with its features, even though it would meet my original needs perfectly. I was aiming just for centralized mail initially, but it opens up the possibilities of more features and considerations of the commercial version in the future. Plus the install was much easier than the typical install of those servers, which is about as much fun as, say, a by-hand install of gentoo.

    Since they had some Debian boxes around for a samba server and some other things, plus they won't pay for RHEL, I set up the open source version on a custom version of Debian Sarge 3.1 with kernel 2.6.14-i386-smp upgraded to 2.6.17-i386-smp. I had to choose i386 because of zimbra support, even though Pentium D's are 64-bit chips.

    I could post more details of how I got it running, but if you're in the same situation, google for Poweredge SC430, iso, and Debian.

    My additional requirements for this plan were:

    * Minimum downtime. Certainly no more than 4 hours.
    * High levels of security.
    * Remove all the mailboxes from the network file server.
    * Their old mail gets copied over to the new server.
    * Web interface is optional, but not to be a new requirement.
    * They can be able to access their mail by Mozilla, Thunderbird, or in some instances, Eudora.

    So, I started reading all of the forums and wikis here deciding how to set this changeover up.

    Situation Details

    How it's currently set up involves numerous mozilla profiles stored in home folders on a network file server, containing their settings and mail. However, a number of them are in non-standard network folders, and a number of them are also on their hard drives -- or even worse, half and half, meaning profile on the network and mailboxes on the hard drive, or vice-versa. All they know is that they click Mozilla or Thunderbird, and they get their mail, even if in the background 10MB mails are multiplying into 1-2GB worth of data when sent to every user.

    So, this means no neat automatic scripts. No imapsync, no nothing. Not all the computers are on the windows domain, so I couldn't rig anything consistently in that manner, either.

    However, I was, however, at least able to find some of the locations of their mailboxes on the network. I decided against using mbox utilities like zimbra's inject utility (zmlmptinject?) to split them up, since they didn't seem as reliable, and it would likely take me time to script it properly. Plus people can't be expected to compact their mbox'es, despite my pleas for even basic cleanup, meaning they could end up moving over all kinds of deleted stuff, unless I'd have a way of reindexing them by referencing mozilla's msf indexes.

    That means, basically, creating new profiles on their desktop, creating dummy accounts to load up their old mailboxes, then dragging and dropping them from their old folders to the mail server with their mail client. If anyone knows a better way in this situation, please tell me now -- I set aside an entire day on the weekend when the office will be out out to go desktop-to-desktop, though I have 2-3 people to help me.

    I hope I didn't miss anything that would prevent that, though I did RTFM as much as possible.

    Nevertheless, I wrote a throwaway-code batch file script to set up a new account for them that sets them up to point both to their old account's mail and their local folders for the ones I could find on the network. For mozilla, all you have to do to script a new profile is create a directory, write a prefs.js file in it, and use it as your profile directory. I prototyped it and put in variables based on dos environment variables like %username%. This will allow for some profiles to be auto-created, though I do have procedures for setting up by hand when it fails.

    As far as another automation thing goes, I was able to pull all their data from their database system, however, to use that to write a script I could dump in zmprov commands with on the server set up side. This script saves tons of time and makes the install of zimbra practically instantaneous.

    I just used php to echo the strings out after reading them from a database to their zimbra-equivalent LDAP formats. The end result was something like this:

    Code:
    # creating and modifying accounts
    ca user@mailserver.com changeme1234
    ma user@mailserver.com displayName "Joe User" givenName "Joe" sn "User" ou "Department Name" title "Job Title" telephoneNumber "Phone Number" company "Company Name" physicalDeliveryOfficeName "Office Number" street "Street Name" l "City Name" st "TN" postalcode "12345" co "USA" zimbraPasswordMustChange "TRUE"
    # aliases
    aaa user@mailserver.com joeuser@mailserver.com
    # distribution lists
    cdl funnieslist@mailserver.com
    adlm funnieslist@mailserver.com joeuser@mailserver.com
    adlm funnieslist@mailserver.com janeuser@mailserver.com
    I was also able to write mozilla vcards with the same code, which I put in the generated prefs.js files. I used some PHP code that pulled from their data, similar to this, to generate them.

    PHP Code:
        $vc[$accountname] = array("begin" => "vcard",
                                  
    "fn" => $name,
                                  
    "n" => "$lastname;$firstname",
                                  
    "org" => $companyname ";" $departmentname,
                                  
    "adr" => ';;' $address ';' $city ';' $state ';' $zip ';USA',
                                  
    'email;internet' => $accountname,
                                  
    'title' => $title,
                                  
    'tel;work' => $phone " x" $extension,
                                  
    'url' => $companyurl,
                                  
    'version' => '2.1',
                                  
    'end' => 'vcard');

        foreach (
    $vc as $k=>$v) {
            echo 
    "\n<br>";
            foreach (
    $v as $k2=>$v2) {
              echo 
    rawurlencode("$k2:$v2\n");
            }
        } 
    Currently, I also have this all running on a test server for testing various things. I never found a clean way to change the domain name, so I can't do the porting first and then deploy.

    So, my overall implementation plan is something like:

    * Put the new mail server on the KVM switch
    * Change the network settings of the old mail server to put it aside.
    * Change the network settings of the new mail server to replace the old mail server.
    * Uninstall the test version of zimbra with
    o ./install.sh -u
    * Reinstall version of zimbra and set it up in place of the new mail server with
    o ./install.sh
    * Change all the ports except smtp and 7071 (I couldn't find how to change this) from their default for paranoia purposes.
    * (note: someone should not allow the installer to quit until an admin pass is entered. otherwise, it seems like it fails)
    * Use a hacked up script from http://easyfwgen.morizot.net/gen/ to make sure no traffic goes to anything but them and ssh (also on a non-default port).
    * Run my huge zmprov script to recreate the users, aliases, and distribution lists.
    o sudo -u zimbra /opt/zimbra/bin/zmprov < /root/zmprov.txt
    * Check to see if it's running and log in as the admin user
    * https://mailserver.com:7071/
    * Global Settings -> Ban all attachments save for jpg and mov, add zip, enable spam filtering, and do anti-virus updates hourly.
    * Turn off logging services to speed up the transfer.
    * Restart zimbra:
    o sudo -u zimbra /opt/zimbra/bin/zmcontrol stop
    # ps ax and kill any zimbra processes
    sudo -u zimbra /opt/zimbra/bin/zmcontrol start
    * ** Go from workstation to workstation, running my mozilla profile setup script, and once their mozilla profiles are set up, copy by hand from their old mailboxes to the zimbra server.
    * Restart logging services
    * Restart zimbra as in above.
    * Fetchmail or otherwise import the undownloaded mail off the old mail server's IP that might have been sitting there. Worst case scenario, I re-import this by hand.

    Sounds like a nightmare scenario, eh? At least in the workstation to workstation part. Well, Zimbra sounds like the cure. I really can't wait to move them off their current setup!

    The only problem I've had, by the way, is that it shows the copied-to-server date under Zimbra instead of the Sent Date, though the real sent date remains intact, after copying from Thunderbird. Are there any plans to use the sent date from the message instead in the future, or ways to do this currently? The only thing I ever saw for it was a flag with imapsync, which unfortunately cannot apply here.

    If you can make any suggestions, or find anything I'm doing flagrantly wrong, please let me know.

    Thanks,

    -Peter

  2. #2
    Join Date
    Oct 2005
    Location
    Calgary, AB
    Posts
    232
    Rep Power
    10

    Default

    Well Peter it sounds you like you really did your homework. I envy your scripting ability!

    When I moved our company to Zimbra it a very similar situation, with the workstations holding the mail files and in a few instances the mail files being on a file server. Only difference being that ALL of the mail clients were Outlook clients.

    I really did a "tough love" number on my users though and you may want to discuss it with your management team there. Basically, all NEW employees are "forced" to use the web interface. I use it myself and love it, but by my using it I can easily do away with the frivolous complaints.

    I also "forced" large portions of our departments onto the web interface. Basically, if you are not an "executive", you have no choice in the matter. Some key points to help promote such a drastic action are:

    1. If you use the web interface, I will back up your mail. You won't lose it if your persoanl desktop happens to crash. If you continue using Outlook with it's rediculous .pst files......don't come crying to me.
    2. If you want to access your e-mail from more than one computer or from remote locations, use the Web Interface not Outlook. If you insist on Outlook, you will have access to your e-mail from one machine only.


    MOVING 5 Years of Crap from Client to Zimbra:

    - don't do it. Once you get into peoples files, you will be blown away as to how much junk they are keeping in their bloated mail clients. It will take too much time per machine to sit there trying to figure out what is being moved and what is not.

    -what I did was again "tough love". Basically I sabotoged the old mail client so it could no longer "receive" mail and told the user that if they needed to look at a message for historical reference, then open the old client and search away.

    The administrative nightmare of your current situation is instantly resolved with a Zimbra implementation. However, by NOT using the Web interface, you are only using a portion of it's capabilities. Besides, we are talking about e-mail here, sending and receiving for business purposes....not having fancy mail clients with smiley faces and flying saucers buzzing around every time someone hits their "Send" button (IncrediMail...blah).

    I think that implementing the web interface as a mandatory function of most users is definitely woth a shot. At the very least you may end up with a 50-50 split or maybe even better. That means that instantley you are reducing your workload/email client support substantially.

    Right now I have about 70-30 split, with 70% using the webclient and 30% still on Outlook. It is fantastic from an administration standpoint and the complaints are dropping off fast. It is just a matter of "getting used to it", plus when users start discovering how wickedly cool "conversation threads" and "Tags" are they begin to prefer the web client.
    Cheers,

    Travis

    "Let's look at this from a standpoint of "Status". What exactly, on the Space Craft, IS working?"
    -Flight Control, Apollo 13

  3. #3
    Join Date
    Jul 2006
    Posts
    3
    Rep Power
    9

    Default

    Thanks.

    I'm a software developer by nature, but I'm pretty good at networking stuff, too, being something of a 90's teenage linux guru, so it comes naturally for me to script everything I can. Plus when it comes to homework, I'm one to always RTFM. Or RTFW(iki). Or RTFF(orum), heh.

    I really wish I could go the "tough love" route to the web client right away without porting their mail, but there's some resistance preventing that. For going to the web client alone, there'd have to be retraining, and some people generally don't find it as smooth and as well-featured as desktop mail apps, even though it has some godly AJAX/SOAP/etc capabilities.

    After all, I showed the PO approval stuff to my brother, who used to work for Depot America, and he said it'd save them a fortune in time and accuracy if they didn't have to copy and paste stuff like that out of e-mails constantly into their CRM app. Some consultant/contractor could probably make some megabucks pitching this software to such places, especially with the management-pleasing commercial support and capabilities available for something like a nickel per seat per day.

    But BTTT, this company still has a number of low-end machines that just barely cut it for their requirements of XP -- 256MB RAM P3-450's, Celery 600's, etc that didn't fare entirely too well against the heavy (but f--ing amazing) AJAX of the web client. Those boxes are pokey enough running Thunderbird and all its RAM-for-speed tradeoffs, even, as long as they're not chunking in the swap.

    Still, I'm going to try the web interface on some of the home/mostly-offsite users. I told them I don't do housecalls.

    They also wanted it somewhat transparent, meaning ideally they'd get back to their workstations with the same mail client and the same mail, just being able to access it anywhere from now. Training them on all these features, no matter how cool they are, was outside the scope of my current project, but the thing I love about Zimbra is that you can meet your basic needs, but have an awesome backend for it and plenty of potential for the future.

    And I probably wouldn't be caught dead dealing with Outlook's crap for a project like this, and fortunately they didn't either back in the day, especially with all the worms running through it -- thankfully, they had foresight to move to Mozilla and eventually Thunderbird on some of the more-recently setup boxes, after they moved off of Eudora.

    The only difference I knew even from the start was reconfiguring their mail clients for IMAP, which even with zimbra, is mostly taken care of by walking station to station and just running \\server\folder\zwizard.bat. Thank God that mozilla's cool enough that you can just script a prefs.js file into a directory and have that do all the work.

    And I totally hear you about junk files. I saw mailboxes that were 1-2GB in size, since e-mail seems to be the poor man's file transfer protocol in most cases -- after a few years, it can really add up. Especially when they leave the spam there. Fortunately, most of them were compliant in cleaning them out, as they're really nice people, and I provided good instructions on how to do so, though it took about two weeks of nagging.

    You can reduce the sizes of an Inbox pretty significantly namely sorting by size and deleting them, especially in the Sent box that no one ever cleans out. Plus some don't have any qualms about hitting Ctrl-A and hitting delete on their mailboxes as I suggested -- I left it to their discretion to do so.

    Still, it's going to be hell to port that stuff over. I'm crossing my fingers on it a bit.

    Thanks for the advice, though, it's much appreciated. I'm glad someone else had to go through as bad, if not worse, of a mail overhaul than I will. I was getting a little shaky reading nothing but nearly one-command imapsync stories.

    And again, thanks Zimbra guys, if you're reading this! Your product's a killer app on so many levels. It's going to be a boon to the open source business sector, especially the kind where contractors set up rock-solid, low-maintenance, low-expense servers for real-world mid-sized/small companies that just want some box to run in the background that if it happens to break, they can just make a call, kind of like how supermarkets and stores do. Just going basic with it and giving those businesses Exchange-like capabilities on the cheap is a goldmine not only for the contractors, but for them as well. Not to mention the non-profit sector, which has zero budget for such things but could really use it. And, as mentioned, there's tons of potential for the big guys to go commericial with this.

    -Peter

  4. #4
    Join Date
    Nov 2005
    Posts
    108
    Rep Power
    9

    Default

    spiderman,

    With regards to your copied dates vs original date issue... do a search in these forums for "zimdates". It's a script that someone wrote to modify emails and add a X-Zimbra-Received header to the message. This header gets used by zmlmtpinject as the original receipt date of the message.

    It's designed for mbox format messages but you may get some idea how to use the info in your situation.

    Good luck with your migration!

    jimbo

  5. #5
    Join Date
    Jul 2006
    Posts
    3
    Rep Power
    9

    Default

    I unfortunately didn't have time to mod the mboxes. I wish I had know sooner. Oh, well.

    So, here's how it went, off the top of my head:

    At 1200, I changed the IP on the old mail server, ran /etc/netstart, and changed it to some extra external IP we had. I then set up the networking on the zimbra box, and moved its cable over to the DMZ.

    When it prompted me for the domain with no MX address, I put in company.com instead of mail.company.com. It worked as expected.

    I set up the admin account as something other than admin, set the password, and changed all the ports.

    It wasn't entirely working, so I had to killall rpc.statd and portmap, then I ran update-rc.d portmap remove to make sure it didn't come back to haunt us. I also set up a zimbra script in /etc/init.d to start up in case of reboot.

    I zmcontrol stop'd twice, killed off any leftover processes, and zmcontrol started.

    Then the mail queue monitor and some other things didn't work. I seached the forums and found that it does expect ssh to run on port 22, so I had to deal with that. The server runs chkrootkit out of cron regularly, just in case something does happen.

    After that, it was sending everything into the deferred queue, so looking through the logs, there were what I guessed were permissions issue. Just to be sure, I ran:

    chown -R zimbra:zimbra /opt/zimbra

    Then after googling for the postfix errors, I found:

    postfix -c /opt/zimbra/postfix/conf set-permissions
    (or something like that for the conf directory)

    After that and restarting, everything worked like a charm.

    We had three other people to help, fortunately. My training on the setup probably wasn't good enough, as it was problematic at first. Others volunteered, but found it way too difficult to account for all these weird possibilities. My written procedures probably would require being rewritten, since the situations were so oddball at times in practice, though I tried to account for it as much as possible.

    My script ran into tons of consequences, as well -- for example, some people were off-domain and had their usernames capitalized, so I had to run the script from cmd by:

    set username=lowercaseusername
    \\server\folder\zw.bat

    And I had to continuously hack at it because of the inconsistencies and a few errors, though I at least had a decent template to do so. But it did set up their mozilla/thunderbird profiles quite nicely, and often we just had to make a single change or two. It was worth it just for the 3-4 people per person it did work automatically for.

    Message filters could be copied pretty decently from a msgfilter.dat thing, too, with a bit of small post-setup. I missed that at first. And I also had to account for copying bookmarks.html to the new profile, which I forgot.

    We were moving folders over for about 5 hours, and probably got half of the people in it over. Some people locked their offices and shut down their computers, even though we told them not to. We plan on doing the rest on Monday, and until then, recommending the web interface until they can be set up and have their mail copied. We've also set an experimental policy that new employees will only use the web interface.

    On a few of them, we decided not to move the mail over yet, since they had tons of mail in tons of folders that would be a pain to move. We decided to leave them be where they were, and either copy them later, or just have an old network folder for a few people.

    Case in point, the server actually quit receiving transferred mails on other computers when we copied over 20,000 mails (no exaggeration), some of which were possibly close to 10 years old, from someone who insisted on keeping them.

    Still, we've probably copied gigs of mailboxes, and it hasn't taken up a gig yet in extra storage space. When stuff's not copying, it's fast, too, though some of that can be attributed to my scripted profile caching it by default on the hard drive in mozilla's offline folders.

    Monday's going to be hell to pay, but it's all worth it. Once it gets going, it's solid and no longer a total mess. I touched base with the users while I was upgrading some of them, and they were really excited about centralized e-mail when I showed it off, and especially getting e-mail from home by web access. I gave out post-it notes to the webmail url, even, and told them to give it a try.

    Sure, centralized webmail's standard for most businesses, but it's neat to see it being launched as a new thing somewhere.

    All in all, it went pretty well for being a nightmare scenario.

    If anyone thinks I should write up this in detail and post it here or maybe even on the wiki, because of the scenario, let me know.

  6. #6
    Join Date
    Aug 2005
    Location
    San Mateo, CA
    Posts
    4,789
    Rep Power
    19

    Thumbs up

    It's late, it's Friday, I need a beer. After reading this I think you sould have a couple. Spiderman... this is absolutely amazing!!!! Great account of all the details. You deserve a t-shirt or two just for writing this up. Please PM me your physical address (no PO box please), and what sizes you want.
    Last edited by KevinH; 08-12-2006 at 09:17 AM.
    Looking for new beta users -> Co-Founder of Acompli. Previously worked at Zimbra (and Yahoo! & VMware) since 2005.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •