Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: Shared Storage space for MailStore or building High Availabiltiy

  1. #1
    Join Date
    Sep 2011
    Posts
    256
    Rep Power
    4

    Default Shared Storage space for MailStore or building High Availabiltiy

    Hi Folks,

    I am trying to configure HA MailBox server with the help of external Load balancer and due to the budget constraint we are evaluating the simpler ways to configure the scenario rather than investing on SAN at this moment.

    We are evaluatiing NFS storage space where two MBOX server would share common /opt/zimbra/message and /opt/zimbra/index. Or thinking to go with DRBD. Please let me know if NFS can work well in this scenario? Can we share the same data store with two servers and create the high availability?

    Just would like to know the forum's opinion on the same. Also, how can we build High Available Zimbra environment in Geographically dispersed locations and/or with Two ISPs?

  2. #2
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    165
    Rep Power
    7

    Default

    I am trying to configure HA MailBox server with the help of external Load balancer and due to the budget constraint we are evaluating the simpler ways to configure the scenario rather than investing on SAN at this moment.
    I'd still suggest to investigate at least decent SAN solution, as often people hours turn out to be more expensive, rather look for decent automation implementation. Although, it adds some layer of complexity any way.

    We are evaluatiing NFS storage space where two MBOX server would share common /opt/zimbra/message and /opt/zimbra/index. Or thinking to go with DRBD. Please let me know if NFS can work well in this scenario? Can we share the same data store with two servers and create the high availability?
    This forum is quite full with DRBD and HA discussions, although, you have to count on some consequences and work overhead (more details below). Actually it depends on what you want to achieve. Here is a good step-by-step discussion on DRBD implementation. http://www.zimbra.com/forums/adminis...rbd-howto.html and another one http://www.zimbra.com/forums/adminis...mbra-drbd.html

    As per Zimbra wiki, there is recommendation to avoid NFS storages, but I must say, that it all depends on actual configuration. Still, you have to cope with potential file locking, thus thinking about some kind of cluster file system, to be clean.

    If you think of geographically dispersed locations, this probably would not be a SAN issue, as you mention budget constraints.

    Just would like to know the forum's opinion on the same. Also, how can we build High Available Zimbra environment in Geographically dispersed locations and/or with Two ISPs?
    Please take in mind, that best practice asks to keep all ZCS servers within the same versions, to eliminate additional issues that may rise in case of troubleshooting. At least major ones. But I managed to link two different versions together (had an issue with poxy and static js/css files). The biggest pitfall here from terms of usability, is in fact, that all ZCS servers still communicate between themselves, especially when finding LDAP for user authentication and authorization, and if these servers are located in really wide area (say one in EU, one in US), you have to count on significant network latency while LDAP discovery & authentication process completes. It would be more wise, to spread these servers because of user location, not because of load balancing, or the other way, one session might be fast, another - slow. Consider LDAP slave implementation on remote servers, to gain increase for auth process.

    DRBD

    DRBD was working well in my former deployment, but I decided to go HA beyond Zimbra servers - on hypervizer level. Supported way of DRBD is Hot/Cold solution, which sync disk block devices (another option is to look at Ceph or GlusterFS - have not tried them, but recommended to look at from High Availability | www.hastexo.com). You may sync all /opt/zimbra folder. The main problem, if it suites you, is that when HA discovers, that main master server is down, it launches virtual IP migration and boots Zimbra on slave server (yes, Zimbra does not run on both servers, as DRBD for that time didn't support active/active solution). Taking in mind this, in my case it took about 2 minutes, to fully load Zimbra server and recover operations. So this is not hot snap in way. You may sync data over internet too (dealing with security kept in mind), but in some cases if net connection is poor, it might not be 1:1 sync of servers.

    Planning your HA infrastructure, keep in mind, that you have to upgrade servers too, and potentially with minimal downtime. So it would be wise to find out things regarding your chosen solution.

  3. #3
    Join Date
    Sep 2011
    Posts
    256
    Rep Power
    4

    Default

    I see so NFS would not be an good option and investing on descent SAN would be an fair idea.

    I am trying to achieve HA at LAN as well as at WAN level. Or trying to set up DR site and evaluating option. What clues you guys can provide at WAN level.
    Adding two MTA can be an option but then how can I have configuration of Site A is replicated to Site B.

    Would appreciate if you can forward me any good docs you might have in this regards.

  4. #4
    Join Date
    Sep 2011
    Posts
    256
    Rep Power
    4

    Default

    Further to that, I was searching through forum/wiki about setting up backup MX server for just accepting mails until the primary server comes up. But didnt find any good article how can one build a backup MX using zimbra.

    Has anyone every configured this scenario? If so kindly share the document with us.

  5. #5
    phoenix is offline Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,587
    Rep Power
    58

    Default

    Quote Originally Posted by blason View Post
    Further to that, I was searching through forum/wiki about setting up backup MX server for just accepting mails until the primary server comes up. But didnt find any good article how can one build a backup MX using zimbra.
    You'd do that the same as you would for any Postfix server - there's tons of articles on the web. I would, however, recommend against setting up your own backup server. A backup MX is a hot target for spammers and you'd be better using a professional service for that, you could get that server as an add-on to another service such as DNS hosting. I use EasyDNS for my DNS and they provide a full backup MX that will forward mail to your server when it's back on-line (after a failure) and they retain the mail for up-to five days.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  6. #6
    Join Date
    Sep 2011
    Posts
    256
    Rep Power
    4

    Default

    Thanks Bill need to explore that option as well.

    Further to that as I stated earlier what options can you recommend to set up zimbra in a geographical dispersed locations with HA? Two MX entries with one backup as we discussed? The main concern is how can we sync up MailStore? Though I agree it wont happen in real time and user is definitely gonna loose few mails but that is understood.

    Also even if we somehow migrate to other site how can users on Internet use the same A record in their mail client to send mails? Just like imagine this scenario

    SITE A - domain example.com

    MX = mail.example.com 10 [Site A]
    mail1.example.com 20 [Site B]

    if users have configured their mail clients using mail.example.com to receive mail or send mail in case of failure doesn't look possible to have them again configured A record for Site B in their mail clients. That would start syncing up emails all over again. And even if do the trick in DNS and add two A records with same name with RoundRobin or LB mechanism few users will still be reached to site A which is down hence wont be able to download or send any messages.

    Now the only option then I think of is using DNS Global Traffic Manager which can eliminate the A records which are down.

    Am I right? Just wanted to be clear on thought process. Or any other approach you can think of?

  7. #7
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    165
    Rep Power
    7

    Default

    My general way of thinking according to your scenario would be as follows. Each level of service might be rethought and rebuilt according to your needs or budget.

    Prerequisites
    • This scenario is not a solution for Active/Active case, regarding geo spread servers, but in some extent it may be a case.
    • This scenario does not implement geo load balancing, but it may be rebuilt to its needs.
    • Depending on your needs, knowledge and will to struggle through all these things, in some cases 3-rd party service options might be valuable in terms of IT costs.
    • Part of these ideas are not tested by me anyway, but there should be docs on the net. As well, this is high level theory, and particular issues may rise. Need investigation.


    General
    Several layers of services are used, and it is wise to review each of service separately to tune up to your requirements. Namely they are:
    • DNS - to cover resolution of IP addresses and hosts
    • Load balancing/redirection - if you need kind of balancing and more fine grained geo distribution
    • MTA/SMTP/antispam/AV - incoming/outgoing mails and MX backup
    • Proxy - if you still need to get some kind of web ui optimisation, single point of access or even load balancing within one location.
    • Mailbox servers - actual mail stores
    • SAN/DAS - depending on budget and options needed



    DNS
    DNS is from one side simple, but from other side may be tough to implement and understand. Especially, if you deply systems in NATed environment. Here to say, Zimbra is very DNS dependant, so this is the base, which should be set correctly in advance.

    In general terms, here in this example, DNS is simple, and records are public. Here we have to point 2 things:
    • MTA resolution (mx record for MX - location A, priority 10; mx record for MX - location B, priority 50). If you need kind of Round Robin balancing, you may point both MX records with the same priority.
    • A records for MXes and for mailboxes. Actual IP resolution for mailboxes should be done to load balancing location, as they will be used in round robin manner.


    Note on Geo distribution - yes, there are DNS service providers, who offer geo distribution of DNS resolution. But you have to keep in mind, that mainly they distribute DNS in locations - US, EU, Asia, MidEast, etc. In this case, if your both locations are set in EU, it actually does not make sense without custom redirection facility, as you may not redirect your users based on these DNS services. They are too wide, or custom solution would cost you a fortune. We use dnsmadeeasy.com services, and they are very, very fine. Including options for backup MX and geo distribution. More over, they offer for decent price geographically spread DNS servers, thus making DNS resolution closer to user - Anycast system. So without your custom redirection it would be not possible to solve it with DNS services only.

    Load balancing/redirection
    On this layer you have to implement custom solution (not any particular in mind for now), where you get User requested URL and source IP, and redirect them according to your policies - whether geographically, or according to particular HA rules or load. You may use Linux HA solution, or do something with regular web servers. Nginx will offer you POP and IMAP proxying too.

    Main point of this, is if your user asks for mail.example.com A record, it hits this load balancing layer (these might be geo distributed too, and work in round robin manner). And after, this layer redirect user to particular mailbox server URL. Or substitute particular IP address.

    MTA...
    Here you have choice - use both MTAs from Zimbra stack, providing, that you install separate servers in separate locations, and additionally install and configure Zimbra LDAP servers with them in Slave configuration - e.g. no writes, no user creation, just pure read for authentication purposes. This should provide domain/user data, if main LDAP server is down, or it is too far, to make authentication process fast.

    You can manage your custom mail retention policies on these Zimbra MTAs. And if, for any case, mailbox server is not accessible, you provide storage for e-mails in queues, untill they get bounced back to sender.

    As Bill mentioned, you have to be aware of SPAMmers, thus, you may implement the same Zimbra rules for antispam/antivirus, including particular services.

    MTAs may work in round robin, or first/second manner, regarding your DNS settings and priority. If the same (say 10) for both MTAs - round robin should work. And it still helps you to distribute MTA load, if machines are not fat ones. If 10 & 50 - the other way, but still working on both machines, and after delivered to particular mailbox.

    As you have a situation, that one mailbox server might be down, you have to implement scripts to let MTA know, that there is another IP address for this mailbox server. Pure DNS solution in this case might not work, or be too slow. But it should, if you provide same A record for the same mailbox hostname, only with different IPs. Pitfall in such is the fact, that it would introduce round robin process, and your cold standby server might be used for mail delivery. With such script, you may configure systems to direct traffic according to your needs and policies, beyond DNS.

    Proxy
    This layer might not be used, but it depends on count of servers and users to be hosted. Zimbra Proxy may be used for logical HTTP/HTTPS/IMAP/POP redirection to particular mailbox servers, if you spread your users on more than one. And it happens in transparent way. Still you have to keep in mind, that if geo spread is wide (say US and EU), your users will experience slow service, if they are located in EU, and Proxy redirects them to US location.

    Mailbox servers
    There is one very nice and valuable product, called zeXtras Suite, which will help you to achieve backups (on Open Source) and additional benefits. It, in conjunction with rsync, may be used to kind of mirror mailbox servers on logical level. Plus, introduce new services, which are available only in NE edition, or not available at all. Say, Hierarchical storage management, Chat, Mobile support with Exchange sync protocol, etc. As well it will help you to manage users data moving between servers. It's a paid product, but I personally think, that their pricing is great.

    SAN/DAS
    This layer relates to storage choice. Mainly kept within budget constraints. Here to point, hierarchical storage management would be valuable, if you server large amount of users, thus letting active messages reside on fast storage subsystem (SAS/SSD) and older mails storing on SATAs. From Zimbra point of view, SATA disks are not reccomended, as Zimbra is very Write intensive. Nor RAID5 is an option. But for old mail archive to lower storage costs, it would be benefitial.

    As well you might look for block level data sync on SAN level (DRBD, or propriatery SAN offerings), or go with Zextras Suite on higher level. And yes, DRBD can be configured to sync between different locations. Yet, you have to keep in mind internet channel stability. But in far locations it will not be 1:1 on a second. I'd rather go with Zextras Suite.

    You may look at Starter SAN solutions from vendors like HP, Dell, EMC or NetApp. Or build your own, FreeNAS, OpenFiler, and some others. Benefitial is ZFS usage, if you can manage it, as you may get lower costs for your SAN, as you do not need RAID controllers to buy, which are pretty expensive.

    DAS - direct attached storage may be an option too, if you decide to sync data with zextras suite, but it will not give you flexibility.

    Hope this info may help you on decision. If particular issues are not clear to you, feel free to post your questions. I'll try to look deeper and look for potential solutions.

  8. #8
    Join Date
    Sep 2011
    Posts
    256
    Rep Power
    4

    Default

    that was a nice explanation j2b. However say if I keep MTA or my MX record with same priority and configured Round Robin having two servers at different locations how can I have MailBox synced with each other since one traffic would from Customer A would flow to location A and then next traffic might flow to Location B. In that case how can I keep syncronising MailStore?

    Like

    Domain = Example.com
    MX = mail1.example.com 10 [San Francisco]
    mail2.example.com 10 [New York]

    Does any body have pictorial diagram of forming high available Email Infrastructure? or Tips on configuring the same? Or the ideal setup at MSP/ISP level

  9. #9
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    165
    Rep Power
    7

    Default

    In basic terms, please see diagram:
    zimbra-ha-mta-communications.png

    Explanation
    In basic, and this diagram, MTA1, MTA2, MB master and MB slave are 4 different servers spread in two locations - SF & NY. For simplicity, I provided C class internal IP addressing, but this should be done on external routable IP addresses, or if you use internal ones, you have to provide additional network layer of tunneling, which adds complexity. Please notice 3-rd octet, which indicates the fact, that these are different subnets! (192.168.0.0/24, 192.168.50.0/24 and 192.168.100.0/24 - sync channel not taken into account here).

    MTA level
    These both work as separate servers and both are online. They receive e-mail on behalf of your domain (example.com), and route them via LMTP to mail.example.com, which is configured mailbox server in your Zimbra system.

    Both MTAs deploy SMTP, SPAM/Virus filter and Zimbra LDAP slave configuration for account resolution.

    Mailbox level
    For this explanation, I use Linux HA and Heartbeat to manage vIP (Virtual IP address), where you have to use 3 from the same subnet. In particular example:
    * 192.168.100.10 - physical mail.example.com (master) IP address;
    * 192.168.100.20 - physical mail.example.com (slave) IP address;
    * 192.168.100.100 - virtual IP, which lives on server, who claim himself to be a master in particular segment of time - not on both.

    In this scenario, if mail.example.com (SF/master) is active, it responds to two IPs - 192.168.100.10 & 192.168.100.100 (VIP). Your Zimbra MTAs consider only one IP according to DNS settings - 192.168.100.100. In case master mailbox fails, Heartbeat notices this and claims master to be down. This initiates IP address takeover by slave server, and 192.168.100.100 moves to 192.168.100.20 machine. But for MTAs, it does not matter, as they connect to 192.168.100.100 any way.

    MB server host Zimbra LDAP Master server, and let data be replicated to LDAP slaves on MTAs. As you sync MB servers, it actually syncs LDAP too.

    Overall
    In general this is very basic example, to get you understand this. What you get from here:
    1. you have 2 separate MTAs, which work as load balancers or mail backup for each other (keeping in mind, that you manage message retention policies).
    2. you get 1 mailbox server listening on mail.example.com and 192.168.100.100, where ever they are - SF or NY, but not in both locations (this regarding fact, that you do not use these for load balancing, just for failover, including geo one, as from my comments above).

    Some complexities due to geo spreading
    In real life example, it would be hard to get 3 IPs from the same subnet, but in different geo locations. You may achieve this if using one service provider in both of these locations, and have to negotiate custom routing. It may cost you $$$. And even not all ISPs may do it due to their network topology. And that is why I offered custom load balancing/redirection to be used between MTAs and mailbox servers. There you may implement your custom rules, and redirect traffic to active/online mailbox server.

    Sync of mailbox servers is separate story, where you may find your best suited way to do this. But it would be wise to use other subnets, or links, if there is such an option. If you use DRBD, this is block level sync, and most of cases will be 1:1 copies, but you can not boot other zimbra, while first one is up. As zimbra is DNS dependant.

    The other way, if you rsync data with a help of OpenSource scripts (wiki and this forum) or zextras suite, then you do not need DRBD, but data will not by in sync up to a second.

    In addition to say, DNSMadeEasy offer host monitoring and logical switch over, if one host goes down. It still costs, but I do not consider it a viable HA. It's rather failover thing, which is a bit different, and mainly takes into account time neede to switch over and data consistency on both nodes. I'd not use DNS for HA, and implement some custom ReverseProxy balancing, which again, can consist of 2 load balancers, on in each location, and can work in round robin manner, even with different IPs (a solution for different ISPs).

    Does it make things more clear?

  10. #10
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    165
    Rep Power
    7

Similar Threads

  1. Storage space is getting low
    By aiko in forum Administrators
    Replies: 3
    Last Post: 10-19-2012, 08:40 AM
  2. New hardware/OS/version - shared storage
    By su_A_ve in forum Installation
    Replies: 0
    Last Post: 06-30-2011, 05:41 AM
  3. Extending storage space for Briefcase
    By tenpagyatso in forum Administrators
    Replies: 0
    Last Post: 10-06-2009, 01:03 PM
  4. zimbra failover with shared storage and same ip address
    By twilighthaevhen in forum Administrators
    Replies: 1
    Last Post: 12-08-2008, 11:11 PM
  5. Shared storage and ZCS / load balancing store servers
    By bureado in forum Administrators
    Replies: 0
    Last Post: 08-22-2008, 11:32 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •