HA Open Zimbra Idea - MailStore via http rather than file system.
I have an idea for modifying the mailstore for imap for added scalability and failover. It would use web servers to store the actual files rather than the standard file system.
I'm hoping to generate some interest and knock this out. I think it should be a pretty simple change and give HUGE gains. It would def be a bit slower to access the message file via http rather than local filesystem, but should be pretty fast and gives simple solution for replication/failover/HA on the open source version of zimbra.
It would work something like below.
- Modify "com.zimbra.cs.store.StoreManager" so that it stores message file via post to web server. It should post to two different servers, for failover.
- The two urls to the file are stored in the database with the messageId.
- When the imap server needs to "get" a message file, it will requrest it from one of the urls listed in the database. If one fails it will failover to the other.
- If a server is low on storage, just add another plain old web server. It would just need a script to receive uploads via post. Then add the server to the list of available web storage servers :)
- Mysql and Ldap are replicated to a second server for application failover. Storage fails over using the above solution.
I'm going to start digging through the code this week to see how doable this is. Let me know if you have any interest in helping out. I could def use a hand from someone who has worked with the Zimbra code.
SAN would work great, but...
I def recognize that SAN is the most common way to go for HA in a mail environment and definitely a good way to go. I'm in search of an in expensive alternative.
My issue is that while the SAN direction seems to be the most common for HA, the infrastructure is prohibitively expensive for a lot of implementations and rather complex to set up, especially in a managed hosting environment.
My goal is not just to provide an HA solution, but one that can be setup very easily with little expense in a managed hosting environment.
So in response to your points:
1) "Application layer aware of of too much."
Well it is already aware of the file system. I'm not sure that having it aware of a url is any more complex than having it aware of a path in the file system.
2) "Post to multiple servers could fail"
True, this is part of the reason I posted, to work out details. I had also considered a replication daemon. It could only copy to one location, the daemon could replicate.
3) "Why not user disk replication"
I have considered this. We used disk replication one of my previous companies to store photo and video file backups/failover. It did work, but required custom OS installs to support and were a bit finicky. We eventually moved to a system like I am describing. It was easy to implement, could be setup on any server w/o kernel other OS modifications and worked very well.
4) "what happens if a post fails"
No problem, it can be replicated later. So long as it is in one location the replication can happen as soon as the secondary storage is up.
5) "what happens if zimbra dies"
This was the reason for the replicated MySQL and LDAP. The second machine would have a hot copy of Zimbra waiting to go. I was figuring for starters two frontend zimbra boxes. One active the other ready to go. If the active went down the secondary would be ready. All the message files would be replicated to the web storage servers separate, so they'd be ready to go.
So yeah SAN sounds good. I think the http idea could provide some additions though:
- easy - if you are low on storage just add a webserver. No need to mount new drives configure new servers... just add it to the list of available storage servers in your Zimbra config.
- cheap - storage is cheap these days. 1TB of storage could be added with shared web host for $7/month.
- managed hosting - this would work with managed hosting enviroments. It would be cost prohibitive to setup a similiar solution with SAN. I'd likely need a rack, servers and custom hardware.
Anyway, I'm gonna give it a shot... my main concern right now is actually speed. We have been successful with this model for photos and videos. The upload speed to the servers could be a deal breaker. If it is fast enough on read/write, I think it should be pretty straight forward to have this running soon.
Next Steps: Starting tests and dev
I'm going to be approaching this project something like this. Of course on my "free time" so could take a bit.
- get Zimbra dev environment setup
- do successful unchanged build
- replace MessageStore filesystem write code with post to remote apache server using jakarta httpclient
- replace MessageStore filesystem read code with http read using jakarta httpclient
- build server
- test delivery and read
- move http storage server list into configuration.
- setup replication code, at delivery time multi-writes.
- test build
- write http replication daemon to run in the background, taking care of any multi-writes that don't make it due to connection failure.
Just a quick overview.... off I go. It'll prob be a week or so before I have any working code, but have to start somewhere.