First of all thank you for the concise and thorough reply, I really appreciate your time and effort doing this
I was under the impression 32Gb should be quite enough. If I am wrong, I still have time to spec up, but I'd really love to see some sort of a calculation table or something that would justify even more RAM. As for disk IO, I do have some fears there - 8 disks in raid10 are normally quite enough, but with encryption thrown in the middle... I just don't know - never tried it before.
Originally Posted by cyberdeath
The plan is raid1 for the OS plus a raid10 for Zimbra. I do wonder whether or not /opt can be local and unencrypted, and I can simply mount a separate raid10 based chunk for the mailstore separately.
You are making a good choice for the OS RAID & the Zimbra RAID. For larger deployments, I would say, if you can afford it (in disk space and cost), go for the RAID 10 as it outperforms RAID6, even though RAID6 is cheaper in terms of monetary cost per GB.
This is going to be a brand machine (probably a Dell or HP), so I'll be using a proper raid controller, with a BBU and the max amount of cache I can get for it
Encryption is always going to place an impact on your overall I/O. The trick is in mitigation. You have done well thus far by running the RAID10 for performance. The only other thing you could do would be to look into hardware RAID setups (which tend to be costly and are not as "standardized" as linux is...which also applies for software RAID as well).
I don't have the option to play with it unfortunately, once the hardware is spec'd out and in place, I have to bring it all up into production. I've googled a LOT looking for zimbra on LUKS use cases and benchmarks, but couldn't find anything. Seems like I'm the first to do that.
I would say your best bet would be to do the following:
Go ahead and install the OS you plan to use with encryption enabled on a box. Then, run a dd test (or some other variant) to see how well reading and writing does on the encrypted volumes. Then, of course, you'll need to install the same OS on a device WITHOUT encryption to see how well it does in comparison. That will give you a real rough estimate of how well your encryption algorithm is going to perform. There are also some great case studies and tests out on the web so you don't have to go through the effort...I would check that out as well.
There's another point, I do plan to have backups done, and with Zimbra, if I back up /opt restore should be easy. However, for that sort of backup, I'll need to stop zimbra services, and I'd rather not do that. Are there any guides that show how to keep the stdby box sync'd, so real HA can work? I'm pretty sure there are ways to do that with the costlier versions of Zimbra, but I'd rather stick to the OSS builds.
As far as personal experience...encryption does hurt...quite a bit on I/O. However, it all depends on how big your need is. Yes, it may be painful doing a backup or restore....but let's just hope you never have to do one...and also that you proactively keep a standby box stood up and ready to take over should it happen.
If the CEO wants more than two months, he will approve the budget for a better storage solution :) In this case however, I'm not looking for block level, but for _brick_ level (guess I got the term for MSExchange). Meaning, I can restore separate mailboxes in case of need, instead of pulling in the entire setup, which might end up being quite the monster.
If you're looking for block level, you're likely looking for LVM. There's a lot of controversy on this because some say that LVM causes a performance hit while others disagree. I personally do not know because I've never compared the two side-by-side. However, LVM *will* let you do snapshots at the filesystem level. The only caveat is the amount of disk space necessary to keep all of these snapshots...are you planning on adding a third level of low performance storage drives to keep them on? I would strongly encourage it if you go this route and need to keep many months of data. As you well know... 1-2 months is good...but what happens when your CFO or CEO needs 6 months back?
The backups are going to be held in another server, hosted elsewhere
What kind of scripting? a simple rsync to sync up /opt or...? Being an old sysadmin, I'm rather professionally lazy, so I really hate reinventing wheels, especially when there's a chance of my new wheel turning out to be square.
After speaking with a Zimbra engineer about the backups, my understanding is the best (or at least the lesser complicated) way would be to simply use the backups generated by the primary mail server and copying it over to a spare Zimbra server. That way it's just a matter of recovering and you're up. You can even setup the spare to do this automatically as it receives the file (through scripting, of course).
Identical, hardware-wise? The stdby machine is going to be a VM probably, can't afford to have another large server up there. I could keep a zimbra setup on the backup server though, with the services stopped of course. Not sure how all the SSL certs will behave if the setup needs to failover though...
Assuming that both machines are identical:
I would personally setup your primary server how you want it then disk duplicate to the other server (Through whatever means you prefer). Then you'll need to change a couple things (IP address...etc) and then you'll want to setup the script to run and backup your data. (Example: Backing up and restoring Zimbra (Open Source Version) - Zimbra :: Wiki
Yup, no biggie
Please note that this method assumes you do not mind losing some of the day's data (based upon the frequency of your interval backups...which I believe is default at 24h).
It's more about rack space in this case. Can't put a SAN in there unfortunately. Besides, failover, as such, should happen to another geo-location, so a local SAN won't help, and implementing iFCP based solutions for a single MTA is really overkill
Using a SAN and disk replication would assuredly be the easiest from a system administrators standpoint. This can be setup however you want to and as frequently...with all the snapshots you want...etc etc. The only catch is the amount of money it will cost to implement.
Yeah, DRBD is actually working on delivering some kind of geoclustering solution, but they are far from being usable yet. So, so far I'm still thinking in terms of rsync
As far as HA is concerned...I have heard a lot said about DRBD. The only catch is you need a fast network interface between the two devices to successfully make it happen (such as the crossover Cat5e or better cables connecting to a NIC on each server). A good article about this topic can be found here: Ajcody-Notes-HA-Linux-How-To - Zimbra :: Wiki
What else should I be backing up, besides /opt, by the way? Anything in /etc or /var ?
Since you are running a RAID1 for the other partition (/), mounting /opt would likely suffice since the mailstores and the majority of the config is underneath that directory.
Sorry about that. The plan is to eventually have ~500 users, from ~30 domains.
I believe you are going down the right track. However, you never said how many domains and mailboxes you plan to have on this machine. You also did not mention the size of the SAS drives and their performance levels (SATA, SCSI... 7.2k, 10k, 15k, etc...). By providing this information, it will help us better assist you and at least warn you of any pitfalls with your specific configuration (since your post is rather vague).
All disks are 15k SAS, 300Gb for the OS and 900Gb for the raid10.
Thanks, I missed that link, and it definitely does provide a lot. So does your post, really appreciated.