ZCS/Dovecot/Cyrus performance comparison

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
jernejp
Posts: 4
Joined: Fri Sep 12, 2014 10:14 pm

ZCS/Dovecot/Cyrus performance comparison

Postby jernejp » Tue Aug 08, 2006 4:25 am

Hello,
These are the results of my testing ZCS version 3.1 OS Edition on CentOS 4.3 (updated) on VMWare ESX server having 2 Xeon 2.8GHz, 3.5GB RAM, 100Mb network with vmware tools installed.
I have tested ZCS vs. Dovecot vs. Cyrus by filling 10 users mailboxes with 2000 mails, where 1700 of these mails were small mails from qmail mailing list and the other 300 were jokes mails (usually from several kB to several MB). There were no special tuning of the system for any of the programs, because all of them were installed via RPM and I used Posftix as MTA for dovecot and Cyrus.
Testset for IMAP testing was something like this: Courier IMAP test with 10 users. I created scripts in perl using Expect (scripts available if interested).

Testset for POP3 testing were some perl scripts using Net::Pop3 module and with simple commands.

Analysis was made with some scripts utilizing gnuplot and if you want detailed results (or scripts), PM me.
The test was made from P4 machine quite close to the server and no other virtual machines were running on VMWare ESX server at the time of the tests.
It turned out that Zimbra is quite slow compared to other two systems. We were discussing what might be the reason, but I think that only developers could tell us. At first glance we accused java for not being optimised for VMware, but there are no such explanation to be found on web. We also recorded the I/O state of the machine while in test, but the results are not analysed yet.
So here are the results:

IMAP tests: IMAP tests

 http://www2.arnes.si/~krklubsls13/imap-select1.png />
 http://www2.arnes.si/~krklubsls13/imap-select2.png />
 http://www2.arnes.si/~krklubsls13/imap-delete1.png />
to be continued in next post...


jernejp
Posts: 4
Joined: Fri Sep 12, 2014 10:14 pm

ZCS/Dovecot/Cyrus performance comparison

Postby jernejp » Tue Aug 08, 2006 4:27 am

POP3 tests: POP3 tests
 http://www2.arnes.si/~krklubsls13/pop3-list1.png />
 http://www2.arnes.si/~krklubsls13/pop3-headers1.png />
 http://www2.arnes.si/~krklubsls13/pop3-retrieve1.png />
 http://www2.arnes.si/~krklubsls13/pop3-delete1.png />
Sorry for spamming but this is the only reasonable way to show the results and performance comparison with real data.
Klug
Elite member
Elite member
Posts: 2358
Joined: Mon Dec 16, 2013 11:35 am
Contact:

ZCS/Dovecot/Cyrus performance comparison

Postby Klug » Tue Aug 08, 2006 12:44 pm

Hello, thanks for the testing.
I'm not sure Zimbra was designed to be a competitor of Dovecot, Cyrus or Courier-MTA as IMAP4/POP3 server.

Just like if you were comparing Zimbra's webmail with Squirrelmail on email opening, moving, searching.
My feeling is this kind of test might be very interesting for the ISP using Zimbra. But in these case, you're way beyong 10 users concurency, aren't you ?
redhat
Posts: 35
Joined: Fri Sep 12, 2014 10:14 pm

ZCS/Dovecot/Cyrus performance comparison

Postby redhat » Wed Aug 09, 2006 12:46 pm

I don't understand the comments from Klug. Can you elaborate, or can someone else explain what he means?
For example, Zimbra is designed for "state-of-the-art messaging and collaboration solutions" which includes IMAP and POP so it IS a "competitor" to the other IMAP/POP servers you list. I'd be surprised if the Zimbra developers were satisfied with having the slowest IMAP server on the market. I haven't seen anything that recommends an alternate IMAP server for large installations. (Is it even possible to use something else?)
And the marketing literature specifically says "The Zimbra Collaboration Suite (ZCS) was designed to be deployed by service providers, such as carriers/operators, other broadband providers, and Internet service providers (ISPs)." so someone at Zimbra must think it's suitable for large installations.
Also, what about SquirrelMail vs. Zimbra? Are you saying that Zimbra is going to be slower, less reliable, or what?
I ask because I'm evaluating Zimbra for a large company right now and I'd really love some feedback. Much of what I see in these forums is coming from people with very small, or just test, installations.
phoenix
Ambassador
Ambassador
Posts: 26417
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

ZCS/Dovecot/Cyrus performance comparison

Postby phoenix » Wed Aug 09, 2006 1:01 pm

The problem with your question (and the tests, IMO) is that you're not comparing like with like. Zimbra is designed to be more than squirrelmail/dovecot/cyrus - it's a complete collaboration suite it's not a direct competitor for those products as they don't provide the same functionality as Zimbra. Those test would only be valid if the other products had the same feature set as zimbra.
You choose a piece of software for the features it has and the problems it will solve in your environment.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
redhat
Posts: 35
Joined: Fri Sep 12, 2014 10:14 pm

ZCS/Dovecot/Cyrus performance comparison

Postby redhat » Wed Aug 09, 2006 2:01 pm

Understood, as far as Zimbra vs. SquirrelMail is concerned. The Zimbra user interface has more work to do so it's reasonable to assume it will be somewhat slower. The comments from Klug (not authoritative I assume) seemed to contradict the Zimbra marketing materials that say ZCS is suitable for large installations and ISPs.
But we're really talking about a test of a very specific feature that's easy to isolate. You/we should be able to compare performance on requests sent directly to the IMAP/POP server from an external client. If Zimbra is significantly slower I'd be curious as to why and whether that indicates a real problem or simply an area for future improvements.
I don't know if the test was conducted accurately or not. In my limited testing (not very scientific) Zimbra worked fine with every IMAP client. This test was just something I came across while trying to find out more about the IMAP/POP server being used. I didn't see anything that identified a specific open source server this was based on, so I was digging around.
If IMAP/POP does turn out to be a bottleneck is it possible to substitute a different server to access the message store? (It looks like messages are not stored in a standard format.)
phoenix
Ambassador
Ambassador
Posts: 26417
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

ZCS/Dovecot/Cyrus performance comparison

Postby phoenix » Wed Aug 09, 2006 2:29 pm

Wel, I can't offer any comment on the code that surrounds the imap/pop functions so I don't know how well it performs. My other comment about the original test is that it's not mentioned how much memory was given to the VM that Zimbra was run in, Zimbra should really be tested on bare metal not a VM and the more memory it has the better it will perform.
Zimbra is ready for large deployments, have you seen the press release section, specifically this article? It is in use at many ISPs and Hosted Service providers - they don't take decisions to install a critical product like Zimbra lightly.
I don't believe that imap or pop are bottlenecks in Zimbra and the answer to your question is no, it's not possible to substitute another server for the Zimbra ones.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
phoenix
Ambassador
Ambassador
Posts: 26417
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

ZCS/Dovecot/Cyrus performance comparison

Postby phoenix » Wed Aug 09, 2006 2:38 pm

I forgot to mention the benchmark information in the blogs here - you may find that more useful for details of performance in a large deployment.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
dkarp
Elite member
Elite member
Posts: 1410
Joined: Fri Sep 12, 2014 9:52 pm

ZCS/Dovecot/Cyrus performance comparison

Postby dkarp » Thu Aug 10, 2006 12:50 am

Well, yes and no. There really is a difference between writing an IMAP server and writing a server that exports an IMAP interface to its content. Your data storage, caching, and various optimizations are tuned to give you the most bang for your buck on all the interfaces you support; what's a no-brainer speedup for a simple IMAP server may not be a great idea for a server like Zimbra that has to field queries like "find all contacts and/or calendar appointments with tag FOO in any folder" on a regular basis.
Which is not to say that we don't care about IMAP and POP performance. For instance, in 3.2 we've heavily optimized the features tested by your select1 and select2 IMAP tests (are they the same?). Multi-message STOREs/EXPUNGEs are also significantly faster. And, of course, our search results will continue to be very good, as that's one of the things that we're optimized around.
The places where our numbers look worst at present are your fetch tests, and this is because the other servers are pre-calculating and caching message structure while we're doing it on the fly. We think we can fix this without too much of a tradeoff, so adding a message structure cache that works with our architecture and transaction system is high on the list for the next major release...

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 7 guests