Page 1 of 1

Suggestions and Advice Errata

Posted: Wed Apr 09, 2014 5:30 pm
by john.holder
Hello there friends! Long time no see, eh?
I saw a few posts in the forum here about concern about this bug going around, and I wanted to share some information that you may or may not find helpful. I also have fielded a few questions about this, so I wanted to talk about it.
I'm no longer a Zimbra employee, but have been a proud supporter- so please don't take any of this as official information from the Zimbra team or moderators. It's also not gonna be popular, but we as admins have to be pragmatic.
We've all been around the block a few times, so I would like to inject sanity into the world that seems to have gone crazy. Not trying to sound elitist, but let's discuss this instead of going to Facebook for all our security advice.
Firstly, don't freak out. It is a big deal, but demanding that the Zimbra team answer questions X Y Z and getting angry at them for not giving you the answers you'd like, isn't helpful. The world won't end, and you've got time.
As has been said, this exploit only involves 64k of data. That's less than a single mail message. That amount is even less than a single java object that controls Zimbra users. So even if you could break into the server, and try to dump the object, you wouldn't get it all because it's more than 64k. If you're worried about data being stolen, then you should probably not be. I realize there are other concerns, but this really is probably the most important point. This means that for an attacker to succeed, they have to do more stuff. So what ever reports you may hear about this, know that it really is limited to 64k and the memory region that contains the Certificate.
Secondly, have a plan for this kind of thing. This won't be the last time, and isn't the first. The nature of computer code gives it the intrinsic characteristic that it is flawed. It just is, and someone will find it. When Zimbra got purchased by Yahoo, I learned something really great there: Have a BCP (Backup Contingency Plan). Have a way to immediately secure your stuff.
One way is to proxy your http requests through something like nginx. Edge servers area great way of proxying and inspecting traffic. So if you want to be able to watch for an attacker, then use an edge server. If you have sensitive data, you probably should try to move to an edge server anyway. If you don't know how, this is a great chance to learn.
There is a rumor going around (from the guys who reported this bug) that because it's SSL, you can't see it. So firstly, this is flat wrong. Secondly, it's irresponsible to tell people they can't see it on a server. That statement assumes that you don't already log those requests or that the https responder isn't capable of it.
AND if you aren't logging those handshake requests, why not? Because it's not default? Because it's difficult? What ever the reason, it's because you didn't turn it on....not because you can't log them. Lots of admins are scared of the idea of using Nginx as an edge proxy. I'm much more scared of being attacked than I am learning something. Good chance to learn.
Turn it on. I don't know if Jetty can't log the handshake, but I know that Nginx can. You can use something like Mongo to proxy the requests, and log the request in mongo. Mongo isn't effected.
This is why edge/proxy servers are important. If zimbra sits in the cloud being accessed, and it gets attacked, it's already too late(regardless of vector). Let them attack something else, and work their way to the actual box. This isn't an end-all to protection, but my simple point is: Don't DMZ your Zimbra (or any other) server. That's an easy solution and completely the wrong one. The Zimbra team can keep the product protected, but you have to keep your server protected. Don't blame them if you have port 22 open, and someone gets your key through this attack.
Finally, I strongly believe in VPN and Bastion servers. For those unfamiliar, a Bastion server sits outside the wall of protection, and allows you immediate access to firefight a problem. For example, you could have a plan that says: I'll lock out the firewall on all ports except smtp and smtps to allow mail, but lock out 80 and 443. Then I'll use the bastion server to upgrade my Zimbra server.
Of course, this doesn't address the real issue. All I've said thus far, is try to watch for it. But if you see it, it's already too late; which is why people are worried. But let's think this through:
There are only 2 legitimate use cases here.

1) Your server is exploited to allow gathering of the cyphers so that it can be spoofed. Then someone sets up some kind of man in the middle attack, intercepts the post, and grabs the password because they can decrypt the data.

2) Your server is exploited within openSSH to steal your private key through a man in the middle attack. Again, assuming they STOLE your ssl cert, and that is the same cert is also your openSSH cert (almost no one does that); the attack would fail. They may get the key, but your SSH client will tell you the fingerprint is you'd know about the man-in-the-middle.
(Disclaimer: Some researches report openSSH isn't vulnerable. This is utterly false. Even the Register is reporting this junk. Only 1.0.1 and 1.0.2-beta releases of OpenSSL are affected including 1.0.1f and 1.0.2-beta1.)
Again, no one can access the Zimbra server through this. This is only used to gather a method to then get access(keys passwords, etc).
We ran some tests to see what I could get back if I were trying to use this exploit.
We had a 100% success rate altering, manipulating, and otherwise gaining access to the SSL cert in memory. But there's an important caveat here. We were only able to do it, if we did it in a window where the minute on the server didn't change. I don't know enough about why, but that's what happened. If the minute changed between the initial bounds checking that it should be doing, and the next request, all versions of openSSL hung up. This makes the window of attacks mall. It has to be initiated and finished within 60 seconds.
Still don't know what that is, or even what it means.
Beyond that, we couldn't use this exploit to do anything on a Zimbra server. We tried. The maximum I got was a hex encoded partial copy of the private key...but the private key with a passphrase put it way out the bounds of the exploit. We tried everything from man in the middle to injecting XSS within Zimlets. Nothin.
That doesn't mean someone clever won't find it, but the point is: There just isn't a way to use this attack to get information easily, so as long as you already are taking best practice steps.
We live in a media driven world. Security researches are right to be alarmed, but not nearly to the extent that people are worried. I even know one family who have decided to discontinue their wireless and use hardlines.

This hasn't left proof of concept yet. Help your friends understand that the media is goin nuts about a potential problem for which even if it is exploited, a simple revocation of the cert updates everyone's browsers.
Java is almost always at fault. Not this time :)
Farewell friends,


Suggestions and Advice Errata

Posted: Wed Apr 09, 2014 5:57 pm
by john.holder
By the way, the widespread reporting of this bug and the mass revocation of certs are gonna do far more damage than the bug itself. Here's why: For a period of about 2 days, everyone will encounter an SSL cert warning once. It's because people are revoking their certs and reissuing. This takes time for things like CDNs and stuff.
In the mean time users are getting warnings from their browsers now. This is going to lul them into seeing this warning and bypassing it; when the time comes that that warning is legit, they will have already learned the behavior that it's ok to click "Proceed anyway". Prime example: The heartbeat website itself is throwing the very warning, during a time when many admins are going there to learn about the bug.