CVE-2019-9670 being actively exploited (Hacked Server)

[calandacomp]

I agree but I think there is a jsp file that they are accessing but it still looks like an attack though soap. I also have another server that I think did not get attacked but zimbra services have locked up twice due to the amount of soap attacks / requests.
Can't I just disable soap or have it goto page not foind, it's not needed is it? They are just single servers

yes, you are right. in my case they where also jsp-files affected which i have to clean-up and/or delete. it was around 5 files but i don't remember the name. with my instructions above you can identify which files are affected. I'm sure you are same close to solution as i was last tuesday.... good luck and if you want, you can contact me by my private mail. if i can, i'd like to help in that case.

[fladnar]

Is this vulnerabilty present in 8.7-8.8 version?. I'm in 8.6 and thinking of an upgrade.

[calandacomp]

[*]

Is this vulnerabilty present in 8.7-8.8 version?. I'm in 8.6 and thinking of an upgrade.

have a look at my steps above. update is REQUIRED to solve problem...

[fladnar]

[*]

Is this vulnerabilty present in 8.7-8.8 version?. I'm in 8.6 and thinking of an upgrade.

have a look at my steps above. update is REQUIRED to solve problem...

is it not enough patching 8.6?

[misosoup]

Hi! My Zimbra is infected as well - thanks a lot to all the helpful people who shared their solutions so far.

Everything seems to be mostly clean now and as it seems nothing is mining any cryptocurrency anymore, _but_ the line "*/60 * * * * /opt/zimbra/lib/zmcheckexpiredcerts" keeps getting added back into the "zimbra" user's crontab. It doesn't do anything because that file isn't there anymore, but it still feels bad knowing that something is still running. I haven't gotten around to patching it yet (will probably do that tonight), http and https ports are still closed to the outside world.

Any ideas?

[halfgaar]

I've mentioned it before as a method against attacks on HTTP traffic, but to reiterate:

If you have an extra server (small VM will do) and if your user base allows it, you can block the HTTP(S) traffic, and only allow the web mail to be used through an authenticated HTTP proxy. I use this kind of pre-login system all the time, and it works really well.

[calandacomp]

[*]

Is this vulnerabilty present in 8.7-8.8 version?. I'm in 8.6 and thinking of an upgrade.

have a look at my steps above. update is REQUIRED to solve problem...

is it not enough patching 8.6?

i'm not sure. i've updated all my 3 affected systems to the newest version. and my step's to clean up the systems was based on infected 8.7.* systems.

best regards

[irv]

Hi There ;)

So, they got me too. First I didn't really notice much, deleted admin accounts but then new admin pop'us and I could not attach new files to emails. Your work is amazing guys.

I'm still monitoring server, and probably will install fresh one system and do system diff to find all others files or so, but so far - looks like system is clean. I can't afford quick clean install as most of files where hand-modyfied by me some time ago ;)

I don't know if you notice that, but zmcheckexpiredcerts is doing cleanup on logs every couple of minutes.
Therefore you may not find anything in logs.
In my case, I do have my own proxy that is working as FW for everything and all apache logs are duplicated - even if system (zimbra) is asking for 127.0.0.1, it goes thru that system too.

root 21438 0.0 0.0 0 0 ? S 11:51 0:00 [kworker/u4:2]
zimbra 22550 0.0 0.0 12720 1216 ? S 12:25 0:00 /opt/zimbra/lib/zmcheckexpiredcerts
zimbra 22600 0.0 0.0 4452 652 ? S 12:25 0:00 sh -c sed -i '/Ajax\.jsp/d' /opt/zimbra/log/*_log.20* > /dev/null 2>&1
zimbra 22601 17.0 0.0 10988 876 ? R 12:25 0:00 sed -i /Ajax\.jsp/d /opt/zimbra/log/access_log.2019-05-07 /opt/zimbra/log/access_log.2019-05-08 /opt/zimbra/log/access_log.2
root 22602 0.0 0.0 18452 1288 pts/1 R+ 12:25 0:00 ps aux

I found that somebody with success (FW proxy rock ;:)) upload a zimlet /opt/zimbra/zimlets-deployed/com_zimbra_clientuploader . I didn't find any wrong code with it, but still i'm not unsure about that. So watch out on that.

If you can't upload attachments, and you will do chmod 755 /opt/zimbra/data/tmp/upload/ run watch stat /opt/zimbra/data/tmp/upload/ command. In my case every 1 minute upload where changes to 555. Looks like zmcheckexpiredcerts where replacing permission on that directory. Maybe to prevent others to upload and hack server?

My attacker IP (who executed shell) was from

I got two entry in crontab -
*/30 * * * * /opt/zimbra/log/zmswatcher
and that zmcheckexpiredcerts

Every time I removed crontab it got replaced. Chattr +i won't help you much here, as you can see in logs there are entry like:

Jun 12 06:30:02 xxx crontab[15327]: (zimbra) REPLACE (zimbra)
Jun 12 06:30:08 xxx crontab[15837]: (zimbra) LIST (zimbra)
Jun 12 06:30:08 xxx crontab[15836]: (zimbra) REPLACE (zimbra)
Jun 12 06:30:38 xxx crontab[16369]: (zimbra) LIST (zimbra)
Jun 12 06:30:38 xxx crontab[16368]: (zimbra) REPLACE (zimbra)
Jun 12 06:30:47 xxx crontab[16403]: (zimbra) LIST (zimbra)

so you need kill all zimbra process couple of times, chattr +i zmcheckexpiredcerts to stop that loop and then be able to start cleanup.

You will get probably couple of .jsp files therefore search for
grep -R '(request.getParameter.' /opt/zimbra/mailboxd
grep -R '(request.getParameter.' /opt/zimbra/jetty
grep -R 'zmswatcher' /opt/zimbra (except store dir)

if you will get HASH from that files, try to search for them

find /opt/zimbra -iname \*.jsp | while read p; do if [ `cat $p | grep -i 9BLs9fCk0neAfizoYygwiwhNsjKHDYyMXbkmGiWsyzM| grep -c ""` -gt 0 ]; then echo $p;fi; done
find /opt/zimbra -iname \*.jsp | while read p; do if [ `cat $p | grep -i Hok8gxZFafGORRLCiowY_vpqNappusQV8agmQkI7wKk| grep -c ""` -gt 0 ]; then echo $p;fi; done
find /opt/zimbra -iname \*.jsp | while read p; do if [ `cat $p | grep -i request.getParameter | grep ppwd | grep -c ""` -gt 0 ]; then echo $p;fi; done

and so on. You can go with -type f and exclude -iname.

Most important thing, some code is added as additional stuff to zimbra jsp files. So before you will delete every jsp, class files - check them. You will see, at end of files wrong code. Remove it, it will help you resume operation

Check your store directory. You should have there only emails, not other files probably.

Other stuff is well described here.
There is quick hack to remove couple of lines of code.

https://blog.tint0.com/2019/03/a-saga-of-code-executions-on-zimbra.html#href1

"That seems to only add an additional layer of authentication and doesn't really block /service/proxy. For the workaround I'd block ProxyServlet from public access altogether. To do that, change the allowed.ports parameter to only 7070 in the servlet tag named ProxyServlet in {zimbra_home}/mailboxd/etc/service.web.xml.in and restart the instance. You could also disable AutoDiscoverServlet there for the XXE."


My infected files:
/opt/zimbra/cbpolicyd/bin/cbstat
/opt/zimbra/lib/zmcheckexpiredcerts
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/js/zimbra/csfe/XZimbra.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/Ajax.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/CryptCore.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/login.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbraAdmin/public/jsp/Debug.jsp
opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbraAdmin/public/jsp/Alert.jsp

After cleanup I found that somebody wanted to open shell, by executing Debug.jsp (zimbra main file, however with extra " " ;) code from attacker). He failed.

So far, system is working well. I'm working now to see how I can migrate and upgrade that system

and on my proxy/firewalls system I run a quick monitoring too, to monitor any HTTP activity that goes to zimbra and block IP address on firewall if there is PUT request to JSP files etc.
So far, 24h hours, I got 2 new IP's, 1x new IP who wanted to execute proxy/127.0.0.1 and so on.

Probably you can write that script to monitor logs on your zimbra too, with iptables -I INPUT -s ip/32 -j DROP.

if you use tail -f -n0 logfile and loop to read every line and parse it, you can get a bit more time to work how to secure systems etc.

Tripwire helps too.

Cherrs.
Kris

[edgar3]

Hi!!, Thanks in advance!
I have a question... ¿Do you know if a WAF block's this attack-vulnerability?
SonicWALL, Watchguard, etc etc

[irv]

I have a question... ¿Do you know if a WAF block's this attack-vulnerability?
SonicWALL, Watchguard, etc etc

if you configure filtering in right way then yes. But not out of box.
however, this won't make your system secure in anyway. you still need to look about patching or heavy-custom build.