Postby irv » Thu Jun 13, 2019 11:59 am
Hi There ;)
So, they got me too. First I didn't really notice much, deleted admin accounts but then new admin pop'us and I could not attach new files to emails. Your work is amazing guys.
I'm still monitoring server, and probably will install fresh one system and do system diff to find all others files or so, but so far - looks like system is clean. I can't afford quick clean install as most of files where hand-modyfied by me some time ago ;)
I don't know if you notice that, but zmcheckexpiredcerts is doing cleanup on logs every couple of minutes.
Therefore you may not find anything in logs.
In my case, I do have my own proxy that is working as FW for everything and all apache logs are duplicated - even if system (zimbra) is asking for 127.0.0.1, it goes thru that system too.
root 21438 0.0 0.0 0 0 ? S 11:51 0:00 [kworker/u4:2]
zimbra 22550 0.0 0.0 12720 1216 ? S 12:25 0:00 /opt/zimbra/lib/zmcheckexpiredcerts
zimbra 22600 0.0 0.0 4452 652 ? S 12:25 0:00 sh -c sed -i '/Ajax\.jsp/d' /opt/zimbra/log/*_log.20* > /dev/null 2>&1
zimbra 22601 17.0 0.0 10988 876 ? R 12:25 0:00 sed -i /Ajax\.jsp/d /opt/zimbra/log/access_log.2019-05-07 /opt/zimbra/log/access_log.2019-05-08 /opt/zimbra/log/access_log.2
root 22602 0.0 0.0 18452 1288 pts/1 R+ 12:25 0:00 ps aux
I found that somebody with success (FW proxy rock ;:)) upload a zimlet /opt/zimbra/zimlets-deployed/com_zimbra_clientuploader . I didn't find any wrong code with it, but still i'm not unsure about that. So watch out on that.
If you can't upload attachments, and you will do chmod 755 /opt/zimbra/data/tmp/upload/ run watch stat /opt/zimbra/data/tmp/upload/ command. In my case every 1 minute upload where changes to 555. Looks like zmcheckexpiredcerts where replacing permission on that directory. Maybe to prevent others to upload and hack server?
My attacker IP (who executed shell) was from
I got two entry in crontab -
*/30 * * * * /opt/zimbra/log/zmswatcher
and that zmcheckexpiredcerts
Every time I removed crontab it got replaced. Chattr +i won't help you much here, as you can see in logs there are entry like:
Jun 12 06:30:02 xxx crontab[15327]: (zimbra) REPLACE (zimbra)
Jun 12 06:30:08 xxx crontab[15837]: (zimbra) LIST (zimbra)
Jun 12 06:30:08 xxx crontab[15836]: (zimbra) REPLACE (zimbra)
Jun 12 06:30:38 xxx crontab[16369]: (zimbra) LIST (zimbra)
Jun 12 06:30:38 xxx crontab[16368]: (zimbra) REPLACE (zimbra)
Jun 12 06:30:47 xxx crontab[16403]: (zimbra) LIST (zimbra)
so you need kill all zimbra process couple of times, chattr +i zmcheckexpiredcerts to stop that loop and then be able to start cleanup.
You will get probably couple of .jsp files therefore search for
grep -R '(request.getParameter.' /opt/zimbra/mailboxd
grep -R '(request.getParameter.' /opt/zimbra/jetty
grep -R 'zmswatcher' /opt/zimbra (except store dir)
if you will get HASH from that files, try to search for them
find /opt/zimbra -iname \*.jsp | while read p; do if [ `cat $p | grep -i 9BLs9fCk0neAfizoYygwiwhNsjKHDYyMXbkmGiWsyzM| grep -c ""` -gt 0 ]; then echo $p;fi; done
find /opt/zimbra -iname \*.jsp | while read p; do if [ `cat $p | grep -i Hok8gxZFafGORRLCiowY_vpqNappusQV8agmQkI7wKk| grep -c ""` -gt 0 ]; then echo $p;fi; done
find /opt/zimbra -iname \*.jsp | while read p; do if [ `cat $p | grep -i request.getParameter | grep ppwd | grep -c ""` -gt 0 ]; then echo $p;fi; done
and so on. You can go with -type f and exclude -iname.
Most important thing, some code is added as additional stuff to zimbra jsp files. So before you will delete every jsp, class files - check them. You will see, at end of files wrong code. Remove it, it will help you resume operation
Check your store directory. You should have there only emails, not other files probably.
Other stuff is well described here.
There is quick hack to remove couple of lines of code.
https://blog.tint0.com/2019/03/a-saga-of-code-executions-on-zimbra.html#href1
"That seems to only add an additional layer of authentication and doesn't really block /service/proxy. For the workaround I'd block ProxyServlet from public access altogether. To do that, change the allowed.ports parameter to only 7070 in the servlet tag named ProxyServlet in {zimbra_home}/mailboxd/etc/service.web.xml.in and restart the instance. You could also disable AutoDiscoverServlet there for the XXE."
My infected files:
/opt/zimbra/cbpolicyd/bin/cbstat
/opt/zimbra/lib/zmcheckexpiredcerts
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/js/zimbra/csfe/XZimbra.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/Ajax.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/CryptCore.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/login.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbraAdmin/public/jsp/Debug.jsp
opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbraAdmin/public/jsp/Alert.jsp
After cleanup I found that somebody wanted to open shell, by executing Debug.jsp (zimbra main file, however with extra " " ;) code from attacker). He failed.
So far, system is working well. I'm working now to see how I can migrate and upgrade that system
and on my proxy/firewalls system I run a quick monitoring too, to monitor any HTTP activity that goes to zimbra and block IP address on firewall if there is PUT request to JSP files etc.
So far, 24h hours, I got 2 new IP's, 1x new IP who wanted to execute proxy/127.0.0.1 and so on.
Probably you can write that script to monitor logs on your zimbra too, with iptables -I INPUT -s ip/32 -j DROP.
if you use tail -f -n0 logfile and loop to read every line and parse it, you can get a bit more time to work how to secure systems etc.
Tripwire helps too.
Cherrs.
Kris