CVE-2019-9670 being actively exploited

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Media
Posts: 7
Joined: Wed May 24, 2017 1:49 pm

Re: CVE-2019-9670 being actively exploited

Postby Media » Thu May 30, 2019 7:45 am

anzigo wrote:Addtionally...

Check for existence of

Code: Select all

/opt/zimbra/lib/zmlogswatch
If it exists, it'll be a recently created binary file. Delete it.

However, there would likely be multiple instances already running. To find them, run (then kill all instances):

Code: Select all

top -p $(pgrep -d ',' zmlogswatch)


Looks like that /opt/zimbra/lib/zmlogswatch binary was actively adding itself back to crontab, so you should then cleanup, or regenerate your zimbra crontab.


My Zimbra is 8.6.0_GA_1242 with the last patch 14.
I've found /opt/zimbra/lib/zmlogswatch but it's not a recent one. His date is 2014-12-15. I don't think it's a part of the malware.


elby
Posts: 14
Joined: Tue May 28, 2019 7:37 am
Contact:

Re: CVE-2019-9670 being actively exploited

Postby elby » Thu May 30, 2019 7:45 am

Thanks to Drake for support :)

Zimbra 8.6.0 GA Network Editions , CentOS 6.6

Patch Zimbra :

wget https://files.zimbra.com/downloads/8.6. ... A_1242.tgz
tar xzf zcs-patch-8.6.0_GA_1242.tgz
cd /tar xzf zcs-patch-8.6.0_GA_1242


Delete global admin accounts

Change password

# Zimbra AJAX Webmail not loading

cd /opt/zimbra/mailboxd
find webapps -type d -exec chmod 0755 {} \;
find webapps -type f -exec chmod 0644 {} \;

# restart Zimbra;
su - zimbra
zmcontrol restart


change the files or delete them

files:

/opt/zimbra/log/.editorinfo --> /opt/zimbra/log/.editorinfo--

/opt/zimbra/log/zmswatch

/opt/zimbra/log/zmswatch.sh

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/Ajax.jsp

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/CryptCore.jsp

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp

/tmp/.cache/.ntp

/tmp/.cache/.kthrotlds

Or :

touch /opt/zimbra/log/zmswatch
chattr +i /opt/zimbra/log/zmswatch

Do the same with other files ...

Replace with the original installation files:

wget https://files.zimbra.com/downloads/8.6. ... 151258.tgz
tar xzf zcs-NETWORK-8.6.0_GA_1153.RHEL6_64.20141215151258.tgz
cd zcs-NETWORK-8.6.0_GA_1153.RHEL6_64.20141215151258/packages/
rpm2cpio zimbra-store-8.6.0_GA_1153.RHEL6_64-20141215151258.x86_64.rpm | cpio -idmv


find /opt/zimbra -name \*.jsp -exec grep --with-filename exec {} \;

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/403.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/sfdc_preauth.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/attachment_blocked.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/login.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbraAdmin/public/jsp/Debug.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/Alert.jsp


top

Kill PID 400%

fix error upload files :

chmod 755 /opt/zimbra/data/tmp/
chmod 755 /opt/zimbra/data/tmp/upload


regenerate crontab

crontab -l -u zimbra
crontab -e -u zimbra

replace with:
===================

# ZIMBRASTART -- DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRAEND
SHELL=/bin/bash
#
# Log pruning
#
30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1
#
# compress logs manually to avoid application pauses when
# handled through the log4j thread
#
50 2 * * * /opt/zimbra/libexec/zmcompresslogs > /dev/null 2>&1
#
# tmp dir cleaning
#
40 2 * * * /opt/zimbra/libexec/zmcleantmp
#
# Status logging
#
*/2 * * * * /opt/zimbra/libexec/zmstatuslog > /dev/null 2>&1
#*/10 * * * * /opt/zimbra/libexec/zmdisklog > /dev/null 2>&1
#
# SSL Certificate Expiration Checks
#
0 0 1 * * /opt/zimbra/libexec/zmcheckexpiredcerts -days 30 -email
#
# Backups
#
# BACKUP BEGIN
# BACKUP END
#
# crontab.store
#
# Log pruning
#
30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
30 2 * * * find /opt/zimbra/log/ -type f -name stacktrace.\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
#
# Report on any database inconsistencies
#
0 23 * * 7 /opt/zimbra/libexec/zmdbintegrityreport -m
#
# Monitor for multiple mysqld to prevent corruption
#
#*/5 * * * * /opt/zimbra/libexec/zmcheckduplicatemysqld -e > /dev/null 2>&1
#
# Check zimbraVersionCheckURL for new update.
# Only runs if this server matches zimbraVersionCheckServer
# Only executes on zimbraVersionCheckInterval. min 2h interval
#
18 */2 * * * /opt/zimbra/libexec/zmcheckversion -c >> /dev/null 2>&1
#
# Invoke "ComputeAggregateQuotaUsageRequest" periodically
#
15 2 * * * /opt/zimbra/libexec/zmcomputequotausage > /dev/null 2>&1
#
# Invoke "client_usage_report.py" periodically to process /opt/zimbra/log/access_log* files
#
55 1 * * * /opt/zimbra/libexec/client_usage_report.py > /dev/null 2>&1
#
# Run zmgsaupdate util to trickeSync galsync accounts
#
49 0 * * 7 /opt/zimbra/libexec/zmgsaupdate > /dev/null 2>&1
#
# crontab.logger
#
# process logs
#
00,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmlogprocess > /tmp/logprocess.out 2>&1
#
# Graph generation
#
#10 * * * * /opt/zimbra/libexec/zmgengraphs >> /tmp/gengraphs.out 2>&1
#
# Daily reports
#
30 23 * * * /opt/zimbra/libexec/zmdailyreport -m
#
# crontab.mta
#
#
# Queue logging
#
0,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmqueuelog
#
# Spam training
#
0 22 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1
#
# Spam training cleanup
#
45 23 * * * /opt/zimbra/bin/zmtrainsa --cleanup >> /opt/zimbra/log/spamtrain.log 2>&1
#
# Spam rule updates
#
45 0 * * * . /opt/zimbra/.bashrc; /opt/zimbra/libexec/zmsaupdate
#
# Dspam cleanup
#
0 1 * * * [ -d /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig ] && find /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig/ -type f -name \*sig -mtime +7 -exec rm {} \; > /dev/null 2>&1
8 4 * * * [ -f /opt/zimbra/data/dspam/system.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/system.log
8 8 * * * [ -f /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log
#
# Spam Bayes auto-expiry
#
20 23 * * * /opt/zimbra/libexec/sa-learn --dbpath /opt/zimbra/data/amavisd/.spamassassin --force-expire --sync > /dev/null 2>&1
#
# Clean up amavisd/tmp
#
15 5,20 * * * find /opt/zimbra/data/amavisd/tmp -maxdepth 1 -type d -name 'amavis-*' -mtime +1 -exec rm -rf {} \; > /dev/null 2>&1
#
# Clean up the quarantine dir
#
0 1 * * * find /opt/zimbra/data/amavisd/quarantine -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1

#ZIMBRAEND -- DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRASTART

===================

or use file /tmp/crontab.zimbra

#help
zmschedulebackup -h

#To Verify Backup Schedules:
zmschedulebackup -q

#To Configure Default Backup Schedules:
zmschedulebackup -D

#To Configure Customized Backup Schedules:
zmschedulebackup -R f "0 1 * * 6" --mail-report i "0 1 * * 0-5" --mail-report d 14d "0 0 * * *" --mail-report

crontab -l |grep -i backup

Lock up crontab

chattr +i /var/spool/cron/zimbra


reboot
Last edited by elby on Thu May 30, 2019 11:52 am, edited 4 times in total.
patsybrian
Posts: 1
Joined: Thu May 30, 2019 8:01 am

Re: CVE-2019-9670 being actively exploited

Postby patsybrian » Thu May 30, 2019 8:08 am

Hi all!
I have followed all steps in Lorenzo's blog (https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/).
But still, the l.sh and wgets keep appearing a second after killing the processes.
Any ideas what I can do?

Thanks so much!
Drake
Posts: 8
Joined: Tue May 28, 2019 8:52 am

Re: CVE-2019-9670 being actively exploited

Postby Drake » Thu May 30, 2019 8:36 am

Fast food, fast dates, fast sex, fast Internet, fast info, fast solutions .......
Take some time to read the topic and the posts before yours pls....
You may also try to replicate the steps described and then ask constructive questions or even propose some solutions.

Good Luck
fcourtaud
Posts: 2
Joined: Thu May 30, 2019 10:29 am

Re: CVE-2019-9670 being actively exploited

Postby fcourtaud » Thu May 30, 2019 10:34 am

Hi everyone,

Below are the steps I followed

First delete the following files

rm /opt/zimbra.log/zmswatch
rm /opt/zimbra.log/zmswatch.sh
/opt/zimbra/lib/zmlogswatch

Kill all the processes you'll get with :

top -p $(pgrep -d ',' zmlogswatch)
top -p $(pgrep -d ',' zmswatch)

zmlogswatch seems to be the one updating the crontab

Clean all of the .jsp, .java and .class you'll find with

grep -R '(request\.getParameter.*' /opt/zimbra/mailboxd
grep -R '(request\.getParameter.*' /opt/zimbra/jetty

cleanup zimbra's crontab

Reset permissions
chmod 1777 /opt/zimbra/data/tmp
chmod 755 /opt/zimbra/data/tmp/upload
chmod 755 /opt/zimbra/data/tmp/nginx
chmod 755 /opt/zimbra/data/tmp/nginx/client
chmod 755 /opt/zimbra/data/tmp/nginx/proxy
chmod 755 /opt/zimbra/data/tmp/nginx/fastcgi

cd /opt/zimbra/mailboxd
find webapps -type d -exec chmod 0755 {} \;
find webapps -type f -exec chmod 0644 {} \;
/opt/zimbra/libexec/zmfixperms -verbose

Update ssh keys as zimra user
zmsshkeygen
zmupdateauthkeys

reboot

Hope this helps
zimbraxtc
Posts: 7
Joined: Mon May 27, 2019 6:13 pm

Re: CVE-2019-9670 being actively exploited

Postby zimbraxtc » Thu May 30, 2019 1:34 pm

Hi all!

I have a old 5.0 installation running on an old hp-server, dont ask me why. Are the 5.0 affected by the virus?

Anyone knows?

Thanks!
cmgonzalez
Posts: 2
Joined: Thu May 30, 2019 2:35 pm

Re: CVE-2019-9670 being actively exploited

Postby cmgonzalez » Thu May 30, 2019 2:38 pm

Caution, this MF, is cloacking the files now using same dates as regular ones....He may be reading this :evil: :evil:
halfgaar
Advanced member
Advanced member
Posts: 82
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Release 8.8.12.GA.3794.UBUNTU16.64

Re: CVE-2019-9670 being actively exploited

Postby halfgaar » Thu May 30, 2019 7:47 pm

You can always use 'stat' to look at the ctime: you can't change that.
jaca_sv
Posts: 8
Joined: Thu Nov 29, 2018 8:17 am

Re: CVE-2019-9670 being actively exploited

Postby jaca_sv » Thu May 30, 2019 8:34 pm

fcourtaud wrote:Clean all of the .jsp, .java and .class you'll find with

grep -R '(request\.getParameter.*' /opt/zimbra/mailboxd
grep -R '(request\.getParameter.*' /opt/zimbra/jetty


Hello fcourtaud,
By "clean all of the .jsp, .java and .class" do you mean delete the suspicious lines or completely delete the files?
All the files found with "request\.getParameter" are not supposed to be in the server?

I already restored the zimbra crontab and locked it with chattr +i (Also removed and touched .kthrotlds and .editorinfo and lock them)
I also blocked python-requests user agent in my nginx and obviously patched my zimbra 8.7.11 to the latest patch.

My next steps will be to remove the zmswatch.sh and delete the logs the script create

-rw-r--r-- 1 zimbra zimbra 100 May 26 03:45 zmswatch.out-20190526.gz
-rw-r--r-- 1 zimbra zimbra 100 May 27 03:38 zmswatch.out-20190527.gz
-rw-r--r-- 1 zimbra zimbra 304 May 28 03:28 zmswatch.out-20190528.gz
-rw-r--r-- 1 zimbra zimbra 236 May 29 03:06 zmswatch.out-20190529.gz
-rw-r--r-- 1 zimbra zimbra 3.0K May 30 03:21 zmswatch.out-20190530
-rwxr-x--- 1 zimbra zimbra 225 May 30 14:11 zmswatch.sh

Thanks everyone for the help!
SPAMSAM
Posts: 2
Joined: Thu May 30, 2019 7:06 pm

Re: CVE-2019-9670 being actively exploited

Postby SPAMSAM » Fri May 31, 2019 2:27 am

Having this exact issue.

I noticed that it would blank out the webmail page after someone logged in. Steps above helped me fix it, though the admin page still loads a blank page. not sure if related

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 8 guests