Cannot upload some files regardless of size or extension

[LunaticRV]

Hello, today I noticed that I can't upload some files especially PDF and XLSX files without a specific pattern.

When I try to upload another PDF or XLSX file, it upload successfuly but on certain files I couldn't upload them. When I checked logs, this is the error I am getting error pasted below.

I fixed permissions and also if its a pure permission issue, I wouldn't be able to upload other files with same extensions. What could be the issue?

Code: Select all

2019-05-27 13:28:13,361 INFO  [qtp509886383-681:https://MAIL_SERVER_IP_MASKED:443/service/soap/SaveDraftRequest] [name=..@...;mid=13;ip=176.xxx.xxx.145;ua=ZimbraWebClient - GC74 (Win)/8.6.0_GA_1242;] soap - SaveDraftRequest elapsed=46
2019-05-27 13:28:13,407 WARN  [qtp509886383-677:https://MAIL_SERVER_IP_MASKED:443/service/upload?fmt=extended,raw] [name=..@...;mid=13;ip=176.xxx.xxx.145;ua=Mozilla/5.0 (Windows NT 10.0;; Win64;; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36;] FileUploadServlet - Unable to store upload.  Deleting name=MyExcel.xlsx, StoreLocation=/opt/zimbra/data/tmp/upload/upload_1a69bf75_16af8a837bf__8000_00000032.tmp, size=32768bytes, isFormField=false, FieldName=upload
java.io.FileNotFoundException: /opt/zimbra/data/tmp/upload/upload_1a69bf75_16af8a837bf__8000_00000032.tmp (Permission denied)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
        at org.apache.commons.io.output.DeferredFileOutputStream.thresholdReached(DeferredFileOutputStream.java:165)
        at org.apache.commons.io.output.ThresholdingOutputStream.checkThreshold(ThresholdingOutputStream.java:221)
        at org.apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:127)
        at com.zimbra.common.util.ByteUtil.copy(ByteUtil.java:726)
        at com.zimbra.cs.service.FileUploadServlet.handlePlainUpload(FileUploadServlet.java:686)
        at com.zimbra.cs.service.FileUploadServlet.doPost(FileUploadServlet.java:530)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:206)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:169)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:127)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
2019-05-27 13:28:13,603 INFO  [qtp509886383-680:https://MAIL_SERVER_IP_MASKED:443/service/soap/SaveDraftRequest] [name=..@...;mid=13;ip=176.xxx.xxx.145;ua=ZimbraWebClient - GC74 (Win)/8.6.0_GA_1242;] soap - SaveDraftRequest elapsed=37
2019-05-27 13:28:27,156 INFO  [ImapSSLServer-0] [ip=MAIL_SERVER_IP_MASKED;


[DualBoot]

Hello,

maybe your server has been compromised.

Regards,

[ferra]

Same problem here. I cannot upload (attach) PDf files. jpg, txt are ok.

Zimbra version Release 8.6.0_GA_1153.RHEL7_64_20141215151110 RHEL7_64 FOSS edition, Patch 8.6.0_P4.

[zimbraargentina]

Hello,
The solution from this is:

AS ROOT

Execute the following command

chmod -R 750 /opt/zimbra/data/tmp/upload/

Isn't restart required.

Please let me know.

Regards,

Marcos M.
SDA

[phoenix]

the same here, after apply patch and runned fixed permissions son webapps because AJAX version isn't working.
zmfixperms isn't working with this.

Any solution?

How about your server has possibly been hacked/compromised? Search the forums for the relevant threads, read those and see if it applies to your server.

[ferra]

I found this message in /var/log/zimbra.log :
zmconfigd[10710]: Rewrite failed: [Errno 1] Operation not permitted: '/opt/zimbra/mailboxd/webapps/zimbra/WEB-INF/web.xml' ([Errno 1] Operation not permitted: '/opt/zimbra/mailboxd/webapps/zimbra/WEB-INF/web.xml')

Running fixpermissions does not solve the issue


$ ls -l /opt/zimbra/mailboxd/webapps/zimbra/WEB-INF/web.xml
-r-xr--r-- 1 zimbra zimbra 24862 Feb 18 09:01 /opt/zimbra/mailboxd/webapps/zimbra/WEB-INF/web.xml

[zimbraargentina]

Hello
The solution is execute as root

chmod -R 750 /opt/zimbra/data/tmp/upload/

no restart required

Let me know
Regards

Marcos M
SDA

[phoenix]

The solution is execute as root

Not necessarily as there's no reason for those file permissions to change. The question to ask yourself is why those permissions changed, did you check or do you know the answer to that? I'd suggest you follow my earlier suggestion and verify if your server is compromised/hacked and if so, then clean it and apply the relevant patches.

[ferra]

Yes, it worked. Thanks a lot

I also fount a process consuming all the CPU, zmswatch.sh in /opt/zimbra/log :

# cat zmswatch.sh
#!/bin/sh
AGENT_FILE='/opt/zimbra/log/zmswatch'
if ps cax | grep -v grep | grep -v "zmswatch.sh" | grep "zmswatch" > /dev/null; then
echo "running"
else
echo "nohup"
nohup /opt/zimbra/log/zmswatch > /dev/null 2>&1 &
fi
sed -i '/Ajax\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/XZimbra\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/login\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/ZimbraCore\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/Debug\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/ppwd=/d' /opt/zimbra/log/*_log.2019*


I just remove zmswatch.sh and zmswatch


I change the password following this wiki : https://wiki.zimbra.com/wiki/Investigat ... ng_Systems

I cannot find where this porocess starts

[JDunphy]

I cannot find where this porocess starts

You are playing whack-a-mole with the attacker. They have a remote command exploit (RCE) and a SSRF (server side request forgery)... Think of it like your zimbra server acting like a proxy to execute commands for that attacker. Check crontab, investigate your .bashrc, at jobs, etc, etc. Linux has so many ways to start something so another solution is to disable the script from being re-saved... if you haven't patched, they could be starting it with a POST request to your zimbra server.

Given the crazy solutions people are doing, I will add one to this list.

Code: Select all

% sudo chattr +i /var/spool/cron/zimbra
% /bin/rm -f /opt/zimbra/log/zmswatch
% touch /opt/zimbra/log/zmswatch
% sudo chattr +i /opt/zimbra/log/zmswatch


That will prevent even root from being able to write to the zimbra crontab and to stop zmswatch program to re-infect until they change the name. Add additional files as shown above to make this more robust (ie. zmswatch.sh) ... To remove the attribute, use the 'chattr -i' option. This does not close the security hole but it does slow the infection because the attacker script can not be written to your file system... until they change the name and begin again. Once they gain root, it is game over and one of the attacking signatures posted by someone else was a remote shell where they can execute arbitrary commands.

FYI, it is possible for an attacker with a 10G link to check the entire ipv4 address every 45 mins. You can't hide from this exploit. Patching is the only remedy short of using a VPN access server and blocking all incoming connections to that trusted VPN address space.