Page 1 of 2

After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Mon Sep 26, 2016 2:15 pm
by cerri
After the upgrade, the WebMail is unreliable: after few hours it stops working without any reasons.
The Console is still accessible and other services too, but the webmail is hanged until the proxy is restarted.

In the logs we can see:

Code: Select all

2016-09-26 16:08:30.741:WARN:oejs.HttpChannel:qtp1684106402-1767: https://HOSTNAME/service/home/~/?auth=co&loc=it&id=96359&part=2
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:51)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:175)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:808)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:419)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:313)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:141)
        at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:752)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
        at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:511)
        at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:653)
        at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:702)
        at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor$GzipBufferCB.process(GzipHttpOutputInterceptor.java:364)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
        at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:143)
        at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:112)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:177)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:163)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:413)
        at com.zimbra.common.util.ByteUtil.copy(ByteUtil.java:726)
        at com.zimbra.common.util.ByteUtil.copy(ByteUtil.java:698)
        at com.zimbra.cs.service.formatter.NativeFormatter.sendbackBinaryData(NativeFormatter.java:630)
        at com.zimbra.cs.service.formatter.NativeFormatter.sendbackBinaryData(NativeFormatter.java:576)
        at com.zimbra.cs.service.formatter.NativeFormatter.sendbackOriginalDoc(NativeFormatter.java:450)
        at com.zimbra.cs.service.formatter.NativeFormatter.handleMessagePart(NativeFormatter.java:275)
        at com.zimbra.cs.service.formatter.NativeFormatter.handleMessage(NativeFormatter.java:162)
        at com.zimbra.cs.service.formatter.NativeFormatter.formatCallback(NativeFormatter.java:129)
        at com.zimbra.cs.service.formatter.Formatter.format(Formatter.java:142)
        at com.zimbra.cs.service.UserServlet.doAuthGet(UserServlet.java:536)
        at com.zimbra.cs.service.UserServlet.doGet(UserServlet.java:314)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:206)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
        at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:169)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.common.filters.Base64Filter.doFilter(Base64Filter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:116)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:473)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:318)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:288)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:318)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:437)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:84)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
        at org.eclipse.jetty.server.Server.handle(Server.java:517)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Thread.java:745)
 
INFO: [HOSTNAME]:8736 [dev] [3.4.8] processors=4, physical.memory.total=13.7G, physical.memory.free=5.0G, swap.space.total=4.0G, swap.space.free=4.0G, heap.memory.used=763.9M, heap.memory.free=209.3M, heap.memory.total=973.1M, heap.memory.max=973.1M, heap.memory.used/total=78.49%, heap.memory.used/max=78.49%, minor.gc.count=133, minor.gc.time=4060ms, major.gc.count=3, major.gc.time=349ms, load.process=1.00%, load.system=7.00%, load.systemAverage=9.00%, thread.count=130, thread.peakCount=163, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operation.size=0, executor.q.priorityOperation.size=0, executor.q.response.size=0, operations.remote.size=0, operations.running.size=0, proxy.count=1, clientEndpoint.count=0, connection.active.count=0, client.connection.count=0, connection.count=0


Any hint?

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Thu Sep 29, 2016 6:46 am
by cerri
Waiting for a response...

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Thu Sep 29, 2016 2:20 pm
by syslint
What is your server type. I can see the load is little high in your server.

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Thu Sep 29, 2016 2:24 pm
by cerri
Thanks for the reply.
The load is normal:

sysadmin@zimbra:~$ top
top - 16:25:21 up 3 days, 5:50, 2 users, load average: 0.16, 0.30, 0.36
Tasks: 195 total, 2 running, 193 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.2%sy, 0.0%ni, 99.3%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 14369468k total, 9362696k used, 5006772k free, 305180k buffers
Swap: 4190204k total, 13360k used, 4176844k free, 3512248k cached

And the load cannot "kill" just the web interface (proxy)...

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Mon Oct 03, 2016 8:57 am
by cerri
Anyone to help?
Anyone from Zimbra?

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Thu Oct 06, 2016 12:12 pm
by ccesario
Hello,

I 'm facing the same problem, I "suspect" that the problem is related to ngix/proxy service. But no solution until now.

best regards,

-Carlos

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Sun Oct 09, 2016 8:18 am
by cerri
Me too.
The solution is to restart the proxy.
The real solution is to disable it: I've got no problems after disabling it.

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Mon Oct 10, 2016 6:30 pm
by ccesario
Hi @cerri,

Yes, I disabled it for testing and we don't have problems. (thank you for sharing it)
But I 'm looking for the root cause, maybe some special config solve it. But I 'm still looking for it :/


Regards,

-Carlos

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Mon Oct 17, 2016 6:13 pm
by ccesario
Any update about it!?

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Posted: Tue Aug 29, 2017 8:44 am
by syed.yakoob
facing the same issue, does anyone have a permanent fix for this?