ERROR: service.PROXY_ERROR 503 service unavailable

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

ERROR: service.PROXY_ERROR 503 service unavailable

Postby kayasaman » Tue Dec 11, 2018 9:47 am

Hi,

I hope someone can help me on this....

I have recently upgraded my Zimbra system from zcs-8.7.11_GA_1854.RHEL6_64 to zcs-8.8.10_GA_3039.RHEL6_64 on CentOS 6. This was done using the official packages, though I had major issues with ldap but finally got that resolved.

The issue I am experiencing at present is that I am simply unable to connect to the Zimbra instance or even issue the zmprov command.

If I try to run zmprov as the Zimbra user I get this output:

Code: Select all

zmprov
ERROR: service.PROXY_ERROR (error while proxying request to target server: HTTP/1.1 503 Service Unavailable)


Running 'zmcontrol status':

Code: Select all

   amavis                  Running
   antispam                Running
   antivirus               Running
   ldap                    Running
   logger                  Running
   mailbox                 Running
   memcached               Running
   mta                     Running
   opendkim                Running
   proxy                   Running
   service webapp          Running
   snmp                    Running
   spell                   Running
   stats                   Running
   zimbra webapp           Running
   zimbraAdmin webapp      Running
   zimlet webapp           Running
   zmconfigd               Running
   


Everything is up and running.

IMAP was working a few hours ago but now I am getting for all clients in the nginx.log:

Code: Select all

[error] 3225#0: *238 An error occurred in mail zmauth:  while SSL handshaking to lookup handler


If I try to bypass the proxy and connect on port 7993 the following shows up in the mailbox.log file:

Code: Select all

2018-12-11 09:09:15,815 ERROR [qtp1286783232-13:https:https://IP/] [] webclient - Error during GetDomainInfoRequest invocation
com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 503 Service Unavailable
ExceptionId:qtp1286783232-13:https:https://IP/:1544519355815:361bf23be2d440db
Code:service.PROXY_ERROR Arg:(url, STR, "https://localhost:7071/service/admin/soap/GetDomainInfoRequest")
   at com.zimbra.common.service.ServiceException.PROXY_ERROR(ServiceException.java:379)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:296)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:193)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:186)
   at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:447)
   at com.zimbra.common.soap.SoapTransport.invokeWithoutSession(SoapTransport.java:433)
   at com.zimbra.client.ZSoapProvisioning.invokeRequest(ZSoapProvisioning.java:80)
   at com.zimbra.client.ZSoapProvisioning.invoke(ZSoapProvisioning.java:58)
   at com.zimbra.client.ZSoapProvisioning.invokeJaxb(ZSoapProvisioning.java:51)
   at com.zimbra.client.ZSoapProvisioning.getDomainInfo(ZSoapProvisioning.java:32)
   at com.zimbra.cs.taglib.tag.GetDomainInfoTag.getInfo(GetDomainInfoTag.java:102)
   at com.zimbra.cs.taglib.tag.GetDomainInfoTag.checkCache(GetDomainInfoTag.java:84)
   at com.zimbra.cs.taglib.tag.GetDomainInfoTag.doTag(GetDomainInfoTag.java:70)
   at org.apache.jsp.tag.web.infra.skin_tag.doTag(skin_tag.java:254)
   at org.apache.jsp.tag.web.infra.skinAndRedirect_tag._jspx_meth_app_005fskin_005f0(skinAndRedirect_tag.java:233)
   at org.apache.jsp.tag.web.infra.skinAndRedirect_tag.doTag(skinAndRedirect_tag.java:156)
   at org.apache.jsp.public_.login_jsp._jspx_meth_app_005fskinAndRedirect_005f0(login_jsp.java:964)
   at org.apache.jsp.public_.login_jsp._jspService(login_jsp.java:220)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
   at org.eclipse.jetty.jsp.JettyJspServlet.service(JettyJspServlet.java:107)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
   at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
   at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:199)
   at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:74)
   at org.eclipse.jetty.servlet.DefaultServlet.sendWelcome(DefaultServlet.java:593)
   at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:479)
   at javax.servlet.http.HttpServlet.doHead(HttpServlet.java:288)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:704)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
   at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   at com.zimbra.webClient.filters.ForwardFilter.doFilter(ForwardFilter.java:88)
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
   at com.zimbra.webClient.filters.SetHeaderFilter.doFilter(SetHeaderFilter.java:248)
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:116)
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:473)
   at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:318)
   at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:288)
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   at com.zimbra.webClient.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:37)
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
   at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
   at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
   at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
   at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:318)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
   at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:84)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
   at org.eclipse.jetty.server.Server.handle(Server.java:517)
   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
   at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
   at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)
   at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
   at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
   at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   at java.lang.Thread.run(Thread.java:748)
   


I have found a similar issue in this thread: viewtopic.php?t=63101 though I don't know if the cause is the same? Certainly I am not sure of what the solution could be as I have tried a few things:

- restart zimbra service
- run extended permission check: /opt/zimbra/libexec/zmfixperms -extended
- rerun the install.sh script
- reset the proxy to defaults: https://wiki.zimbra.com/wiki/Enabling_Z ... _memcached

Checking that the proxy and ports are running and listening:

Code: Select all

# lsof -i :443
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nginx   3222 zimbra   11u  IPv4 975451      0t0  TCP *:https (LISTEN)
nginx   3224 zimbra   11u  IPv4 975451      0t0  TCP *:https (LISTEN)
nginx   3225 zimbra   11u  IPv4 975451      0t0  TCP *:https (LISTEN)
nginx   3226 zimbra   11u  IPv4 975451      0t0  TCP *:https (LISTEN)
nginx   3227 zimbra   11u  IPv4 975451      0t0  TCP *:https (LISTEN)


Code: Select all

lsof -i :11211
COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
memcached 3201 zimbra   26u  IPv4 975387      0t0  TCP *:memcache (LISTEN)
memcached 3201 zimbra   27u  IPv6 975388      0t0  TCP *:memcache (LISTEN)
memcached 3201 zimbra   28u  IPv4 975465      0t0  TCP server.domain.com:memcache->server.domain.com:40642 (ESTABLISHED)
memcached 3201 zimbra   29u  IPv4 975467      0t0  TCP server.domain.com:memcache->server.domain.com:40644 (ESTABLISHED)
memcached 3201 zimbra   30u  IPv4 975469      0t0  TCP server.domain.com:memcache->server.domain.com:40646 (ESTABLISHED)
memcached 3201 zimbra   31u  IPv4 975471      0t0  TCP server.domain.com:memcache->server.domain.com:40648 (ESTABLISHED)
nginx     3224 zimbra   14u  IPv4 975464      0t0  TCP server.domain.com:40642->server.domain.com:memcache (ESTABLISHED)
nginx     3225 zimbra   16u  IPv4 975466      0t0  TCP server.domain.com:40644->server.domain.com:memcache (ESTABLISHED)
nginx     3226 zimbra   18u  IPv4 975468      0t0  TCP server.domain.com:40646->server.domain.com:memcache (ESTABLISHED)
nginx     3227 zimbra   20u  IPv4 975470      0t0  TCP server.domain.com:40648->server.domain.com:memcache (ESTABLISHED)


Looking at the listening ports:

Code: Select all

22 4964/sshd
25 4173/master
80 3222/nginx
110 3222/nginx
111 1265/rpcbind
143 3222/nginx
199 1567/snmpd
389 871/slapd
443 3222/nginx
465 4173/master
587 4173/master
993 3222/nginx
995 3222/nginx
2049 -
3310 3925/clamd
4949 1651/perl
5666 1212/nrpe
7025 2600/java
7071 2600/java
7072 2600/java
7073 2600/java
7110 2600/java
7143 2600/java
7171 893/java
7306 2415/mysqld
7780 3971/httpd
7993 2600/java
7995 2600/java
8080 2600/java
8443 2600/java
8465 3946/opendkim
10024 3289/amavisd
10025 4173/master
10026 3289/amavisd
10027 4173/master
10028 4173/master
10029 4173/master
10030 4173/master
10032 3289/amavisd
10050 1621/zabbix_agentd
11211 3201/memcached
23232 3248/perl
23233 3250/perl
33890 -
36602 1595/xinetd
36915 1287/rpc.statd
39079 1411/rpc.mountd
40286 1411/rpc.mountd
40852 1344/rpc.statd
42675 1411/rpc.mountd
43635 -
45228 1339/rpc.statd
45334 1411/rpc.mountd
46344 1344/rpc.statd
46544 1411/rpc.mountd
55998 1411/rpc.mountd
56829 1287/rpc.statd
58861 1339/rpc.statd



I think the issue started when trying to access a very large folder inside a mailbox. For this particular folder it was very difficult to get any client to open it as it kept timing out so after reading around I issued the:

Code: Select all

zmprov ms `zmhostname` zimbraImapInactiveSessionEhcacheSize 104857600


then restarted the mailbox service. Again attempted to access the folder using both Thunderbird and Evolution. Finally Evolution managed to pull in around 98 emails before disconnecting and I think that's when things went pop.... ??


What more information could I provide to be of help or does anyone have a solution for this??

Many thanks for any ideas!


kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Postby kayasaman » Tue Dec 11, 2018 8:38 pm

Just discovered something interesting in the zmmailboxd.out file:

- primarily these parts standout to me but I have no idea on if these are the cause and if so what a fix for them could be

INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],UNAVAILABLE}{/zimlet}
WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service} has uncovered http methods for path: /
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd


Code: Select all

zmthrdump: Requested thread dump [PID 2600] at Tue Dec 11 11:12:45 2018
2018-12-11 11:12:46.665:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-11 11:12:46.667:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-11 11:12:46.670:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-11 11:12:46.672:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-11 11:12:46.672:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-11 11:12:46.792:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],UNAVAILABLE}{/zimlet}
2018-12-11 11:12:46.801:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,null,UNAVAILABLE}{/zimbraAdmin}
2018-12-11 11:12:46.806:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@4690b489{/zimbra,null,UNAVAILABLE}{/zimbra}
2018-12-11 11:12:46.822:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@4b14c583{/service,null,UNAVAILABLE}{/service}
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
2018-12-11 11:14:34.808:INFO::main: Logging initialized @3817ms
JettyMonitor monitoring thread pool qtp1286783232{STOPPED,10<=0<=250,i=0,q=0}
Zimbra server reserving server socket port=7110 bindaddr=null ssl=false
Zimbra server reserving server socket port=7995 bindaddr=null ssl=false
Zimbra server reserving server socket port=7143 bindaddr=null ssl=false
Zimbra server reserving server socket port=7993 bindaddr=null ssl=false
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
2018-12-11 11:14:37.762:INFO:oejs.SetUIDListener:main: Setting umask=027
2018-12-11 11:14:37.763:INFO:oejs.SetUIDListener:main: Current rlimit_nofiles (soft=524288, hard=524288)
2018-12-11 11:14:37.763:INFO:oejs.SetUIDListener:main: Set rlimit_nofiles (soft=65536, hard=65536)
2018-12-11 11:14:37.772:INFO:oejs.SetUIDListener:main: Opened ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-11 11:14:37.773:INFO:oejs.SetUIDListener:main: Opened ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-11 11:14:37.773:INFO:oejs.SetUIDListener:main: Opened ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-11 11:14:37.774:INFO:oejs.SetUIDListener:main: Opened ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-11 11:14:37.775:INFO:oejs.SetUIDListener:main: Opened ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-11 11:14:37.775:INFO:oejs.SetUIDListener:main: Setting GID=500
2018-12-11 11:14:37.776:INFO:oejs.SetUIDListener:main: Setting UID=500
2018-12-11 11:14:37.786:INFO:oejs.Server:main: jetty-9.3.5.v20151012
2018-12-11 11:14:53.634:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service} has uncovered http methods for path: /
Zimbra server process is running as uid=500 euid=500 gid=500 egid=500
2018-12-11 11:15:13.854:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-11 11:15:16.589:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-11 11:15:16.871:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service}
MultiException[javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}', javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd]
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:846)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
Suppressed:
   |javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
   |   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   |   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   |   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   |   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   |   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   |   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   |   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   |   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   |   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   |   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   |   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.server.Server.start(Server.java:405)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   |   at java.security.AccessController.doPrivileged(Native Method)
   |   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   |   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   |   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   |   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   |   at java.lang.reflect.Method.invoke(Method.java:498)
   |   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   |   at org.eclipse.jetty.start.Main.start(Main.java:457)
   |   at org.eclipse.jetty.start.Main.main(Main.java:75)
Caused by:
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-11 11:15:21.584:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-11 11:15:22.986:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,STARTING}{/zimbra} has uncovered http methods for path: /
2018-12-11 11:15:25.407:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,AVAILABLE}{/zimbra}
2018-12-11 11:15:31.886:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-11 11:15:32.175:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,STARTING}{/zimbraAdmin} has uncovered http methods for path: /
2018-12-11 11:15:34.061:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,AVAILABLE}{/zimbraAdmin}
2018-12-11 11:15:34.640:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],STARTING}{/zimlet} has uncovered http methods for path: /
2018-12-11 11:15:34.642:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-12-11 11:15:34.652:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-12-11
2018-12-11 11:15:34.666:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-12-11 11:15:34.729:INFO:oejs.ServerConnector:main: Started ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-11 11:15:34.752:INFO:oejus.SslContextFactory:main: x509=X509@350da119(jetty,h=[x-ray.optiplex-networks.com],w=[]) for SslContextFactory@367d34c0(file:///opt/zimbra/jetty_base/etc/keystore,null)
2018-12-11 11:15:34.836:INFO:oejs.ServerConnector:main: Started ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-11 11:15:34.839:INFO:oejs.ServerConnector:main: Started ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-11 11:15:34.841:INFO:oejs.ServerConnector:main: Started ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-11 11:15:34.847:INFO:oejs.ServerConnector:main: Started ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-11 11:15:34.847:INFO:oejs.Server:main: Started @63857ms



I could be just clutching at straws here but a quick search for: "couldn't lock rootdir: /opt/zimbra/data/mailboxd"

came up with this link: viewtopic.php?t=62916 which suggested bug 104846: https://bugzilla.zimbra.com/show_bug.cgi?id=104846

However, I don't think this bug seems relevant in this case because the 'active session cache' is not in use as there are no sessions since connectivity is unavailable at this time:

Code: Select all

$ for x in $(seq 1 6); do ls -lh /opt/zimbra/data/mailboxd; done
total 26K
drwxr-x--- 5 zimbra zimbra 5 Dec 11 11:15 imap-active-session-cache_85be333d34866255dbb2082ee3fb1a79544d0878
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 imap-inactive-session-cache_fc164b40eddbe2ccbc469be50b2e7494ea03619a
drwxr-xr-x 2 zimbra zimbra 2 Dec  2 22:54 spnego
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 sync-state-item-cache_eca9d558d785a39bf537e61b19815681b15d280f
total 26K
drwxr-x--- 5 zimbra zimbra 5 Dec 11 11:15 imap-active-session-cache_85be333d34866255dbb2082ee3fb1a79544d0878
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 imap-inactive-session-cache_fc164b40eddbe2ccbc469be50b2e7494ea03619a
drwxr-xr-x 2 zimbra zimbra 2 Dec  2 22:54 spnego
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 sync-state-item-cache_eca9d558d785a39bf537e61b19815681b15d280f
total 26K
drwxr-x--- 5 zimbra zimbra 5 Dec 11 11:15 imap-active-session-cache_85be333d34866255dbb2082ee3fb1a79544d0878
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 imap-inactive-session-cache_fc164b40eddbe2ccbc469be50b2e7494ea03619a
drwxr-xr-x 2 zimbra zimbra 2 Dec  2 22:54 spnego
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 sync-state-item-cache_eca9d558d785a39bf537e61b19815681b15d280f
total 26K
drwxr-x--- 5 zimbra zimbra 5 Dec 11 11:15 imap-active-session-cache_85be333d34866255dbb2082ee3fb1a79544d0878
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 imap-inactive-session-cache_fc164b40eddbe2ccbc469be50b2e7494ea03619a
drwxr-xr-x 2 zimbra zimbra 2 Dec  2 22:54 spnego
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 sync-state-item-cache_eca9d558d785a39bf537e61b19815681b15d280f
total 26K
drwxr-x--- 5 zimbra zimbra 5 Dec 11 11:15 imap-active-session-cache_85be333d34866255dbb2082ee3fb1a79544d0878
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 imap-inactive-session-cache_fc164b40eddbe2ccbc469be50b2e7494ea03619a
drwxr-xr-x 2 zimbra zimbra 2 Dec  2 22:54 spnego
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 sync-state-item-cache_eca9d558d785a39bf537e61b19815681b15d280f
total 26K
drwxr-x--- 5 zimbra zimbra 5 Dec 11 11:15 imap-active-session-cache_85be333d34866255dbb2082ee3fb1a79544d0878
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 imap-inactive-session-cache_fc164b40eddbe2ccbc469be50b2e7494ea03619a
drwxr-xr-x 2 zimbra zimbra 2 Dec  2 22:54 spnego
drwxr-x--- 5 zimbra zimbra 5 Dec  2 23:12 sync-state-item-cache_eca9d558d785a39bf537e61b19815681b15d280f


I will keep digging and in worst case scenario rebuild though if there is something wrong in the /opt/zimbra/store directory then I am not sure what the answer would be.....??
User avatar
king0770
Zimbra Employee
Zimbra Employee
Posts: 237
Joined: Fri Sep 12, 2014 10:44 pm
Contact:

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Postby king0770 » Tue Dec 11, 2018 9:49 pm

Check to see if the /opt/zimbra/jetty/work/service/ directory is owned by root. If so, make sure /opt/zimbra/jetty/work/service/ is owned by zimbra.

Try...

As zimbra
-----------------
zmcontrol stop

As root
-----------
chown -R zimbra:zimbra /opt/zimbra/jetty/work/service

ls -ld /opt/zimbra/jetty/work/service

drwxr-x--- 3 zimbra zimbra 4096 Dec 11 14:38 /opt/zimbra/jetty/work/service <<== should see similar output

ls -ld /opt/zimbra/jetty/work/zimbraAdmin/jsp/ <<== jsp directory should NOT exist after zmcontrol stop, remove the jsp directory if it exists while ZCS services are stopped

As zimbra
---------------
zmcontrol start
--
Rick King
Zimbra Support Team
kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Postby kayasaman » Tue Dec 11, 2018 10:00 pm

Thanks for the response!


Without changing any permissions:

- jsp directory owned by zimbra

Code: Select all

ls -l /opt/zimbra/jetty/work/service/
total 1
drwxr-x--- 2 zimbra zimbra 2 Dec 11 20:51 jsp


Checking the service directory:

Code: Select all

ls -l /opt/zimbra/jetty/work/       
total 8
drwxr-x--- 3 zimbra zimbra   3 Dec 11 08:39 resource-cache
drwxr-x--- 3 zimbra zimbra   3 Dec 11 20:51 service
drwxr-x--- 2 zimbra zimbra   2 Dec 11 08:20 spnego
-rw------- 1 root   zimbra 944 Dec 11 20:51 start_8796029365807146772.properties
drwxr-xr-x 3 zimbra zimbra   3 Dec 11 08:26 zimbra
drwxr-x--- 3 zimbra zimbra   3 Dec 11 20:52 zimbraAdmin
drwxr-x--- 2 zimbra zimbra   2 Dec 11 20:52 zimlet


Here is the output after stopping Zimbra:

Running the chown command:

Code: Select all

# chown -R zimbra:zimbra /opt/zimbra/jetty/work/service
chown: cannot access `/opt/zimbra/jetty/work/service': No such file or directory


This directory doesn't exist which it should according to your output?

Code: Select all

ls -ld /opt/zimbra/jetty/work/service
ls: cannot access /opt/zimbra/jetty/work/service: No such file or directory



This also doesn't exist which is ok:

Code: Select all

ls -ld /opt/zimbra/jetty/work/zimbraAdmin/jsp/
ls: cannot access /opt/zimbra/jetty/work/zimbraAdmin/jsp/: No such file or directory


Should I create the /opt/zimbra/jetty/work/service/ directory manually?
User avatar
king0770
Zimbra Employee
Zimbra Employee
Posts: 237
Joined: Fri Sep 12, 2014 10:44 pm
Contact:

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Postby king0770 » Tue Dec 11, 2018 10:22 pm

Start up ZCS services again (if you hadn't already), and see if the error still occurs, if so check the /opt/zimbra/log/zmmailboxd.out file
--
Rick King
Zimbra Support Team
kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Postby kayasaman » Tue Dec 11, 2018 10:31 pm

Thanks again. I have started up the ZCS service but unfortunately the issue is the same :-(

Here is the zmmailboxd.out part since shutdown / restart:

Code: Select all

zmthrdump: Requested thread dump [PID 22447] at Tue Dec 11 22:25:28 2018
2018-12-11 22:25:30.318:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-11 22:25:30.325:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-11 22:25:30.330:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-11 22:25:30.331:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-11 22:25:30.335:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-11 22:25:30.404:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],UNAVAILABLE}{/zimlet}
2018-12-11 22:25:30.414:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,null,UNAVAILABLE}{/zimbraAdmin}
2018-12-11 22:25:30.421:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@4690b489{/zimbra,null,UNAVAILABLE}{/zimbra}
2018-12-11 22:25:30.451:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@4b14c583{/service,null,UNAVAILABLE}{/service}
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
2018-12-11 22:27:14.400:INFO::main: Logging initialized @3461ms
JettyMonitor monitoring thread pool qtp1286783232{STOPPED,10<=0<=250,i=0,q=0}
Zimbra server reserving server socket port=7110 bindaddr=null ssl=false
Zimbra server reserving server socket port=7995 bindaddr=null ssl=false
Zimbra server reserving server socket port=7143 bindaddr=null ssl=false
Zimbra server reserving server socket port=7993 bindaddr=null ssl=false
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
2018-12-11 22:27:17.614:INFO:oejs.SetUIDListener:main: Setting umask=027
2018-12-11 22:27:17.614:INFO:oejs.SetUIDListener:main: Current rlimit_nofiles (soft=524288, hard=524288)
2018-12-11 22:27:17.615:INFO:oejs.SetUIDListener:main: Set rlimit_nofiles (soft=65536, hard=65536)
2018-12-11 22:27:17.628:INFO:oejs.SetUIDListener:main: Opened ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-11 22:27:17.628:INFO:oejs.SetUIDListener:main: Opened ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-11 22:27:17.629:INFO:oejs.SetUIDListener:main: Opened ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-11 22:27:17.630:INFO:oejs.SetUIDListener:main: Opened ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-11 22:27:17.630:INFO:oejs.SetUIDListener:main: Opened ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-11 22:27:17.631:INFO:oejs.SetUIDListener:main: Setting GID=500
2018-12-11 22:27:17.631:INFO:oejs.SetUIDListener:main: Setting UID=500
2018-12-11 22:27:17.643:INFO:oejs.Server:main: jetty-9.3.5.v20151012
2018-12-11 22:27:32.204:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service} has uncovered http methods for path: /
Zimbra server process is running as uid=500 euid=500 gid=500 egid=500
2018-12-11 22:27:50.704:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-11 22:27:53.067:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-11 22:27:53.270:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service}
MultiException[javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}', javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd]
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:846)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
Suppressed:
   |javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
   |   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   |   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   |   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   |   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   |   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   |   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   |   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   |   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   |   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   |   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   |   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.server.Server.start(Server.java:405)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   |   at java.security.AccessController.doPrivileged(Native Method)
   |   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   |   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   |   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   |   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   |   at java.lang.reflect.Method.invoke(Method.java:498)
   |   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   |   at org.eclipse.jetty.start.Main.start(Main.java:457)
   |   at org.eclipse.jetty.start.Main.main(Main.java:75)
Caused by:
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-11 22:27:57.302:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-11 22:27:58.601:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,STARTING}{/zimbra} has uncovered http methods for path: /
2018-12-11 22:28:00.470:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,AVAILABLE}{/zimbra}
2018-12-11 22:28:06.170:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-11 22:28:06.555:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,STARTING}{/zimbraAdmin} has uncovered http methods for path: /
2018-12-11 22:28:08.126:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,AVAILABLE}{/zimbraAdmin}
2018-12-11 22:28:09.069:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],STARTING}{/zimlet} has uncovered http methods for path: /
2018-12-11 22:28:09.071:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-12-11 22:28:09.083:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-12-11
2018-12-11 22:28:09.089:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-12-11 22:28:09.154:INFO:oejs.ServerConnector:main: Started ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-11 22:28:09.208:INFO:oejus.SslContextFactory:main: x509=X509@350da119(jetty,h=[x-ray.optiplex-networks.com],w=[]) for SslContextFactory@367d34c0(file:///opt/zimbra/jetty_base/etc/keystore,null)
2018-12-11 22:28:09.310:INFO:oejs.ServerConnector:main: Started ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-11 22:28:09.315:INFO:oejs.ServerConnector:main: Started ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-11 22:28:09.317:INFO:oejs.ServerConnector:main: Started ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-11 22:28:09.320:INFO:oejs.ServerConnector:main: Started ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-11 22:28:09.320:INFO:oejs.Server:main: Started @58382ms
User avatar
king0770
Zimbra Employee
Zimbra Employee
Posts: 237
Joined: Fri Sep 12, 2014 10:44 pm
Contact:

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Postby king0770 » Tue Dec 11, 2018 10:54 pm

Hmmm…

Couldn't lock rootDir: /opt/zimbra/data/mailboxd

It seems a corruption issue with mailboxd directory. Could you please follow below steps to fix the issue.

su - zimbra
zmmailboxdctl stop

exit

As root…

mv /opt/zimbra/data/mailboxd /opt/zimbra/data/mailboxd.org
mkdir /opt/zimbra/data/mailbox/
chmod 755 /opt/zimbra/data/mailbox
chown zimbra:zimbra /opt/zimbra/data/mailbox

su - zimbra
zmmailboxd restart
--
Rick King
Zimbra Support Team
kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Postby kayasaman » Tue Dec 11, 2018 11:09 pm

Unfortunately the issue persists :-(

I moved the original mailboxd to mailboxd.org and created the mailbox dir... though upon restart the system automatically a new mailboxd directory.

Again zmmailboxd.out:

Code: Select all

OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
2018-12-11 23:02:15.519:INFO::main: Logging initialized @2488ms
JettyMonitor monitoring thread pool qtp1286783232{STOPPED,10<=0<=250,i=0,q=0}
Zimbra server reserving server socket port=7110 bindaddr=null ssl=false
Zimbra server reserving server socket port=7995 bindaddr=null ssl=false
Zimbra server reserving server socket port=7143 bindaddr=null ssl=false
Zimbra server reserving server socket port=7993 bindaddr=null ssl=false
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
2018-12-11 23:02:18.053:INFO:oejs.SetUIDListener:main: Setting umask=027
2018-12-11 23:02:18.061:INFO:oejs.SetUIDListener:main: Current rlimit_nofiles (soft=524288, hard=524288)
2018-12-11 23:02:18.062:INFO:oejs.SetUIDListener:main: Set rlimit_nofiles (soft=65536, hard=65536)
2018-12-11 23:02:18.070:INFO:oejs.SetUIDListener:main: Opened ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-11 23:02:18.071:INFO:oejs.SetUIDListener:main: Opened ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-11 23:02:18.071:INFO:oejs.SetUIDListener:main: Opened ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-11 23:02:18.072:INFO:oejs.SetUIDListener:main: Opened ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-11 23:02:18.073:INFO:oejs.SetUIDListener:main: Opened ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-11 23:02:18.073:INFO:oejs.SetUIDListener:main: Setting GID=500
2018-12-11 23:02:18.074:INFO:oejs.SetUIDListener:main: Setting UID=500
2018-12-11 23:02:18.084:INFO:oejs.Server:main: jetty-9.3.5.v20151012
2018-12-11 23:02:32.053:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service} has uncovered http methods for path: /
Zimbra server process is running as uid=500 euid=500 gid=500 egid=500
2018-12-11 23:02:50.303:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-11 23:02:53.016:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-11 23:02:53.227:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service}
MultiException[javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}', javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd]
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:846)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
Suppressed:
   |javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
   |   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   |   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   |   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   |   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   |   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   |   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   |   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   |   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   |   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   |   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   |   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   |   at org.eclipse.jetty.server.Server.start(Server.java:405)
   |   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   |   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   |   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   |   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   |   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   |   at java.security.AccessController.doPrivileged(Native Method)
   |   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   |   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   |   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   |   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   |   at java.lang.reflect.Method.invoke(Method.java:498)
   |   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   |   at org.eclipse.jetty.start.Main.start(Main.java:457)
   |   at org.eclipse.jetty.start.Main.main(Main.java:75)
Caused by:
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
   at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
   at javax.servlet.GenericServlet.init(GenericServlet.java:244)
   at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
   at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
   at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
   at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
   at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
   at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
   at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
   at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
   at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
   at org.eclipse.jetty.server.Server.start(Server.java:405)
   at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
   at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
   at org.eclipse.jetty.server.Server.doStart(Server.java:372)
   at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
   at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
   at org.eclipse.jetty.start.Main.start(Main.java:457)
   at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-11 23:02:58.013:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-11 23:02:58.587:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,STARTING}{/zimbra} has uncovered http methods for path: /
2018-12-11 23:03:00.400:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,AVAILABLE}{/zimbra}
2018-12-11 23:03:06.390:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-11 23:03:06.588:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,STARTING}{/zimbraAdmin} has uncovered http methods for path: /
2018-12-11 23:03:08.005:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,AVAILABLE}{/zimbraAdmin}
2018-12-11 23:03:08.615:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],STARTING}{/zimlet} has uncovered http methods for path: /
2018-12-11 23:03:08.616:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-12-11 23:03:08.629:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-12-11
2018-12-11 23:03:08.642:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-12-11 23:03:08.698:INFO:oejs.ServerConnector:main: Started ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-11 23:03:08.715:INFO:oejus.SslContextFactory:main: x509=X509@350da119(jetty,h=[x-ray.optiplex-networks.com],w=[]) for SslContextFactory@367d34c0(file:///opt/zimbra/jetty_base/etc/keystore,null)
2018-12-11 23:03:08.783:INFO:oejs.ServerConnector:main: Started ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-11 23:03:08.785:INFO:oejs.ServerConnector:main: Started ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-11 23:03:08.787:INFO:oejs.ServerConnector:main: Started ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-11 23:03:08.799:INFO:oejs.ServerConnector:main: Started ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-11 23:03:08.800:INFO:oejs.Server:main: Started @55769ms


In /opt/zimbra/data directory:

Code: Select all

$ ls -l
total 89
drwxr-xr-x  2 zimbra  zimbra  2 Dec  2 22:50 altermime
drwxrwxr-x  8 zimbra  zimbra  9 Dec 11 23:02 amavisd
drwxrwxr-x  3 zimbra  zimbra  3 Sep 28 15:54 cbpolicyd
drwxrwxr-x  4 zimbra  zimbra  5 Dec 11 23:03 clamav
drwxrwxr-x  4 zimbra  zimbra  4 Dec 11 08:21 dns
drwxr-xr-x  8 zimbra  zimbra  8 Dec 11 08:16 httpd
drwxr-xr-x  7 zimbra  zimbra 29 Dec 11 08:08 ldap
drwxr-xr-x  5 zimbra  zimbra  5 Dec  2 22:50 ldap.bak
drwxr-xr-x  2 zimbra  zimbra  2 Dec 11 22:57 mailbox
drwxr-x---  3 zimbra  zimbra  4 Dec 11 23:02 mailboxd
drwxr-xr-x  6 zimbra  zimbra  7 Dec 11 22:27 mailboxd.org
drwxr-xr-x  3 zimbra  zimbra  3 Dec 11 08:15 nginx
drwxrwxr-x  2 zimbra  zimbra  2 Sep 28 15:54 opendkim
drwxr-xr-x  4 postfix zimbra  4 Dec  2 22:50 postfix
drwxr-xr-x  3 zimbra  zimbra  3 Dec  2 22:50 sasl2
drwxr-xr-x  4 zimbra  zimbra  4 Dec 11 08:21 snmp
drwxr-xr-x  4 zimbra  zimbra  4 Dec 11 08:14 spamassassin
drwxrwxrwt 13 zimbra  zimbra 77 Dec 11 23:08 tmp


zmprov -v still not functional:

Code: Select all

$ zmprov -v
ERROR: service.PROXY_ERROR (error while proxying request to target server: HTTP/1.1 503 Service Unavailable)
com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 503 Service Unavailable
ExceptionId:main:1544569573135:d21dc1e971841ffb
Code:service.PROXY_ERROR Arg:(url, STR, "https://localhost:7071/service/admin/soap/AuthRequest")
   at com.zimbra.common.service.ServiceException.PROXY_ERROR(ServiceException.java:379)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:296)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:193)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:186)
   at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:447)
   at com.zimbra.common.soap.SoapTransport.invokeWithoutSession(SoapTransport.java:433)
   at com.zimbra.cs.account.soap.SoapProvisioning.invokeRequest(SoapProvisioning.java:624)
   at com.zimbra.cs.account.soap.SoapProvisioning.invoke(SoapProvisioning.java:669)
   at com.zimbra.cs.account.soap.SoapProvisioning.soapAdminAuthenticate(SoapProvisioning.java:559)
   at com.zimbra.cs.account.soap.SoapProvisioning.soapZimbraAdminAuthenticate(SoapProvisioning.java:588)
   at com.zimbra.cs.account.ProvUtil.initProvisioning(ProvUtil.java:1043)
   at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:4099)
User avatar
king0770
Zimbra Employee
Zimbra Employee
Posts: 237
Joined: Fri Sep 12, 2014 10:44 pm
Contact:

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Postby king0770 » Tue Dec 11, 2018 11:48 pm

What do you have for the following?

zmprov -l gs `zmhostname` | grep -i service <<== note the use of the minus L option
--
Rick King
Zimbra Support Team
kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Postby kayasaman » Tue Dec 11, 2018 11:51 pm

This is the output:

Code: Select all

$ zmprov -l gs `zmhostname` | grep -i service
zimbraChatServiceEnabled: TRUE
zimbraHttpContextPathBasedThreadPoolBalancingFilterRules: /service:max=80%
zimbraMtaAuthURL: https://server.domain.com:443/service/soap/
zimbraServiceEnabled: amavis
zimbraServiceEnabled: antivirus
zimbraServiceEnabled: antispam
zimbraServiceEnabled: opendkim
zimbraServiceEnabled: logger
zimbraServiceEnabled: service
zimbraServiceEnabled: zimbra
zimbraServiceEnabled: zimbraAdmin
zimbraServiceEnabled: zimlet
zimbraServiceEnabled: mailbox
zimbraServiceEnabled: memcached
zimbraServiceEnabled: mta
zimbraServiceEnabled: stats
zimbraServiceEnabled: proxy
zimbraServiceEnabled: snmp
zimbraServiceEnabled: ldap
zimbraServiceEnabled: spell
zimbraServiceHostname: server.domain.com
zimbraServiceInstalled: amavis
zimbraServiceInstalled: antivirus
zimbraServiceInstalled: antispam
zimbraServiceInstalled: opendkim
zimbraServiceInstalled: logger
zimbraServiceInstalled: mailbox
zimbraServiceInstalled: memcached
zimbraServiceInstalled: mta
zimbraServiceInstalled: dnscache
zimbraServiceInstalled: stats
zimbraServiceInstalled: proxy
zimbraServiceInstalled: snmp
zimbraServiceInstalled: spell
zimbraServiceInstalled: ldap
zimbraStatThreadNamePrefix: AnonymousIoService


I just changed the actual fqdn to server.domain.com but otherwise is correct.

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 18 guests