Page 1 of 1

Simultaneous ZCS 8.6 > 8.8.12 & Ubuntu 14.04LTS to 16.04 to 18.04 Upgrade

Posted: Thu Apr 04, 2019 5:53 pm
by packetdog
So let's get it out there- this was really dumb, caused by a miscommunication, and a junior admin not snapping the virtual server prior to attempting anything. Ugh.

So, the do-release-upgrade command was run twice in succession on an Ubuntu 14.04 box, updating it first to 16.04 LTS and then to 18.04 LTS. This broke lots of things, including Zimbra. Here's what we've accomplished so far:

- We moved the /opt/zimbra folder to zimbra.old, and installed the 8.8.12 version of ZCS to /opt/zimbra using the -s switch. (Since 8.6 was not released for Ubuntu 18.04)
- We moved this new installation to /opt/zimbra.new and moved /opt/zimbra.old back to /opt/zimbra.
- We removed the contents of the [/opt/zimbra/] bin common/bin common/lib common/libexec common/sbin common/share lib libexec directories.
- We copied all of the contents of [/opt/zimbra.new] bin common/bin common/lib common/libexec common/sbin common/share lib libexec directories to [/opt/zimbra] respectively.
- We ran and seemingly successfully upgraded the ZCS install using the zmsetup.pl script. It walked through all the steps and patches very nicely.

Amazingly, all of the services started, mail is flowing in and out, and users can login to the web interface, IMAP daemon, and admins can access the admin portal!

Output of zmcontrol -v is: Release 8.8.12.GA.3794.UBUNTU18.64 UBUNTU18_64 FOSS edition.

zmcontrol start looks like this:

Code: Select all

zimbra@servername:~$ zmcontrol start
Host mail.servername.com
        Starting zmconfigd...Done.
        Starting logger...Done.
        Starting mailbox...Done.
        Starting memcached...Done.
        Starting amavis...Done.
        Starting antispam...Done.
        Starting antivirus...Done.
        Starting opendkim...Done.
        Starting snmp...Done.
        Starting spell...Done.
        Starting mta...Done.
        Starting stats...Done.
        Starting service webapp...Done.
        Starting zimbra webapp...Done.
        Starting zimbraAdmin webapp...Done.
        Starting zimlet webapp...Done.
zimbra@servername:~$


HOWEVER, I still have 2 unresolved issues that I know about, and I'm sure countless others that I don't know about. I wondered if the gurus here could help me isolate these issues.

Issue 1:
When attempting to run the 8.8.12 installer to do a 'cleaner' in-place upgrade I receive an error as follows:

Code: Select all

root@servername:~/zcs-8.8.12_GA_3794.UBUNTU18_64.20190329045002# ./install.sh

Operations logged to /tmp/install.log.jcDtcnRP
Checking for existing installation...
    zimbra-drive...FOUND zimbra-drive-1.0.12.1542291479-1.u18
    zimbra-imapd...NOT FOUND
    zimbra-patch...FOUND zimbra-patch-8.8.12.1552477867.p0-1.u18
    zimbra-mta-patch...FOUND zimbra-mta-patch-8.8.12.1552427139.p0-1.u18
    zimbra-proxy-patch...FOUND zimbra-proxy-patch-8.8.12.1552427139.p0-1.u18
    zimbra-license-tools...NOT FOUND
    zimbra-license-extension...NOT FOUND
    zimbra-network-store...NOT FOUND
    zimbra-network-modules-ng...NOT FOUND
    zimbra-chat...FOUND zimbra-chat-2.0.2.1546498111-1.u18
    zimbra-talk...NOT FOUND
    zimbra-ldap...FOUND zimbra-ldap-8.8.12.GA.3794.UBUNTU18.64
    zimbra-logger...FOUND zimbra-logger-8.8.12.GA.3794.UBUNTU18.64
    zimbra-mta...FOUND zimbra-mta-8.8.12.GA.3794.UBUNTU18.64
    zimbra-dnscache...NOT FOUND
    zimbra-snmp...FOUND zimbra-snmp-8.8.12.GA.3794.UBUNTU18.64
    zimbra-store...FOUND zimbra-store-8.8.12.GA.3794.UBUNTU18.64
    zimbra-apache...FOUND zimbra-apache-8.8.12.GA.3794.UBUNTU18.64
    zimbra-spell...FOUND zimbra-spell-8.8.12.GA.3794.UBUNTU18.64
    zimbra-convertd...NOT FOUND
    zimbra-memcached...FOUND zimbra-memcached-1:1.4.37-2.u18
    zimbra-proxy...FOUND zimbra-proxy-8.8.12.GA.3794.UBUNTU18.64
    zimbra-archiving...NOT FOUND
    zimbra-core...FOUND zimbra-core-8.8.12.GA.3794.UBUNTU18.64
ZCS upgrade from 8.8.12 to 8.8.12 will be performed.
Validating ldap configuration
Unknown Error: It should be impossible to reach this statement.


If I leave Zimbra running while executing the install.sh above, the install.log file looks like this:

Code: Select all

COMMAND: ldap start
slapd already running: pid 20115


..and if Zimbra is not running, then the install.log looks like this:

Code: Select all

COMMAND: ldap start
Started slapd: pid 38209


So I'm not sure how to diagnose that farther. I'm concerned about it more for future patching and upgrades, not so much to run it now since the server and all services are running.

Issue 2:
When logged in as an admin, if I attempt to access anything related to stats, I receive a pop up box like this:
Image

Then /opt/zimbra/log/mailbox.log shows this:

Code: Select all

2019-04-04 12:40:17,200 WARN  [qtp573673894-198:https:https://mail.servername.com:7071/service/admin/soap/GetLoggerStatsRequest] [name=packetdog@servername.com;mid=5;ip=192.168.10.254;port=51235;ua=ZimbraWebClient
- GC73 (Win);soapId=22abf379;] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: system failure: Unable to read logger stats
ExceptionId:qtp573673894-198:https:https://mail.servername.com:7071/service/admin/soap/GetLoggerStatsRequest:1554399617200:d7dbbc4b7271db7b
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
        at com.zimbra.cs.service.admin.GetLoggerStats.execfetch(GetLoggerStats.java:453)
        at com.zimbra.cs.service.admin.GetLoggerStats.fetchColumnData(GetLoggerStats.java:268)
        at com.zimbra.cs.service.admin.GetLoggerStats.handle(GetLoggerStats.java:175)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:646)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:491)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:278)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:307)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:216)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:211)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
        at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:169)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:125)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:473)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:318)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:288)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:318)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:437)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:84)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
        at org.eclipse.jetty.server.Server.handle(Server.java:517)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
        at java.base/java.net.Socket.connect(Socket.java:591)
        at java.base/java.net.Socket.connect(Socket.java:540)
        at java.base/java.net.Socket.<init>(Socket.java:436)
        at java.base/java.net.Socket.<init>(Socket.java:246)
        at com.zimbra.cs.service.admin.GetLoggerStats.execfetch(GetLoggerStats.java:394)
        ... 60 more
2019-04-04 12:40:17,205 INFO  [qtp573673894-198:https:https://mail.servername.com:7071/service/admin/soap/GetLoggerStatsRequest] [name=packetdog@servername.com;mid=5;ip=192.168.10.254;port=51235;ua=ZimbraWebClient
- GC73 (Win);soapId=22abf379;] soap - GetLoggerStatsRequest elapsed=6


I've run the zmsyslogsetup command, but it errors out as follows:

I've also tried reinstalling rsyslog, no change. I also tried zmupdateauthkeys.

Any ideas?

Re: Simultaneous ZCS 8.6 > 8.8.12 & Ubuntu 14.04LTS to 16.04 to 18.04 Upgrade

Posted: Sat Apr 20, 2019 9:33 am
by packetdog
Hi Folks,

For Issue 2 above, still don't have any working stats, and service status is offline. I noticed in the last logfile above that it references my Gateway IP (192.168.10.254) at the very end. Is that normal? I would expect it to reference the IP of the Zimbra server, or the IP of the computer making the request. Could this point to something misconfigured? Any guidance is appreciated.

Re: Simultaneous ZCS 8.6 > 8.8.12 & Ubuntu 14.04LTS to 16.04 to 18.04 Upgrade

Posted: Sat Apr 20, 2019 10:40 am
by phoenix
If you want a 'clean installation' then my suggestion would be to use the ZeXtras Migration tool to move your original server to a new VM. BTW, ZCS on the version of Ubuntu that you have is still considered beta.

Re: Simultaneous ZCS 8.6 > 8.8.12 & Ubuntu 14.04LTS to 16.04 to 18.04 Upgrade

Posted: Sun Aug 04, 2019 1:44 pm
by packetdog
I wanted to update everyone. The ZExtras Migration wouldn't install on the broken server, so it was more or less a no-go.

So, for what it's worth, I rebuilt a Ubuntu 16.04 VM and reinstalled Zimbra on it. I loosely followed this guide (here: https://syslint.com/blog/tutorial/zimbra-server-migration-and-zimbra-account-transfer-the-perfect-method/) to get everything migrated over, and it's worked really really well. I was surprised. Thank you to everyone for the help!