hello folks,

I'm having a serious issue here.

We run the appliance version of Zimbra 6.07. Everything was running fine until I noticed today that some mails were bounced. I tried to open the GUI management console and got nothing. It waited forever for the services status to display but nothing happened. So finally, I decided to reboot the appliance.

After the reboot, I suddenly was welcomed with 'localhost.localdomain'. This is NOT good. After a bit of trouble shooting I noticed that the localconfig.xml could not be found according to the logs. The file was there, but it was empty???? I copied a backup copy into place but I'm still stuck here. Nothing is working.

I tried the troubleshooting tips from alot of posts around here (DNS, MX, hosts file, resolv.conf) All naming related things seem to be ok, zmsshkeygen pukes me an error message which I think could be related.

Generating public/private dsa key pair.
Your identification has been saved in /opt/zimbra/.ssh/zimbra_identity.
Your public key has been saved in /opt/zimbra/.ssh/zimbra_identity.pub.
The key fingerprint is:
6e:1b:b2:36:03:93:85:7a:de:62:b3:89:79:49:bd:bd zimbra.infra.local
ERROR: service.FAILURE (system failure: unable to lookup server by name: zimbra.infra.local message: [LDAP: error code 49 - Invalid Credentials]) (cause: javax.naming.AuthenticationException [LDAP: error code 49 - Invalid Credentials])


also, the mailbox.log has some nice things:
2010-10-26 22:35:52,899 INFO [ImapServer-10546] [] imap - [172.16.1.11] connected
2010-10-26 22:35:52,900 INFO [Pop3Server-11147] [ip=172.16.1.11;] pop - connected
2010-10-26 22:35:52,907 INFO [Pop3Server-11147] [ip=172.16.1.11;] pop - quit from client
2010-10-26 22:35:52,908 INFO [Pop3Server-11147] [] ProtocolHandler - Handler exiting normally
2010-10-26 22:35:52,914 INFO [ImapServer-10546] [] ProtocolHandler - Handler exiting normally
2010-10-26 22:35:52,973 INFO [btpool0-2511://localhost:7071/service/admin/soap/GetDomainInfoRequest] [ip=127.0.0.1;] soap - GetDomainInfoRequest
2010-10-26 22:35:53,024 INFO [btpool0-2511://localhost:7071/service/admin/soap/GetDomainInfoRequest] [ip=127.0.0.1;] soap - GetDomainInfoRequest
2010-10-26 22:36:21,461 INFO [btpool0-2509://zimbra.vmguru.nl/service/soap/] [name=meagain@vmguru.nl;mid=12;ip=86.82.120.204;ua= Zimbra Desktop/1.0.4_1833_Mac;] soap - GetInfoRequest
2010-10-26 22:37:20,812 INFO [Shutdown] [] log - Shutdown hook executing
2010-10-26 22:37:21,610 INFO [Shutdown] [] StatsImageServlet - Servlet StatsImageServlet shutting down
2010-10-26 22:37:21,610 INFO [Shutdown] [] FileUploadServlet - Servlet FileUploadServlet shutting down
2010-10-26 22:37:21,610 INFO [Shutdown] [] PublicICalServlet - Servlet PublicICalServlet shutting down
2010-10-26 22:37:21,610 INFO [Shutdown] [] account - Servlet PreAuthServlet shutting down
2010-10-26 22:37:21,610 INFO [Shutdown] [] mailbox - Servlet UserServlet shutting down
2010-10-26 22:37:21,610 INFO [Shutdown] [] ContentServlet - Servlet ContentServlet shutting down
2010-10-26 22:37:21,610 INFO [Shutdown] [] soap - Servlet AdminServlet shutting down
2010-10-26 22:37:21,610 INFO [Shutdown] [] TcpServer/7025 - LmtpServer initiating shutdown
2010-10-26 22:37:21,610 INFO [MailboxPurge] [] purge - Shutting down purge thread.
2010-10-26 22:37:21,631 INFO [Shutdown] [] TcpServer/7025 - LmtpServer shutting down idle thread pool
2010-10-26 22:37:21,631 INFO [Shutdown] [] TcpServer/110 - Pop3Server initiating shutdown
2010-10-26 22:37:21,631 INFO [Shutdown] [] TcpServer/110 - Pop3Server shutting down idle thread pool
2010-10-26 22:37:21,631 INFO [Shutdown] [] TcpServer/995 - Pop3SSLServer initiating shutdown
2010-10-26 22:37:21,611 INFO [LmtpServer] [] TcpServer/7025 - finished accept loop
2010-10-26 22:37:21,631 INFO [Pop3Server] [] TcpServer/110 - finished accept loop
2010-10-26 22:37:21,631 INFO [Pop3SSLServer] [] TcpServer/995 - finished accept loop
2010-10-26 22:37:21,675 INFO [Shutdown] [] TcpServer/995 - Pop3SSLServer shutting down idle thread pool
2010-10-26 22:37:21,675 INFO [Shutdown] [] TcpServer/143 - ImapServer initiating shutdown
2010-10-26 22:37:21,675 INFO [Shutdown] [] TcpServer/143 - ImapServer shutting down idle thread pool
2010-10-26 22:37:21,675 INFO [Shutdown] [] TcpServer/993 - ImapSSLServer initiating shutdown
2010-10-26 22:37:21,675 INFO [Shutdown] [] ProtocolHandler - graceful shutdown requested
2010-10-26 22:37:21,675 INFO [ImapServer] [] TcpServer/143 - finished accept loop
2010-10-26 22:37:21,675 INFO [ImapSSLServer] [] TcpServer/993 - finished accept loop
2010-10-26 22:37:25,771 WARN [Shutdown] [] session - exception recording unloaded session's RECENT limit
com.zimbra.common.service.ServiceException: system failure: getting database connection
ExceptionId:Shutdown:1288125445770:050c902ea1947ab 8
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:248)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 328)
at com.zimbra.cs.mailbox.Mailbox$MailboxChange.getCon nection(Mailbox.java:310)
at com.zimbra.cs.mailbox.Mailbox.getOperationConnecti on(Mailbox.java:1130)
at com.zimbra.cs.db.DbMailItem.saveMetadata(DbMailIte m.java:670)
at com.zimbra.cs.mailbox.MailItem.saveMetadata(MailIt em.java:2856)
at com.zimbra.cs.mailbox.MailItem.saveMetadata(MailIt em.java:2849)
at com.zimbra.cs.mailbox.Folder.checkpointRECENT(Fold er.java:565)
at com.zimbra.cs.mailbox.Mailbox.recordImapSession(Ma ilbox.java:2436)
at com.zimbra.cs.imap.ImapFolder.snapshotRECENT(ImapF older.java:154)
at com.zimbra.cs.imap.ImapFolder.unregister(ImapFolde r.java:143)
at com.zimbra.cs.imap.ImapHandler.unsetSelectedFolder (ImapHandler.java:825)
at com.zimbra.cs.imap.TcpImapHandler.dropConnection(T cpImapHandler.java:155)
at com.zimbra.cs.imap.ImapHandler.dropConnection(Imap Handler.java:3719)
at com.zimbra.cs.tcpserver.ProtocolHandler.hardShutdo wn(ProtocolHandler.java:246)
at com.zimbra.cs.tcpserver.ProtocolHandler.gracefulSh utdown(ProtocolHandler.java:237)
at com.zimbra.cs.tcpserver.TcpServer.shutdownActiveHa ndlers(TcpServer.java:116)
at com.zimbra.cs.tcpserver.TcpServer.shutdown(TcpServ er.java:147)
at com.zimbra.cs.tcpserver.TcpServer.shutdown(TcpServ er.java:131)
at com.zimbra.cs.imap.ImapServer.shutdownImapServers( ImapServer.java:111)
at com.zimbra.cs.util.Zimbra.shutdown(Zimbra.java:299 )
at com.zimbra.soap.SoapServlet.destroy(SoapServlet.ja va:138)
at org.mortbay.jetty.servlet.ServletHolder.destroyIns tance(ServletHolder.java:318)
at org.mortbay.jetty.servlet.ServletHolder.doStop(Ser vletHolder.java:289)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.servlet.ServletHandler.doStop(Se rvletHandler.java:174)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.servlet.SessionHandler.doStop(Se ssionHandler.java:125)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.handler.ContextHandler.doStop(Co ntextHandler.java:591)
at org.mortbay.jetty.webapp.WebAppContext.doStop(WebA ppContext.java:505)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerCollection.doStop (HandlerCollection.java:169)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerCollection.doStop (HandlerCollection.java:169)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.handler.DebugHandler.doStop(Debu gHandler.java:135)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.Server.doStop(Server.java:283)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.Server$ShutdownHookThread.run(Se rver.java:561)
Caused by: com.mysql.jdbc.exceptions.MySQLNonTransientConnect ionException: Server connection failure during transaction. Due to underlying exception: 'java.net.ConnectException: Connection refused'.

** BEGIN NESTED EXCEPTION **

java.net.ConnectException
MESSAGE: Connection refused

STACKTRACE:

java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl .java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSoc ketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.j ava:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.j ava:366)
at java.net.Socket.connect(Socket.java:525)
at java.net.Socket.connect(Socket.java:475)
at java.net.Socket.<init>(Socket.java:372)
at java.net.Socket.<init>(Socket.java:215)
at com.mysql.jdbc.StandardSocketFactory.connect(Stand ardSocketFactory.java:256)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:271)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2921)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 555)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:285)
at java.sql.DriverManager.getConnection(DriverManager .java:582)
at java.sql.DriverManager.getConnection(DriverManager .java:154)
at org.apache.commons.dbcp.DriverManagerConnectionFac tory.createConnection(DriverManagerConnectionFacto ry.java:68)
at com.zimbra.cs.db.DbPool$ZimbraConnectionFactory.cr eateConnection(DbPool.java:239)
at org.apache.commons.dbcp.PoolableConnectionFactory. makeObject(PoolableConnectionFactory.java:294)
at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:1148)
at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:96)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 309)
at com.zimbra.cs.mailbox.Mailbox$MailboxChange.getCon nection(Mailbox.java:310)
at com.zimbra.cs.mailbox.Mailbox.getOperationConnecti on(Mailbox.java:1130)
at com.zimbra.cs.db.DbMailItem.saveMetadata(DbMailIte m.java:670)
at com.zimbra.cs.mailbox.MailItem.saveMetadata(MailIt em.java:2856)
at com.zimbra.cs.mailbox.MailItem.saveMetadata(MailIt em.java:2849)
at com.zimbra.cs.mailbox.Folder.checkpointRECENT(Fold er.java:565)
at com.zimbra.cs.mailbox.Mailbox.recordImapSession(Ma ilbox.java:2436)
at com.zimbra.cs.imap.ImapFolder.snapshotRECENT(ImapF older.java:154)
at com.zimbra.cs.imap.ImapFolder.unregister(ImapFolde r.java:143)
at com.zimbra.cs.imap.ImapHandler.unsetSelectedFolder (ImapHandler.java:825)
at com.zimbra.cs.imap.TcpImapHandler.dropConnection(T cpImapHandler.java:155)
at com.zimbra.cs.imap.ImapHandler.dropConnection(Imap Handler.java:3719)
at com.zimbra.cs.tcpserver.ProtocolHandler.hardShutdo wn(ProtocolHandler.java:246)
at com.zimbra.cs.tcpserver.ProtocolHandler.gracefulSh utdown(ProtocolHandler.java:237)
at com.zimbra.cs.tcpserver.TcpServer.shutdownActiveHa ndlers(TcpServer.java:116)
at com.zimbra.cs.tcpserver.TcpServer.shutdown(TcpServ er.java:147)
at com.zimbra.cs.tcpserver.TcpServer.shutdown(TcpServ er.java:131)
at com.zimbra.cs.imap.ImapServer.shutdownImapServers( ImapServer.java:111)
at com.zimbra.cs.util.Zimbra.shutdown(Zimbra.java:299 )
at com.zimbra.soap.SoapServlet.destroy(SoapServlet.ja va:138)
at org.mortbay.jetty.servlet.ServletHolder.destroyIns tance(ServletHolder.java:318)
at org.mortbay.jetty.servlet.ServletHolder.doStop(Ser vletHolder.java:289)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.servlet.ServletHandler.doStop(Se rvletHandler.java:174)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.servlet.SessionHandler.doStop(Se ssionHandler.java:125)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.handler.ContextHandler.doStop(Co ntextHandler.java:591)
at org.mortbay.jetty.webapp.WebAppContext.doStop(WebA ppContext.java:505)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerCollection.doStop (HandlerCollection.java:169)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerCollection.doStop (HandlerCollection.java:169)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.handler.DebugHandler.doStop(Debu gHandler.java:135)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.Server.doStop(Server.java:283)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.Server$ShutdownHookThread.run(Se rver.java:561)


** END NESTED EXCEPTION **


Attempted reconnect 3 times. Giving up.
at com.mysql.jdbc.SQLError.createSQLException(SQLErro r.java:888)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2997)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 555)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:285)
at java.sql.DriverManager.getConnection(DriverManager .java:582)
at java.sql.DriverManager.getConnection(DriverManager .java:154)
at org.apache.commons.dbcp.DriverManagerConnectionFac tory.createConnection(DriverManagerConnectionFacto ry.java:68)
at com.zimbra.cs.db.DbPool$ZimbraConnectionFactory.cr eateConnection(DbPool.java:239)
at org.apache.commons.dbcp.PoolableConnectionFactory. makeObject(PoolableConnectionFactory.java:294)
at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:1148)
at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:96)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 309)
... 47 more
2010-10-26 22:37:25,792 INFO [Shutdown] [] ProtocolHandler - graceful shutdown requested
2010-10-26 22:37:25,811 INFO [Shutdown] [] ProtocolHandler - graceful shutdown requested


So, it just died and now I have a box of fragments. Plz help me out here cos it's our production mail server so basically we're down as far as mail is concerned.

Greatly appreciate help

cheers,
Alex