Hello Forum,

some system facts first:
- Ubuntu 10.04 LTS,
- Virtual Machine, Running on ESXi 4.1
- 64 Bit
- 3328 MB RAM
- 4 Processors assigned
- Zimbra 7.1.1_GA_3196.UBUNTU10_64 UBUNTU10_64 FOSS edition

Successful installation without any issues. Changed the following during the installation
- Alternative HTTP port as there is an existing Apache2 on the machine
- Alternative HTTPS port
- HTTPS only mode

System starts up ok but when monitoring the log file

Code:
/opt/zimbra/log/zmconfigd.log
the following does not look like it should be that way

Code:
...
2011-06-15 11:23:47,967 zmconfigd CRITICAL [19186-rewrites]   File "/opt/zimbra/zimbramon/pylibs/state.py", line 683, in rewriteConfig     os.unlink(to) 
2011-06-15 11:23:47,969 zmconfigd CRITICAL [19186-rewrites]   File "/opt/zimbra/lib/jars/jython-2.5.1.jar/Lib/os.py", line 342, in remove     raise OSError(0, "couldn't delete file", path) 
2011-06-15 11:23:47,970 zmconfigd CRITICAL [19186-rewrites] Rewrite failed: [Errno 0] couldn't delete file: '/opt/zimbra/postfix/conf/master.cf' ([Errno 0] couldn't delete file: '/opt/zimbra/postfix/conf/master.cf')
...
2011-06-15 11:23:48,444 zmconfigd WARNING [19186-rewrites] Rewrote: /opt/zimbra/conf/amavisd.conf with mode 440 (0.08 sec)
2011-06-15 11:23:48,460 zmconfigd WARNING [19186-rewrites] Rewrote: /opt/zimbra/conf/salocal.cf with mode 440 (0.02 sec)
2011-06-15 11:23:48,710 zmconfigd ERROR [19186-postconf] Executed: /opt/zimbra/postfix/sbin/postconf -e mail_owner='postfix' returned 1 (14 - 112) (1.07 sec)
2011-06-15 11:23:49,732 zmconfigd ERROR [19186-postconf] Executed: /opt/zimbra/postfix/sbin/postconf -e bounce_notice_recipient='postmaster' returned 1 (14 - 112) (1.02 sec)
2011-06-15 11:23:50,746 zmconfigd ERROR [19186-postconf] Executed: /opt/zimbra/postfix/sbin/postconf -e content_filter='smtp-amavis:[127.0.0.1]:10024' returned 1 (14 - 112) (1.01 sec)
2011-06-15 11:23:51,757 zmconfigd ERROR [19186-postconf] Executed: /opt/zimbra/postfix/sbin/postconf -e relayhost='' returned 1 (14 - 112) (1.01 sec)
2011-06-15 11:23:52,769 zmconfigd ERROR [19186-postconf] Executed: /opt/zimbra/postfix/sbin/postconf -e smtpd_sasl_authenticated_header='no' returned 1 (14 - 112) (1.01 sec)
2011-06-15 11:23:53,782 zmconfigd ERROR [19186-postconf] Executed: /opt/zimbra/postfix/sbin/postconf -e broken_sasl_auth_clients='yes' returned 1 (14 - 112) (1.01 sec)
2011-06-15 11:23:54,796 zmconfigd ERROR [19186-postconf] Executed: /opt/zimbra/postfix/sbin/postconf -e minimal_backoff_time='300s' returned 1 (14 - 112) (1.01 sec)
I do not want to flood this whole thread with all the error messages but it looks like every postconf command failing. When leaving the system up and running for a while the following is logged repeatedly

Code:
2011-06-15 11:25:47,813 zmconfigd CRITICAL [19186-rewrites]   File "/opt/zimbra/zimbramon/pylibs/state.py", line 683, in rewriteConfig     os.unlink(to) 
2011-06-15 11:25:47,815 zmconfigd CRITICAL [19186-rewrites]   File "/opt/zimbra/lib/jars/jython-2.5.1.jar/Lib/os.py", line 342, in remove     raise OSError(0, "couldn't delete file", path) 
2011-06-15 11:25:47,816 zmconfigd CRITICAL [19186-rewrites] Rewrite failed: [Errno 0] couldn't delete file: '/opt/zimbra/postfix/conf/master.cf' ([Errno 0] couldn't delete file: '/opt/zimbra/postfix/conf/master.cf')
2011-06-15 11:25:47,818 zmconfigd WARNING [19186-MainThread] All rewrite threads completed in 0.01 sec
2011-06-15 11:25:47,819 zmconfigd WARNING [19186-MainThread] All restarts completed in 0.00 sec
2011-06-15 11:26:47,832 zmconfigd WARNING [19186-MainThread] Fetching All configs
2011-06-15 11:26:48,131 zmconfigd WARNING [19186-MainThread] All configs fetched in 0.30 seconds
2011-06-15 11:26:49,033 zmconfigd WARNING [19186-MainThread] Watchdog: service antivirus status is OK.
2011-06-15 11:26:49,190 zmconfigd CRITICAL [19186-rewrites]   File "/opt/zimbra/zimbramon/pylibs/state.py", line 683, in rewriteConfig     os.unlink(to) 
2011-06-15 11:26:49,190 zmconfigd CRITICAL [19186-rewrites]   File "/opt/zimbra/lib/jars/jython-2.5.1.jar/Lib/os.py", line 342, in remove     raise OSError(0, "couldn't delete file", path) 
2011-06-15 11:26:49,191 zmconfigd CRITICAL [19186-rewrites] Rewrite failed: [Errno 0] couldn't delete file: '/opt/zimbra/postfix/conf/master.cf' ([Errno 0] couldn't delete file: '/opt/zimbra/postfix/conf/master.cf')
When shutting down Zimbra using the regular command
Code:
service zimbra stop
a stack trace of the Java VM with the following content is created on every shutdown
Code:
zmthrdump: Requested thread dump [PID 16142] at Wed Jun 15 11:22:44 2011
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):

"DestroyJavaVM" prio=10 tid=0x00007fa5dc1f6800 nid=0x3f1b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

....

"pool-2-thread-1" prio=10 tid=0x00007fa5e4c04800 nid=0x3fbf runnable [0x00007fa5e246f000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
	- locked <0x00000000d8402b40> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:462)
	at java.net.ServerSocket.accept(ServerSocket.java:430)
	at org.jivesoftware.wildfire.filetransfer.proxy.ProxyConnectionManager$1.run(ProxyConnectionManager.java:96)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

"SocketSendingTracker" daemon prio=10 tid=0x00007fa5e4cb3000 nid=0x3fbe in Object.wait() [0x00007fa5e24b0000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000d8156788> (a org.jivesoftware.wildfire.net.SocketSendingTracker$1)
	at org.jivesoftware.wildfire.net.SocketSendingTracker$1.run(SocketSendingTracker.java:78)
	- locked <0x00000000d8156788> (a org.jivesoftware.wildfire.net.SocketSendingTracker$1)

"Secure Socket Listener" daemon prio=10 tid=0x00007fa5e4d3a800 nid=0x3fbd runnable [0x00007fa5e24f1000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
	- locked <0x00000000d8156898> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:462)
	at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:261)
	at org.jivesoftware.wildfire.net.SSLSocketAcceptThread.run(SSLSocketAcceptThread.java:148)

"Secure Socket Listener" daemon prio=10 tid=0x00007fa5e4144800 nid=0x3fb9 runnable [0x00007fa5e25f5000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
	- locked <0x00000000d816b908> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:462)
	at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:261)
	at org.jivesoftware.wildfire.net.SSLSocketAcceptThread.run(SSLSocketAcceptThread.java:148)

"Auditor" prio=10 tid=0x00007fa5e4799000 nid=0x3fb8 in Object.wait() [0x00007fa5e2636000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000d8402d88> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:509)
	- locked <0x00000000d8402d88> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:462)

"MUC cleanup" prio=10 tid=0x00007fa5e4596800 nid=0x3fb7 in Object.wait() [0x00007fa5e2677000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000d8402e50> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:509)
	- locked <0x00000000d8402e50> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:462)

"Sessions cleanup" prio=10 tid=0x00007fa5e4c19000 nid=0x3fb4 in Object.wait() [0x00007fa5e26b8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000d8404270> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:509)
	- locked <0x00000000d8404270> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:462)

"PooledByteBufferExpirer-0" daemon prio=10 tid=0x00007fa5e41bf800 nid=0x3fb2 waiting on condition [0x00007fa5e26f9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.mina.common.PooledByteBufferAllocator$Expirer.run(PooledByteBufferAllocator.java:249)

"FileLogWriter.FsyncThread" prio=10 tid=0x00007fa5e4d0b000 nid=0x3fb1 waiting on condition [0x00007fa5e273a000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at com.zimbra.cs.redolog.logger.FileLogWriter$FsyncThread.run(FileLogWriter.java:521)

....

"btpool0-5 - Acceptor1 SslSelectChannelConnector@0.0.0.0:7071" prio=10 tid=0x00007fa5e4211000 nid=0x3f4a runnable [0x00007fa5e89af000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000000d82ad268> (a sun.nio.ch.Util$2)
	- locked <0x00000000d82ad258> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000d82ad1d0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
	at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
	at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
	at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
	at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

"btpool0-4 - Acceptor0 SslSelectChannelConnector@0.0.0.0:7071" prio=10 tid=0x00007fa5e41fc800 nid=0x3f49 runnable [0x00007fa5e89f0000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000000d82acd58> (a sun.nio.ch.Util$2)
	- locked <0x00000000d82acd48> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000d82accc0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
	at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
	at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
	at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
	at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

....

"Low Memory Detector" daemon prio=10 tid=0x00007fa5e4001800 nid=0x3f37 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00000000416cc800 nid=0x3f36 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00000000416ca000 nid=0x3f35 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00000000416c7800 nid=0x3f34 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00000000416c5800 nid=0x3f33 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00000000416aa000 nid=0x3f32 in Object.wait() [0x00007fa5e96d9000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000da3070a0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
	- locked <0x00000000da3070a0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00000000416a2800 nid=0x3f31 in Object.wait() [0x00007fa5e971a000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000da020848> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked <0x00000000da020848> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x000000004169c000 nid=0x3f30 runnable 

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000041528000 nid=0x3f20 runnable 

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x000000004152a000 nid=0x3f21 runnable 

"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x000000004152c000 nid=0x3f22 runnable 

"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x000000004152d800 nid=0x3f23 runnable 

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000415bd000 nid=0x3f2b runnable 
"VM Periodic Task Thread" prio=10 tid=0x00007fa5e400d000 nid=0x3f38 waiting on condition
(Needed to shorten the stack trace - otherwise the text was to long to post)

The same happens when the system is just left running for a while (Installed yesterday so I can't really say anything yet about the periods, Looks like it happens after 6 - 8 hours). Anyhow checking the status of Zimbra after a while using the command
Code:
service zimbra status
shows the following
Code:
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Host 
	antispam                Stopped
		zmamavisdctl is not running
	antivirus               Stopped
		zmamavisdctl is not running
		zmclamdctl is not running
	ldap                    Stopped
	logger                  Stopped
		zmlogswatchctl is not running
	mailbox                 Stopped
		mysql.server is not running.
		zmmailboxdctl is not running.
	mta                     Stopped
		postfix is not running
		zmsaslauthdctl is not running
	snmp                    Stopped
		zmswatch is not running.
	spell                   Stopped
		zmapachectl is not running
	stats                   Stopped
	zmconfigd               Stopped
		zmconfigd is not running.
As said above once this happened one can also find a new Java VM stack trace in the log directory.

Even though the installation completed without any issues all of the above does not look so good. Does anybody have an idea of what is going on? Any help would be greatly appreciated.

Thanks in advance.

Sincerely

Henning Malzahn