Mail lost after uncontrolled shutdown

[6233maxxer]

Yesterday we had a power loss in a datacenter and Zimbra was shut down improperly. 

After reboot the MySQL database was compromised, but the customer didn't realize it so it has been running for some hours. during this time some mails got delivered, and some did not, even if the MX server (not part of zimbra) delivered them to Zimbra! 

Here's an excerpt of a failed delivery:

Jul 8 11:48:11 ms1 postfix/smtpd[5275]: 55ED93725E7E: client=mx1.domain.com[46.1.2.3]
Jul 8 11:48:11 ms1 postfix/cleanup[5288]: 55ED93725E7E: message-id=<!&!AAAAAAAAAAAYAAAAAAAAACuGk2kwFDVMisJYrVTsoMHCgAAAEAAAAH+fmSKLP31PoQAyMp5QaJwBAAAAAA==@senderdomain.it>
Jul 8 11:48:11 ms1 postfix/smtpd[5275]: disconnect from mx1.domain.com[46.1.2.3]
Jul 8 11:48:11 ms1 postfix/qmgr[4546]: 55ED93725E7E: from=<sender@senderdomain.it>, size=21394, nrcpt=1 (queue active)
Jul 8 11:48:11 ms1 amavis[4169]: (04169-02) ESMTP::10026 /opt/zimbra/data/amavisd/tmp/amavis-20150708T114543-04169-326Z2x2W: <sender@senderdomain.it> -> <recipient@recipientdomain.it> SIZE=21394 Received: from
ms1.domain.com ([127.0.0.1]) by localhost (ms1.domain.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP for <recipient@recipientdomain.it>; Wed, 8 Jul 2015 11:48:11 +0200 (CEST)
Jul 8 11:48:11 ms1 amavis[4169]: (04169-02) Checking: Z0aLStQ8cfRN ORIGINATING/MYNETS [46.1.2.3] <sender@senderdomain.it> -> <recipient@recipientdomain.it>
Jul 8 11:48:11 ms1 postfix/dkimmilter/smtpd[5284]: connect from localhost[127.0.0.1]
Jul 8 11:48:11 ms1 postfix/dkimmilter/smtpd[5284]: 724C737264F0: client=localhost[127.0.0.1]
Jul 8 11:48:11 ms1 postfix/cleanup[5288]: 724C737264F0: message-id=<!&!AAAAAAAAAAAYAAAAAAAAACuGk2kwFDVMisJYrVTsoMHCgAAAEAAAAH+fmSKLP31PoQAyMp5QaJwBAAAAAA==@senderdomain.it>
Jul 8 11:48:11 ms1 opendkim[4350]: 724C737264F0: no signing table match for 'sender@senderdomain.it'
Jul 8 11:48:11 ms1 postfix/qmgr[4546]: 724C737264F0: from=<sender@senderdomain.it>, size=21877, nrcpt=1 (queue active)
Jul 8 11:48:11 ms1 amavis[4169]: (04169-02) FWD from <sender@senderdomain.it> -> <recipient@recipientdomain.it>,BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10030): 250 2.0.0 Ok: queued as 724C737264F0
Jul 8 11:48:11 ms1 postfix/dkimmilter/smtpd[5284]: disconnect from localhost[127.0.0.1]
Jul 8 11:48:11 ms1 amavis[4169]: (04169-02) Passed CLEAN {RelayedInternal}, ORIGINATING/MYNETS LOCAL [46.1.2.3]:48163 [31.1.2.3] <sender@senderdomain.it> -> <recipient@recipientdomain.it>, Queue-ID: 55ED93725E7E, Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAACuGk2kwFDVMisJYrVTsoMHCgAAAEAAAAH+fmSKLP31PoQAyMp5QaJwBAAAAAA==@senderdomain.it>, mail_id: Z0aLStQ8cfRN, Hits: -, size: 21394, queued_as: 724C737264F0, 157 ms
Jul 8 11:48:11 ms1 amavis[4162]: (04162-04) ESMTP::10032 /opt/zimbra/data/amavisd/tmp/amavis-20150708T114511-04162-0xDgzu2a: <sender@senderdomain.it> -> <recipient@recipientdomain.it> SIZE=21877 Received: from ms1.domain.com ([127.0.0.1]) by localhost (ms1.domain.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP for <recipient@recipientdomain.it>; Wed, 8 Jul 2015 11:48:11 +0200 (CEST)
Jul 8 11:48:11 ms1 postfix/smtp[4894]: 55ED93725E7E: to=<recipient@recipientdomain.it>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.17, delays=0.01/0/0/0.16, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10030): 250 2.0.0 Ok: queued as 724C737264F0)
Jul 8 11:48:11 ms1 postfix/qmgr[4546]: 55ED93725E7E: removed

As you can see in the bold line the mail wasn't delivered to LMTP but to 10030 (postfix again). Why that? How can I recover those mails? thanks

[6233maxxer]

hm I must correct myself, the delivery got lost in the mass of the MX queue, but was indeed delivered to LMTP. Sadly I cannot find it into zimbra web client!



Is it possible that the mail was received by mailbox server, stored to disk but NOT to MySQL?

[phoenix]

I can't really answer the question on whether it's not stored in MySQL but you could try using zmlmtpinject to see if it appears in the user's mailbox.

[6233maxxer]

thanks.

so after further investigation I indeed found the data on disk, but it's missing from mysql.

The only way to make ZWC aware of those mails is using zmlmtpinject, right? thanks again

[phoenix]

As far as I know, zmlmtpinject would be the easiest way to get those messages back into the system.