New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Ask questions about your setup or get help installing ZCS server (ZD section below).
andrey.ivanov
Posts: 26
Joined: Wed Aug 08, 2018 8:44 am

New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Postby andrey.ivanov » Fri Aug 24, 2018 2:17 pm

We have two new types of exceptions in mailboxd.log after upgrade 8.8.9_P2 -> 8.8.9_P3 (NE, CentOS7). One of them is described in the forum thread viewtopic.php?f=15&t=64708 and concerns exceptions during SearchRequests. To find them search the file mailboxd.log for the keyword isSortByReadFlag. We have about 1000 of these per day.

The second new type of exceptions concerns the index code, search for the keyword ZimbraHit to find them:

Code: Select all

grep 'index - Caught ServiceException trying to compare ZimbraHit' /opt/zimbra/log/mailbox.log


We have about 150000 of those per working day. They look like this :

Code: Select all

2018-08-24 10:01:33,636 INFO  [qtp1286783232-154439:https:https://webmail.server.com/service/soap/SearchRequest] [name=some.user@my.zimbra.server;mid=10891;oip=123.45.67.89;port=56774;ua=ZimbraWebClient - FF61 (Win)/8.8.9_GA_3019;soapId=1aa2d45c;] index - Caught ServiceException trying to compare ZimbraHit {"s":2956,"d":1528962759000,"l":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:2","cid":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:-12504","f":"r","rev":55606,"id":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:12504","su":"Date de master M4S","fr":"Ceci est un mail automatique, veuillez ne ...","cm":true,"e":[{"a":"internal_mail@our.domain","d":"toto","p":"toto via toto","t":"f"}],"sf":"0"} to ZimbraHit {"s":9100,"d":1529062749000,"l":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:2","cid":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:12521","f":"r","rev":55706,"id":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:12520","su":"Inquiry about MSc. program","fr":"Dear Sirs: I hope this email finds you well. I am sending this to kindly ask about the masters program course language. I come from an English ...","cm":true,"e":[{"a":"some-email@somewhere.com","d":"John","p":"Jane Doe","t":"f"}],"sf":"0"}


It has appeared because of the changes in the file ./zm-mailbox/store/src/java/com/zimbra/cs/index/ZimbraHit.java at line 340 (diff from 8.8.9_P2 -> 8.8.9_P3) adding the new function protected static final int compareByReadFlag(boolean ascending, ZimbraHit lhs, ZimbraHit rhs). Part of the code generating this exception:

Code: Select all

+        try {
+            long left = getReadStatus(lhs);
+            long right = getReadStatus(rhs);
+            long result = right - left;
+            if (result > 0)
+                retVal = 1;
+            else if (result < 0)
+                retVal = -1;
+            else
+                retVal = 0;
+        } catch (ServiceException e) {
+            ZimbraLog.index.info("Caught ServiceException trying to compare ZimbraHit %s to ZimbraHit %s",
+                lhs, rhs);
+            ZimbraLog.index.debug(e);
+        }


I've opened a Zimbra ticket concerning these two new bugs in Patch 3 of 8.8.9.


User avatar
gabrieles
Advanced member
Advanced member
Posts: 87
Joined: Tue Feb 14, 2017 9:40 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Postby gabrieles » Thu Mar 07, 2019 3:10 pm

Edit, ain't solved anything, the problem is still there.

Had the same identical problem on my personal account. I thought that i can have solved it today, but wasn't.
- The strange behaviour started with ZCS 8.8.9_P2. Now i'm on 8.8.9_P8, the issue stands still
- I've found the issue when investigating slowness at the click on the mountpoint of a shared /Inbox from another account.
- Mine was the only account (out of ~50) experiencing the problem. The suspect /Inbox was shared and mounted in the identical way with other accounts, but not showing the problem.
- At each click mailbox.log produced 7221 lines of index - Caught ServiceException trying to compare ZimbraHit (9MB each time!). In the /Inbox there were exactly 7221 messages.
- Tried to reindex my mailbox and even the sharing mailbox, with no effect.
- I've got another mountpoint to the /Sent of the same account. There's absolutely no problem clicking or working on it.

Deleting and recreating the mountpoint resolved temporarily the problem.

zmmailbox -z -m gabrieles df /MyMountpoint
zmmailbox -z -m gabrieles cm /MyMountpoint sharingaccount@mydomain.com /Inbox

But some hours after, it resurfaced: when clicking that specific folder, it fills 9MB of index - Caught ServiceException trying to compare ZimbraHit . Will keep investigating.
User avatar
gabrieles
Advanced member
Advanced member
Posts: 87
Joined: Tue Feb 14, 2017 9:40 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Postby gabrieles » Fri Mar 08, 2019 11:24 am

Funny thing:
yesterday's mailbox.log was 720MB.
yesterday's mailbox.log without the Caught ServiceException trying to compare ZimbraHit lines was 22MB
andrey.ivanov
Posts: 26
Joined: Wed Aug 08, 2018 8:44 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Postby andrey.ivanov » Fri Mar 08, 2019 11:34 am

Yep. it is still not fixed in 8.8.10_P7 (i have not tested 8.8.11) though the support has confirmed the bug in August 2018 :
----
ZimbraHit error has been identified as a bug by the Dev team and it will be fixed in future release. This is an internal bug (ZCS-6022) number for the reference, which you may take a follow up any time with us by raising a case.
----
jgarat
Posts: 1
Joined: Wed Jun 12, 2019 9:50 pm

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Postby jgarat » Wed Jun 12, 2019 10:04 pm

gabrieles wrote:Edit, ain't solved anything, the problem is still there.

Had the same identical problem on my personal account. I thought that i can have solved it today, but wasn't.
- The strange behaviour started with ZCS 8.8.9_P2. Now i'm on 8.8.9_P8, the issue stands still
- I've found the issue when investigating slowness at the click on the mountpoint of a shared /Inbox from another account.
- Mine was the only account (out of ~50) experiencing the problem. The suspect /Inbox was shared and mounted in the identical way with other accounts, but not showing the problem.
- At each click mailbox.log produced 7221 lines of index - Caught ServiceException trying to compare ZimbraHit (9MB each time!). In the /Inbox there were exactly 7221 messages.
- Tried to reindex my mailbox and even the sharing mailbox, with no effect.
- I've got another mountpoint to the /Sent of the same account. There's absolutely no problem clicking or working on it.

Deleting and recreating the mountpoint resolved temporarily the problem.

zmmailbox -z -m gabrieles df /MyMountpoint
zmmailbox -z -m gabrieles cm /MyMountpoint sharingaccount@mydomain.com /Inbox

But some hours after, it resurfaced: when clicking that specific folder, it fills 9MB of index - Caught ServiceException trying to compare ZimbraHit . Will keep investigating.




Hi Gabrieles, yesterday we upgrade from Zimbra Collaboration Open Source 8.8.6 to 8.8.12 Patch 2 and we are experimenting the same problems! We use a lot Shared Inboxes.

You still have this issue? Can you share how do you resolved?

Thanks!

Juan
User avatar
gabrieles
Advanced member
Advanced member
Posts: 87
Joined: Tue Feb 14, 2017 9:40 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Postby gabrieles » Thu Jun 13, 2019 8:30 am

Apparently was caused by the shared folder sorting order.
By default the sorting order of that folder was descending by date.
In some cases, I pressed the blue dot to visualize only the unread messages.
Pressing again the blue dot SEEMS to bring back to the previous state of descending by date, but actually don't.
If you pay careful attention, not all the message are displayed and the order is a little bit strange.
It leaves the folder in a sort of stranded state that s##t a ton of exception in the logs, various exceptions for EACH message in the folder.

The solution is to force back the folder in a well defined order, sorting manually by date, descending.

Return to “Installation and Upgrade”

Who is online

Users browsing this forum: JDunphy and 8 guests