Gmail / ActiveSync delete command loop sync causing sync issues and high CPU usage on mobiles and Zimbra server

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
Gummikavalier
Posts: 3
Joined: Tue Apr 04, 2017 3:15 pm

Gmail / ActiveSync delete command loop sync causing sync issues and high CPU usage on mobiles and Zimbra server

Postby Gummikavalier » Tue Nov 26, 2019 10:29 am

Hi fellow sysadmins!

We have been troubleshooting for 6 months with NG mobile module an issue where several makes and models of Android from Nokia, Motorola, Huawei and Sony mobiles accumulate activesync delete commands into their internal queue locally on the mobile device itself. This happens with Google's default ActiveSync and Gmail implementations configured to sync emails from Exchange server instead of Google.

iPhones and Samsung Android devices are not affected. Also Outlook app by Microsoft on Android is not affected. Even SailfishOS email app configured with Exchange support is not affected. The issue really seems to be only with Gmail application coupled with ActiveSync, but it is likely that lots of people get hit by it because it is the built-in option for all new Android devices.

Currently we are running Zimbra 8.8.15 NE patch4 and latest Android versions and apps on our mobile devices. We can reproduce this issue on our system with 100% success rate on any Android device so far excluding Samsung. Samsung is good only because ActiveSync implementation or email client on Samsung does not use delete command at all.

We have a case (#00876714) open for this for the developers of Zimbra, and they have come to conclusion that the issue happens directly on the Android devices.

When left unattended for many Android client devices for longer periods of time, accumulating obsolete delete commands in every sync start causing DoS-like behaviour to the server. In addition to server CPU load, it also causes significant CPU load and battery drain on the affected mobile device, as well as numerous issues like email and calendar items not being up to date on the device. This is because the sync eventually times out or gets killed by the watchdog process of Android.

It is possible to fix this condition on the mobile device with NG modules reset sync option (full wipe is not required), or by recreating the user's account on it.

Since NG modules introduced good logging options, we have been able to keep track of the issue since. We found out that the root cause is the delete-command that happens on modern Android devices when any item is deleted from the Trash-folder (on the device). Deletions from Inbox are in fact move operations to Trash and don't cause problems until the user triggers the actual delete for them by cleaning up his/her Trash-folder on the phone.

Exception to this rule are Huawei devices (at least Honor 8) that issue delete-command directly on the items under Inbox, instead of moving them to Trash first. Because of this the problems have been most visible to our users with Huawei devices.

You can find if you are affected by the issue by checking the sync.log (/opt/zimbra/log/sync.log), and looking for any repeating delete lines with the same item id number on them:

Code: Select all

   sync - Deleting item [xxxxxx] for [username]


where [xxxxxx] is always the same item ID number. [Username] is the account of the user on whose device the issue is happening. You can also see the deviceid of Android to distinguish it from other possible devices the user has in the logs and to know what device to reset.

Reproducing the issue so that it is visible in the logs may require that you don't use Push sync on the mobile device, but it still does not prevent the issue from happening on Push enabled devices.

It is ok to see one deletion per item but not twice or hundreds of times like in our case has been. Even one repeating deletion of the same itemid number means that sync is malfunctioning on that device. You are likely to see a lots and lots of repeating items for deletion, as the phone never recovers from this state automatically, and will try to delete the same items forever and ever.

We use this command to find out any repeating delete commands in our sync.logs (perl required, also note log path and name):

Code: Select all

zfgrep -i ' sync - deleting item ' /opt/zimbra/log/sync.log.2019-11-25.gz | perl -ne 'chomp; if((/^(\S+\s+\S+).*;oip=([^;]+).*;(id=[^;]+).* sync - deleting item (.+)$/i)) { $count{my $id="$3 deleting $4"}++; $last{$id}="$1 $2"; } END { foreach(sort { $count{$a}<=>$count{$b} } keys %count) { printf("%5d %s (%s)\n",$count{$_},$_,$last{$_}) if $count{$_}>1; } }'


If you are affected by this same issue, please report it into this thread, so we can see how widespread this problem really is. So far it seems that Zimbra developers could not reproduce the issue with the same success rate on their test systems as we do, if at all.


Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 5 guests