[Solved] Clamd failes and subject of messages shows up as Unchecked every few days

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
davidkillingsworth
Advanced member
Advanced member
Posts: 196
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3829.UBUNTU14.64 -Patch 1

[Solved] Clamd failes and subject of messages shows up as Unchecked every few days

Postby davidkillingsworth » Tue Jul 30, 2019 7:22 am

Server:

Code: Select all

Release 8.8.12.GA.3794.UBUNTU14.64 UBUNTU14_64 FOSS edition, Patch 8.8.12_P4.


I just updated today from Patch 2 to Patch 4.

Clamd version (from freshclam.log):

Code: Select all

WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.99.4 Recommended version: 0.101.2


Every few days, I keep getting the situation where messages are delivered with "Unchecked" in their subject line.

I can manually restart zimbra with a "zontrol restart" but it comes back every few days.

Today, I decided to take a look at what might be causing this and found this in the log when the zimbra services start

Code: Select all

Jul 30 11:51:30 mail clamd[30211]: TCP: Bound to [127.0.0.1]:3310
Jul 30 11:51:30 mail clamd[30211]: TCP: Setting connection queue length to 200
Jul 30 11:51:30 mail clamd[30211]: LOCAL: Unix socket file /opt/zimbra/data/clamav/clamav.sock
Jul 30 11:51:30 mail clamd[30211]: LOCAL: Setting connection queue length to 200
Jul 30 11:51:30 mail clamd[30211]: daemonize() failed: Cannot allocate memory
Jul 30 11:51:30 mail clamd[30211]: Socket file removed.
Jul 30 11:51:30 mail amavis[30041]: (30041-01) (!)connect to /opt/zimbra/data/clamav/clamav.sock failed, attempt #1: Can't connect to a UNIX socket /opt/zimbra/data/clamav/clamav.sock: No such file or directory
Jul 30 11:51:30 mail amavis[30041]: (30041-01) (!)ClamAV-clamd av-scanner FAILED: run_av error: Too many retries to talk to /opt/zimbra/data/clamav/clamav.sock (All attempts (1) failed connecting to /opt/zimbra/data/clamav/clamav.sock) at (eval 148) line 613.\n
Jul 30 11:51:30 mail amavis[30041]: (30041-01) (!)WARN: all primary virus scanners failed, considering backups
Jul 30 11:51:30 mail amavis[30041]: (30041-01) (!!)AV: ALL VIRUS SCANNERS FAILED


When I try to manually check the services, I see the folowing, which doesn't make any sense.

Code: Select all

zimbra@mail:~/data/clamav$ zmcontrol status
Host mail.myhost.com
   amavis                  Running
   antispam                Running
   antivirus               Running
   dnscache                Running
   ldap                    Running
   logger                  Running
   mailbox                 Running
   memcached               Running
   mta                     Running
   opendkim                Running
   proxy                   Running
   service webapp          Running
   snmp                    Running
   spell                   Running
   stats                   Running
   zimbra webapp           Running
   zimbraAdmin webapp      Running
   zimlet webapp           Running
   zmconfigd               Running
zimbra@mail:~/data/clamav$ zmantivirusctl stop
Stopping clamd...done.
Stopping freshclam...done.
zimbra@mail:~/data/clamav$ zmantivirusctl start
Starting amavisd-mc...amavisd-mc is already running.
Starting amavisd...amavisd is already running.
Starting clamd...failed.
Starting freshclam...done.
zimbra@mail:~/data/clamav$ zmantivirusctl restart
Stopping amavisd... done.
Stopping amavisd-mc... done.
Starting amavisd-mc...done.
Starting amavisd...done.
Stopping clamd...done.
Starting clamd...failed.
Stopping freshclam...done.
Starting freshclam...done.
zimbra@mail:~/data/clamav$ zmclamdctl status
clamd is running.
zimbra@mail:~/data/clamav$ zmclamdctl stop 
Stopping clamd...done.
zimbra@mail:~/data/clamav$ zmclamdctl start
Starting clamd...failed.
zimbra@mail:~/data/clamav$ zmantivirusctl status
antivirus is running
zimbra@mail:~/data/clamav$ zmclamdctl status
clamd is running.


Clamd fails to start, but then when I check it's status, it is running.

Any idea what's going on?

I did find that I had some duplicates in /opt/zimbra/data/clamav/db/

Code: Select all

zimbra@mail:~/data/clamav/db$ ls -al
total 479508
drwxr-xr-x 10 zimbra zimbra      4096 Jul 30 12:16 .
drwxrwxr-x  4 zimbra zimbra      4096 Jul 30 12:16 ..
-rw-r-----  1 zimbra zimbra   1013248 Jan  2  2019 bytecode.cld
-rw-r-----  1 zimbra zimbra    207879 Jul 30 11:52 bytecode.cvd
drwxr-x---  2 zimbra zimbra      4096 May 21  2016 clamav-067d7d74e7db25496b87dbf761186fe1.tmp
drwxr-x---  2 zimbra zimbra      4096 Nov  9  2016 clamav-278a82d9b2c3fd7be2d4619e1652882b.tmp
drwxr-x---  2 zimbra zimbra      4096 Nov  9  2016 clamav-4d234370c55f3805833208b2e6a4870c.tmp
drwxr-x---  2 zimbra zimbra      4096 Nov  9  2016 clamav-4e0ab5775afb14c96bf117438cd53466.tmp
drwxr-x---  2 zimbra zimbra      4096 Nov 13  2016 clamav-60e9d3622749eee696cda4a47066900e.tmp
drwxr-x---  2 zimbra zimbra      4096 May 13  2016 clamav-79abe18ba0ab500e60858078dd42eb79.tmp
drwxr-x---  2 zimbra zimbra      4096 Jun  1  2016 clamav-97fe75be43aacdf44e1b5a983578b096.tmp
drwxr-x---  2 zimbra zimbra      4096 May 21  2016 clamav-c4caf97b60239b9a82cc51161cb5398b.tmp
-rw-r-----  1 zimbra zimbra 136675328 Jul 29 20:02 daily.cld
-rw-r-----  1 zimbra zimbra  45067320 Jul 30 11:50 daily.cvd
-rw-r-----  1 zimbra zimbra 307499008 Jul 30 11:50 main.cld
-rw-------  1 zimbra zimbra      2548 Jul 30 12:16 mirrors.dat


I ran the following:

Code: Select all

cd /opt/zimbra/data/clamav/db/
mv *.* /tmp/clamavdbback/
/opt/zimbra/common/bin/freshclam --config-file=/opt/zimbra/conf/freshclam.conf


This cleaned out duplicates and old .tmp files in the db folder that I had and this is the result of what's there now.

Code: Select all

zimbra@mail:~/data/clamav/db$ ls -al
total 159532
drwxr-xr-x 2 zimbra zimbra      4096 Jul 30 14:16 .
drwxrwxr-x 4 zimbra zimbra      4096 Jul 30 13:18 ..
-rw-r----- 1 zimbra zimbra    207879 Jul 30 12:48 bytecode.cvd
-rw-r----- 1 zimbra zimbra  45067320 Jul 30 12:46 daily.cvd
-rw-r----- 1 zimbra zimbra 117892267 Jul 30 12:46 main.cvd
-rw------- 1 zimbra zimbra        52 Jul 30 14:16 mirrors.dat


I still get the clamd start failures though.

I will wait a few days to see if the unchecked issue comes back.

Any suggestions on what the problems are with the lock file?
Last edited by davidkillingsworth on Tue Jul 30, 2019 11:05 am, edited 1 time in total.


User avatar
DualBoot
Elite member
Elite member
Posts: 1073
Joined: Mon Apr 18, 2016 8:18 pm
Location: Earth
ZCS/ZD Version: ZCS FLOSS - 8.7.11 Mutli servers

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Postby DualBoot » Tue Jul 30, 2019 9:16 am

Hello,

related to the log you provided, it is a problem of memory :
Jul 30 11:51:30 mail clamd[30211]: daemonize() failed: Cannot allocate memory


Regards,
davidkillingsworth
Advanced member
Advanced member
Posts: 196
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3829.UBUNTU14.64 -Patch 1

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Postby davidkillingsworth » Tue Jul 30, 2019 10:06 am

DualBoot wrote:Hello,

related to the log you provided, it is a problem of memory :
Jul 30 11:51:30 mail clamd[30211]: daemonize() failed: Cannot allocate memory


Regards,


Thanks for pointing that out. This is a VPS with less than 10 users on it. The VPS has 4GB of memory.

I see the following:

Code: Select all

zimbra@mail:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          3952       3716        236          0         11        113
-/+ buffers/cache:       3591        361
Swap:            0          0          0


Code: Select all

top - 18:04:04 up 1 day,  7:27,  1 user,  load average: 0.56, 0.56, 0.58
Tasks: 185 total,   2 running, 183 sleeping,   0 stopped,   0 zombie
%Cpu(s): 57.1 us, 11.7 sy,  0.0 ni, 30.9 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem:   4047820 total,  3768496 used,   279324 free,    12432 buffers
KiB Swap:        0 total,        0 used,        0 free.   116900 cached Mem


Is it a case of memory running out after a certain number of days? Any way to mitigate other than paying for more memory with the VPS provider?

Thanks,
David
phoenix
Ambassador
Ambassador
Posts: 26244
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Postby phoenix » Tue Jul 30, 2019 10:20 am

Your 4GB is below the recommended minimum of 8GB.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
davidkillingsworth
Advanced member
Advanced member
Posts: 196
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3829.UBUNTU14.64 -Patch 1

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Postby davidkillingsworth » Tue Jul 30, 2019 11:04 am

phoenix wrote:Your 4GB is below the recommended minimum of 8GB


Got it. Will bump it up then.

Thanks for the responses.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 459
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 8.7.11_P12 RHEL6 Network Edition
Contact:

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Postby JDunphy » Tue Jul 30, 2019 6:39 pm

davidkillingsworth wrote:Tasks: 185 total, 2 running, 183 sleeping, 0 stopped, 0 zombie
%Cpu(s): 57.1 us, 11.7 sy, 0.0 ni, 30.9 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 4047820 total, 3768496 used, 279324 free, 12432 buffers
KiB Swap: 0 total, 0 used, 0 free. 116900 cached Mem[/code]

Is it a case of memory running out after a certain number of days? Any way to mitigate other than paying for more memory with the VPS provider?

This is why you still want some swap. Add a simple swap file is more than enough so that a few pages can reside on there from time to time during peak shortfalls. Kernel will move least recently used pages out which is much faster than accesses back through the fs for something aborting for all its pages again. Just enough to ride through some memory peaks. Long term, more memory is probably warranted but you need to watch the server with vmstat, etc to know for sure. A swap file should do the trick given you probably don't want to re-partition for a few memory short falls. You could also tune down some of the big memory hogs but that is a hard recommendation to make without watching in more detail.You can tell my age because I always configure a small amount of swap (reliability) vs (pure performance).

Note: you can add a swap file without a reboot or restarting zimbra so it fairly quick for a production machine. If you need it to remain, put it in your /etc/fstab for future reboots.

Ref: https://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/
Ref: https://www.linux.com/news/all-about-linux-swap-space
davidkillingsworth
Advanced member
Advanced member
Posts: 196
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3829.UBUNTU14.64 -Patch 1

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Postby davidkillingsworth » Wed Jul 31, 2019 9:55 am

JDunphy wrote:
davidkillingsworth wrote:Tasks: 185 total, 2 running, 183 sleeping, 0 stopped, 0 zombie
%Cpu(s): 57.1 us, 11.7 sy, 0.0 ni, 30.9 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 4047820 total, 3768496 used, 279324 free, 12432 buffers
KiB Swap: 0 total, 0 used, 0 free. 116900 cached Mem[/code]

Is it a case of memory running out after a certain number of days? Any way to mitigate other than paying for more memory with the VPS provider?

This is why you still want some swap. Add a simple swap file is more than enough so that a few pages can reside on there from time to time during peak shortfalls. Kernel will move least recently used pages out which is much faster than accesses back through the fs for something aborting for all its pages again. Just enough to ride through some memory peaks. Long term, more memory is probably warranted but you need to watch the server with vmstat, etc to know for sure. A swap file should do the trick given you probably don't want to re-partition for a few memory short falls. You could also tune down some of the big memory hogs but that is a hard recommendation to make without watching in more detail.You can tell my age because I always configure a small amount of swap (reliability) vs (pure performance).

Note: you can add a swap file without a reboot or restarting zimbra so it fairly quick for a production machine. If you need it to remain, put it in your /etc/fstab for future reboots.

Ref: https://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/
Ref: https://www.linux.com/news/all-about-linux-swap-space


Interesting. Thanks for this.

I have checked about adding physical memory and my VPS provider only allows upgrades of the whole VPS size. I don't need more CPU or disk space, so it's a case of spending a bunch of extra money just for 4GB more RAM. That sort of makes running my own zimbra server a little less practical. However, having the knowledge of being able to add a swap file effectively, make that more feasible.

I read both articles, but what would you recommend in terms of size for a standard default Ubuntu \ Zimbra installation? Would you mind writing up a little guide? I think this could help for those of us that are running servers in environments that are limited by how much physical memory we can assign, such as VPS environments.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 459
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 8.7.11_P12 RHEL6 Network Edition
Contact:

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Postby JDunphy » Wed Jul 31, 2019 3:14 pm

davidkillingsworth wrote:I read both articles, but what would you recommend in terms of size for a standard default Ubuntu \ Zimbra installation? Would you mind writing up a little guide? I think this could help for those of us that are running servers in environments that are limited by how much physical memory we can assign, such as VPS environments.

Hi David,

The old school answer was 2x memory but I have not done that in many years given how big memory sizes have grown. I run memory size for my swap size if I have plenty of disk. So 4GB would be plenty for you and then watch the system and your logs for memory short falls and any disk bottlenecks. If your disk controller has enough bandwidth and your disks are keeping up than I wouldn't worry too much about the odd swap usage. If you are still getting memory allocation fails than it is time to tune memory usage down or add more memory. If you are not happy with your search responsiveness with the web clients than you might need to get a little more creative to find that happy mix for your usage. Given the cost of ram, many admin's will double the ram sizes vs invest in the time and effort to tune. For some VPS providers, that generally doubles the cost of your hosting so I understand where you are coming from.

This is a better link explaining how to size them which is about 20% of memory these days and provides a little more background including how to size swap usage.

ref: https://www.redhat.com/en/blog/do-we-really-need-swap-modern-systems

Jim

Return to “Administrators”

Who is online

Users browsing this forum: Google [Bot], jasggomes and 24 guests