Rspamd: A replacement for Spamassassin & Postscreen

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
phoenix
Ambassador
Ambassador
Posts: 25163
Joined: Fri Sep 12, 2014 9:56 pm

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby phoenix » Wed Nov 08, 2017 9:39 am

No, that just passes mail to an anti-virus system that's installed on your server it is not an internal component of rspamd - the first paragraph states that:

Antivirus module (new in Rspamd 1.4) provides integration with virus scanners. Currently supported are ClamAV, F-Prot, Sophos (via SAVDI) and Avira (via SAVAPI).
I deliberately avoided that to make it easier for people to install rspamd.


Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
MisterM75
Advanced member
Advanced member
Posts: 66
Joined: Sat Aug 05, 2017 7:10 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby MisterM75 » Thu Nov 09, 2017 8:58 am

Hello

It is possible that is so slow at the scan level of the mail?


vstakhov wrote:DNS sockets pool is fine. It is used as a tradeoff between security and source port randomization and performance (do not open a socket on each request). This pool is slowly rotated over time. Here are the default settings:

Code: Select all

% rspamadm configdump options.dns
*** Section options.dns ***
timeout = 1.0;
sockets = 16;
retransmits = 5;
servers = "127.0.0.1";

*** End of section options.dns ***


So it is 16 sockets per DNS server. You can modify it in `local.d/options.inc` if you want:

Code: Select all

dns {
  sockets = 4;
}


WRT SA rules: Rspamd is mostly designed to improve SA regexp rules corpus. HTML/MIME eval rules were too bad to port them back to Rspamd...


Mz
MisterM75
Advanced member
Advanced member
Posts: 66
Joined: Sat Aug 05, 2017 7:10 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby MisterM75 » Thu Nov 09, 2017 9:01 am

Hello

Code: Select all

filter = "full";


Code: Select all

/\bBreasts\b/i


I have an error when I make this change ...

phoenix wrote:Just in case anyone else wants to ban an email based on specific word content, here's an answer from the Rspamd 'issues' page from a question by our own @MisterM75:

In the local.d/multimap.conf add:

Code: Select all

reject_content {
type = "content";
filter = "full";
map = "${LOCAL_CONFDIR}/local.d/content.map";
symbol = "REJECT_CONTENT";
prefilter = true;
action = "reject";
regexp = true;
}


In the content.map: /local.d/content.map add

Code: Select all

#content filter
/\bBreasts\b/i
(blank line)

The blank line at the end is REQUIRED.

Restart rspamd, from then on any modifications to the content.map will automatically be read instantly!

This question and answer is here: https://github.com/vstakhov/rspamd/issues/1831 - any other tweaks, enhancements or configuration settings are always welcome in this thread. :)
MisterM75
Advanced member
Advanced member
Posts: 66
Joined: Sat Aug 05, 2017 7:10 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby MisterM75 » Sun Nov 12, 2017 8:05 am

Hello
When we check and send messages in junk mail knowing that it is spam, is it possible to add this function to RSPAMD?
Because at the moment, when we do it, we do it for the antispam of Zimbra and not RSPAMD ...
I hope you understood me ...
Yours truly
Mz
MisterM75
Advanced member
Advanced member
Posts: 66
Joined: Sat Aug 05, 2017 7:10 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby MisterM75 » Sun Nov 12, 2017 8:26 am

Hello my friend
What did you provide?
This sripts?

Just replace it?

Code: Select all

 #!/bin/bash
 #
 # ***** BEGIN LICENSE BLOCK *****
 # Zimbra Collaboration Suite Server
 # Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2012, 2013, 2014, 2015, 2016 Synacor, Inc.
 #
 # This program is free software: you can redistribute it and/or modify it under
 # the terms of the GNU General Public License as published by the Free Software Foundation,
 # version 2 of the License.
 #
 # This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
 # without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 # See the GNU General Public License for more details.
 # You should have received a copy of the GNU General Public License along with this program.
 # If not, see < gnu dot org /licenses/>.
 # ***** END LICENSE BLOCK *****
 #
 
 # This section trains the system ham/spam accounts
 #
 autoTrainSystem() {   
 
 # This is the section for extracting the email to a
 # couple of temp directories for spam & ham
  timestampit "Starting spam/ham extraction from system accounts."
  spamdir=`mktemp -d -t spam.XXXXXXX` || exit 1
  hamdir=`mktemp -d -t ham.XXXXXXX` || exit 1
  /opt/zimbra/libexec/zmspamextract ${spam_account} -o ${spamdir}
  /opt/zimbra/libexec/zmspamextract ${ham_account} -o ${hamdir}
  timestampit "Finished extracting spam/ham from system accounts."
 
 # This is the actual section for rspamd training
  timestampit "Starting rspamd system accounts training."
 
 # Let's do a test here to see if rspamc is doing it's thing!
 # List some stats before training
 # the passwords for these rspamc commands need to be changed for your server
 
   timestampit "List rspam stats before training."
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword stat
 
 # do the spam directory
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_spam  ${spamdir}
 
 # do the ham directory
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_ham  ${hamdir}
 
 # List some stats after training
   timestampit "List rspam stats after training."
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword stat
   timestampit "Finished rspamd training."
 # End of the rspamd training section for system ham/spam accounts
 
  /bin/rm -rf ${spamdir} ${hamdir}
 
 }
 
 # The following is the section that trains rspamd for the user $FOLDER (ham or spam)
 #
 trainAccountFolder() {
 
 timestampit  "Starting rspamd user accounts training"
  tempdir=`mktmpdir ${MODE}`
  if [ "x${MODE}" = "xspam" ]; then
    FOLDER=${FOLDER:=junk}
  elif [ "x${MODE}" = "xham" ]; then
    FOLDER=${FOLDER:=inbox}
  fi
 
 # extract the user ham/spam and train rspamd
 timestampit  "Starting rspamd $MODE training for $USER using folder $FOLDER"
  /opt/zimbra/libexec/zmspamextract -r -m $USER -o ${tempdir} -q in:${FOLDER}
   
  if [ "x${MODE}" = "xspam" ]; then
    /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_spam ${tempdir} || exit 1
    FOLDER=${FOLDER:=junk}
  elif [ "x${MODE}" = "xham" ]; then
    /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_ham ${tempdir} || exit 1
    FOLDER=${FOLDER:=inbox}
   fi
 timestampit  "Finished rspamd $MODE training for $USER using folder $FOLDER"
 
  /bin/rm -rf ${tempdir}
 
 }

 mktmpdir() {
  mktemp -d "${zmtrainsa_tmp_directory:-${zimbra_tmp_directory}}/rspamd.$$.$1.XXXXXX" || exit 1
 }
 
 timestampit() {
   SIMPLE_DATE=`date +%Y%m%d%H%M%S`
   echo "$SIMPLE_DATE $1"
 }
 
 usage() {
   echo "Usage: $0 <user> <spam|ham> [folder]"
   exit 1
 }
 
 if [ x`whoami` != xzimbra ]; then
     echo Error: must be run as zimbra user
   exit 1
 fi
 
 source `dirname $0`/zmshutil || exit 1
 zmsetvars
 
 amavis_dspam_enabled=`/opt/zimbra/bin/zmprov -l gs ${zimbra_server_hostname} zimbraAmavisDSPAMEnabled | grep zimbraAmavisDSPAMEnabled: | awk  '{print $2}'`
 amavis_dspam_enabled=$(echo $amavis_dspam_enabled | tr A-Z a-z)
 antispam_mysql_enabled=$(echo $antispam_mysql_enabled | tr A-Z a-z)
 zmtrainsa_cleanup_host=$(echo $zmtrainsa_cleanup_host | tr A-Z a-z)
 
 if [ "x${zimbra_spam_externalIsSpamAccount}" = "x" ]; then
   spam_account="-s"
 else
   spam_account="-m ${zimbra_spam_externalIsSpamAccount}"
 fi
 
 if [ "x${zimbra_spam_externalIsNotSpamAccount}" = "x" ]; then
   ham_account="-n"
 else
   ham_account="-m ${zimbra_spam_externalIsNotSpamAccount}"
 fi
 
 # Set db_path
 if [ x"$antispam_mysql_enabled" = "xtrue" ]; then
   db_path='/opt/zimbra/data/amavisd/mysql/data'
 else
   db_path='/opt/zimbra/data/amavisd/.spamassassin'
 fi
 
 # No argument mode uses zmspamextract for auto-training.
 if [ x$1 = "x" ]; then
   autoTrainSystem
   exit
 fi
 
 if [ x$1 = "x--cleanup" ]; then
   if [ x${zmtrainsa_cleanup_host} = "xtrue" ]; then
     timestampit "Starting spam/ham cleanup"
     mydir=`mktemp -d -t cleanup.XXXXXX` || exit 1
     /opt/zimbra/libexec/zmspamextract ${spam_account} -o ${mydir} -d
     /opt/zimbra/libexec/zmspamextract ${ham_account} -o ${mydir} -d
     /bin/rm -rf ${mydir}
     timestampit "Finished spam/ham cleanup"
   else
     timestampit "Cleanup skipped: $zimbra_server_hostname is not a spam/ham cleanup host."
  fi
   exit
 fi
 
 USER=$1
 MODE=`echo $2 | tr A-Z a-z`
 FOLDER=$3
 
 if [ "x${MODE}" != "xspam" -a "x${MODE}" != "xham" ]; then
   usage
 fi
 
 if [ "x${USER}" = "x" ]; then
  usage
 fi
 
 trainAccountFolder
 
 exit 0


Ha yes, great ???

Mz
MisterM75
Advanced member
Advanced member
Posts: 66
Joined: Sat Aug 05, 2017 7:10 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby MisterM75 » Sun Nov 12, 2017 8:29 am

MisterM75 wrote:Hello my friend
What did you provide?
This sripts?

Just replace it?

Code: Select all

 #!/bin/bash
 #
 # ***** BEGIN LICENSE BLOCK *****
 # Zimbra Collaboration Suite Server
 # Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2012, 2013, 2014, 2015, 2016 Synacor, Inc.
 #
 # This program is free software: you can redistribute it and/or modify it under
 # the terms of the GNU General Public License as published by the Free Software Foundation,
 # version 2 of the License.
 #
 # This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
 # without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 # See the GNU General Public License for more details.
 # You should have received a copy of the GNU General Public License along with this program.
 # If not, see < gnu dot org /licenses/>.
 # ***** END LICENSE BLOCK *****
 #
 
 # This section trains the system ham/spam accounts
 #
 autoTrainSystem() {   
 
 # This is the section for extracting the email to a
 # couple of temp directories for spam & ham
  timestampit "Starting spam/ham extraction from system accounts."
  spamdir=`mktemp -d -t spam.XXXXXXX` || exit 1
  hamdir=`mktemp -d -t ham.XXXXXXX` || exit 1
  /opt/zimbra/libexec/zmspamextract ${spam_account} -o ${spamdir}
  /opt/zimbra/libexec/zmspamextract ${ham_account} -o ${hamdir}
  timestampit "Finished extracting spam/ham from system accounts."
 
 # This is the actual section for rspamd training
  timestampit "Starting rspamd system accounts training."
 
 # Let's do a test here to see if rspamc is doing it's thing!
 # List some stats before training
 # the passwords for these rspamc commands need to be changed for your server
 
   timestampit "List rspam stats before training."
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword stat
 
 # do the spam directory
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_spam  ${spamdir}
 
 # do the ham directory
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_ham  ${hamdir}
 
 # List some stats after training
   timestampit "List rspam stats after training."
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword stat
   timestampit "Finished rspamd training."
 # End of the rspamd training section for system ham/spam accounts
 
  /bin/rm -rf ${spamdir} ${hamdir}
 
 }
 
 # The following is the section that trains rspamd for the user $FOLDER (ham or spam)
 #
 trainAccountFolder() {
 
 timestampit  "Starting rspamd user accounts training"
  tempdir=`mktmpdir ${MODE}`
  if [ "x${MODE}" = "xspam" ]; then
    FOLDER=${FOLDER:=junk}
  elif [ "x${MODE}" = "xham" ]; then
    FOLDER=${FOLDER:=inbox}
  fi
 
 # extract the user ham/spam and train rspamd
 timestampit  "Starting rspamd $MODE training for $USER using folder $FOLDER"
  /opt/zimbra/libexec/zmspamextract -r -m $USER -o ${tempdir} -q in:${FOLDER}
   
  if [ "x${MODE}" = "xspam" ]; then
    /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_spam ${tempdir} || exit 1
    FOLDER=${FOLDER:=junk}
  elif [ "x${MODE}" = "xham" ]; then
    /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_ham ${tempdir} || exit 1
    FOLDER=${FOLDER:=inbox}
   fi
 timestampit  "Finished rspamd $MODE training for $USER using folder $FOLDER"
 
  /bin/rm -rf ${tempdir}
 
 }

 mktmpdir() {
  mktemp -d "${zmtrainsa_tmp_directory:-${zimbra_tmp_directory}}/rspamd.$$.$1.XXXXXX" || exit 1
 }
 
 timestampit() {
   SIMPLE_DATE=`date +%Y%m%d%H%M%S`
   echo "$SIMPLE_DATE $1"
 }
 
 usage() {
   echo "Usage: $0 <user> <spam|ham> [folder]"
   exit 1
 }
 
 if [ x`whoami` != xzimbra ]; then
     echo Error: must be run as zimbra user
   exit 1
 fi
 
 source `dirname $0`/zmshutil || exit 1
 zmsetvars
 
 amavis_dspam_enabled=`/opt/zimbra/bin/zmprov -l gs ${zimbra_server_hostname} zimbraAmavisDSPAMEnabled | grep zimbraAmavisDSPAMEnabled: | awk  '{print $2}'`
 amavis_dspam_enabled=$(echo $amavis_dspam_enabled | tr A-Z a-z)
 antispam_mysql_enabled=$(echo $antispam_mysql_enabled | tr A-Z a-z)
 zmtrainsa_cleanup_host=$(echo $zmtrainsa_cleanup_host | tr A-Z a-z)
 
 if [ "x${zimbra_spam_externalIsSpamAccount}" = "x" ]; then
   spam_account="-s"
 else
   spam_account="-m ${zimbra_spam_externalIsSpamAccount}"
 fi
 
 if [ "x${zimbra_spam_externalIsNotSpamAccount}" = "x" ]; then
   ham_account="-n"
 else
   ham_account="-m ${zimbra_spam_externalIsNotSpamAccount}"
 fi
 
 # Set db_path
 if [ x"$antispam_mysql_enabled" = "xtrue" ]; then
   db_path='/opt/zimbra/data/amavisd/mysql/data'
 else
   db_path='/opt/zimbra/data/amavisd/.spamassassin'
 fi
 
 # No argument mode uses zmspamextract for auto-training.
 if [ x$1 = "x" ]; then
   autoTrainSystem
   exit
 fi
 
 if [ x$1 = "x--cleanup" ]; then
   if [ x${zmtrainsa_cleanup_host} = "xtrue" ]; then
     timestampit "Starting spam/ham cleanup"
     mydir=`mktemp -d -t cleanup.XXXXXX` || exit 1
     /opt/zimbra/libexec/zmspamextract ${spam_account} -o ${mydir} -d
     /opt/zimbra/libexec/zmspamextract ${ham_account} -o ${mydir} -d
     /bin/rm -rf ${mydir}
     timestampit "Finished spam/ham cleanup"
   else
     timestampit "Cleanup skipped: $zimbra_server_hostname is not a spam/ham cleanup host."
  fi
   exit
 fi
 
 USER=$1
 MODE=`echo $2 | tr A-Z a-z`
 FOLDER=$3
 
 if [ "x${MODE}" != "xspam" -a "x${MODE}" != "xham" ]; then
   usage
 fi
 
 if [ "x${USER}" = "x" ]; then
  usage
 fi
 
 trainAccountFolder
 
 exit 0


Ha yes, great ???

Mz



Which file to replace???????

MZ
phoenix
Ambassador
Ambassador
Posts: 25163
Joined: Fri Sep 12, 2014 9:56 pm

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby phoenix » Sun Nov 12, 2017 8:29 am

MisterM75 wrote:When we check and send messages in junk mail knowing that it is spam, is it possible to add this function to RSPAMD?
Because at the moment, when we do it, we do it for the antispam of Zimbra and not RSPAMD ...
If you are using the (modified) zmtrainsa script that I've provided then it trains the rspamd system by dumping the Junk folder spam to a temp folder then uses rspamc to train the rspamd anti-spam system.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
MisterM75
Advanced member
Advanced member
Posts: 66
Joined: Sat Aug 05, 2017 7:10 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby MisterM75 » Sun Nov 12, 2017 9:43 am

Hello Phoenix

phoenix wrote:
MisterM75 wrote:Hello my friend
What did you provide?
This sripts?

Just replace it?



Ha yes, great ???

Mz
Yes, that's the script and it work well. You'll also see some statistics in the log file showing the number of spam etc. - I did that just to check everything is OK but if you don't want that just remove or comment out those statements. I hope you kept a copy of your original script. ;) You will need to replace the zmtrainsa script after each upgrade and I find it easier to do that rather than modify the cron job.


We agree that when I'm in Outlook, is that when it's junk, is I click on the button that allows me to send to the unwanted directory, it sends information to RSPAMD, that's it, if I understood your sentence ???

This cron spot, I change it or ????

Code: Select all

sh -x /opt/zimbra/bin/zmtrainsa_test2 &>>/opt/zimbra/log/zmtrainsa_test.log  <-- check the output to see if it's working correctly.


Mz
phoenix
Ambassador
Ambassador
Posts: 25163
Joined: Fri Sep 12, 2014 9:56 pm

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby phoenix » Sun Nov 12, 2017 9:50 am

MisterM75 wrote:Hello my friend
What did you provide?
This sripts?

Just replace it?



Ha yes, great ???

Mz
Yes, that's the script and it work well. You'll also see some statistics in the log file showing the number of spam etc. - I did that just to check everything is OK but if you don't want that just remove or comment out those statements. I hope you kept a copy of your original script. ;) You will need to replace the zmtrainsa script after each upgrade and I find it easier to do that rather than modify the cron job.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
User avatar
JDunphy
Advanced member
Advanced member
Posts: 133
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: Release 8.7.9_GA_1794.RHEL6_64_2017
Contact:

Re: Rspamd: A replacement for Spamassassin & Postscreen

Postby JDunphy » Mon Nov 20, 2017 5:26 pm

Just a heads up for others in case anyone is experiencing high load factors. Over the past week after a month of it working flawlessly, I have been fighting CPU alerts with my test machine with zimbra + rspamd. Rspamd was pegging the cpu for 2+ hours based on 90% threshold alerts and would get an email alert (probably 3-4 over the past 10 days). When I logged in, rspamd was running with 97-99% CPU. I tried various things including restarting rspamd which helped for a few days. Decreased the pool of sockets for lookups, etc. Today, I went a little further and restarted zimbra and rspamd... That still left the load factor at about 1 so I looked at my zimbra crontab and changed updating zmtrainsa (Bill's updated script) to train the bayes from every 5 minutes to 1 time per day. Load factor dropped to 0.1 so hopefully my over training stopped the problem so this appears to be self induced by me.

I still have not created any custom filters or rules but false positives in the default configuration were too high for our email mix, so I increased add_header to 8.0. Our default configuration continues to have problems with mail from Schwab, golf digest, NY Times, and some other mailing lists we have subscribed. For the email accounts we have going through this test server we are seeing about 8-10 false positives per day and about 20-25 false negatives. The corresponding SA which is heavily tuned and modified has 0 false positives and 1 false negative every 3-4 days for the same email. The rspamd bayes training has been interesting. For somethings it works really really fast but for others - we train the same message every day. We have 2 people assigned to this test project going through their SA and Rspamd zimbra accounts each day. That is about as accurate as we know how to do this trial. Rspamd appears to rely more on ip and uri reputation then our SA spam configuration is our belief. They both are fed the same email from our front end with some custom milters we wrote... so its behavior is like postscreen in that regard.

I am not worried at this time about the false positives and false negatives as once we begin to focus on rules, that should go away. Much of the initial effort has been to gain some operational experience with it. Our spam is very targeted and business focused. For example, rspamd will penalize a single URL (odd count of URI's) which some email signatures were tipping email to false positives. Each spam system has their share of oddities so not trying to be critical here. Determining the proper score for those rules vs ham/spam in an automatic way should help when that is completed with rspamd. On the other side, SA after 8 months, just got their's working again this past Sunday night so there can be challenges in building default rule sets and appropriate scores.

Another oddities is about ipv6.

When rspamd was running with 99% cpu, I observed an associated outgoing ipv6 spike on the test machine in the same 2 hour time interval... just a small spike for a few minutes so no way to know if that was related. I didn't investigate further but assumed resolver. I have also noticed that SYN Flooding on port 53 with sending cookies messages from time to time. Don't know if they are related to rspamd but we have seen some clever methods with spam to punish sites doing blacklist lookups with a large number of NS records to what I assume is tar pitting services. We have not seen that on the SA zimbra instance given the same email and running the same version of bind and configuration as the rspamd test machine. I don't know if rspamd even follows /etc/resolv.conf and points to our bind resolver. There are no incoming connections allowed on this test server other than from our MX and our browsers to zimbra. We have a host firewall to enforce ip access via src addresses for any connections. ipv6 is enabled on the interface but we have the firewall blocking all incoming connections and do not have zimbra configured to listen on ipv6 addresses.

Return to “Administrators”

Who is online

Users browsing this forum: Bing [Bot] and 17 guests