Page 1 of 2

Cant shake bruteforce monkey

Posted: Fri Oct 19, 2018 3:59 pm
by copowpow
I am having issues combating brute-force attempts on my ZCS server.

I have fail2ban in place but it is not helping in this situation, they just come back from a different IP address. My accounts keep getting locked out.Ive blocked entire country's (brazil, indonesia, china) with ip lists from ip2location.com

The only work around is to rename the account and create a new dummy account with the old name, let it lock out and forward the email to the old re-named address. Thats the only way I can keep some of my users from getting locked out, and it seems to be getting worse as accounts that were not targets are starting to become targets.

Is there anything I can do to help with this? Re-captcha on the login screen? Change the URL of the login page (does that even help?)?

Any suggestions would help greatly as we cant seem to get a break from these botnets!

Re: Cant shake bruteforce monkey

Posted: Fri Oct 19, 2018 4:18 pm
by vavai
Hi
copowpow wrote:I am having issues combating brute-force attempts on my ZCS server.

I have fail2ban in place but it is not helping in this situation, they just come back from a different IP address. My accounts keep getting locked out.Ive blocked entire country's (brazil, indonesia, china) with ip lists from ip2location.com

The only work around is to rename the account and create a new dummy account with the old name, let it lock out and forward the email to the old re-named address. Thats the only way I can keep some of my users from getting locked out, and it seems to be getting worse as accounts that were not targets are starting to become targets.

Is there anything I can do to help with this? Re-captcha on the login screen? Change the URL of the login page (does that even help?)?

Any suggestions would help greatly as we cant seem to get a break from these botnets!


What is your main problem? Do you have some compromised account due to brute-force attack or does it simply too many attempt on SMTP connection and worry it would lead to high CPU usage?

Do you activating "failed login lockout policy" and it randomly lock some account due to failed login attempt?

I've never activate "failed login lockout policy" as it can be used by wrong people (or robot) to brute-force attack and affecting innocent account. What I do :

1. Enforce strong password
2. Activate fail2ban
3. Limit trusted network
4. Activate 2FA
5. If possible, change default port, something like disable port 25 or 587, activate other port as incoming port and submission port and adjust my cloud anti spam destination port.

Re: Cant shake bruteforce monkey

Posted: Fri Oct 19, 2018 7:05 pm
by L. Mark Stone
If you use fail2ban and a password lockout policy, just set the fail2ban threshold lower than the lockout policy.

In this way, a brute force attacker will get their IP blocked before the user's mailbox is set to Locked Out status; the legitimate user can keep using their email.

You can do the same thing with Zimbra's DoSFilter, instead of fail2ban if you wish.

Hope that helps,
Mark

Re: Cant shake bruteforce monkey

Posted: Sat Oct 20, 2018 1:29 am
by zimico
Hi Mark,
I have tested zimbra dosfilter with lower threshold and fail login policy. Dosfilter recoginzed bad ip but the account was still locked out.
Could you show me your current configuration.
Regards,
Minh.

Re: Cant shake bruteforce monkey

Posted: Sat Oct 20, 2018 3:06 pm
by L. Mark Stone
zimico wrote:Hi Mark,
I have tested zimbra dosfilter with lower threshold and fail login policy. Dosfilter recoginzed bad ip but the account was still locked out.
Could you show me your current configuration.
Regards,
Minh.


Just did a blog post on this:
https://www.missioncriticalemail.com/20 ... -together/

Please take a look and let me know if that works for you?

All the best,
Mark

Re: Cant shake bruteforce monkey

Posted: Sat Oct 20, 2018 6:00 pm
by JDunphy
L. Mark Stone wrote:
Just did a blog post on this:
https://www.missioncriticalemail.com/20 ... -together/
Mark

That is really well done Mark. Got me thinking that perhaps an enhancement should someone wish to take this further at the fw/packet level.

1) add a single entry via iptables that matches an ipset that is time based for removal
2) add the ip address dynamically from information provided by dosfilter

We have a 4 hr ipset and a 12 hr ipset that our incoming relays will put any bad actors into time out. I have seen 1000's of ip's added in a few minutes without load to the server. Super efficient and the bad guy burns all their ip's and hopefully goes away.

where this is in your /etc/sysconfig/iptables or whatever method you use for your fw rules.

Code: Select all


-A Block -m set --match-set blacklist24hr src -j DROP

and the ipset's have been created via:

Code: Select all

% ipset create blacklist4hr hash:ip hashsize 4096 timeout 14400
% ipset create blacklist24hr hash:ip hashsize 4096 timeout 86400

then you have a script looking for triggers from logs or whatever and they are added like thus:

Code: Select all

ipset add blacklist4hr X.X.X.X

Not as elaborate as fail2ban but ipsets are super efficient and scale massively and the block is in real-time without any restart of your firewall. If this was a multi-server zimbra install, one could do it on the parameter via syslog and a script looking for those ip addresses.Note: Because the ip's decay automatically and are removed there is nothing further to do after addition... ie. a hacker that has used TOR being a usage case and why you want to might want to decay those ip's.

Note: there are variations one could do without writing any scripts and using the ipset just with FW rules. Example: control number of connections per ip over a time interval.

Code: Select all

-A Block -m set --match-set someblacklist src -j DROP
-A Block -p tcp --dport 25 -m state --state NEW -m recent --set --name SMTP
-A Block -p tcp --dport 25 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 --rttl --name SMTP -j SET --add-set someblacklistr src
#allow incoming traffic if not more than 10 connections in 60 seconds per ip
-A Block -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT

Re: Cant shake bruteforce monkey

Posted: Sat Oct 20, 2018 11:22 pm
by vavai
Hi Mark & JDunphy,

Thank you for share valuable discussion. I'll be take a chance to deploy what you described on your blog as It seems it will help me a lot to deal with brute-force attack and improving our anti spam strategy.

Re: Cant shake bruteforce monkey

Posted: Tue Oct 23, 2018 12:52 am
by copowpow
L. Mark Stone wrote:Just did a blog post on this:
https://www.missioncriticalemail.com/20 ... -together/

Please take a look and let me know if that works for you?

All the best,
Mark



Thank you Mark, I am implementing this tonight! I will report back once I get a few days under these new settings. Very glad that Zimbra is adding these features to the DoS protection!

Re: Cant shake bruteforce monkey

Posted: Wed Oct 24, 2018 1:49 pm
by Newt0n
L. Mark Stone wrote:If you use fail2ban and a password lockout policy, just set the fail2ban threshold lower than the lockout policy.

In this way, a brute force attacker will get their IP blocked before the user's mailbox is set to Locked Out status; the legitimate user can keep using their email.

You can do the same thing with Zimbra's DoSFilter, instead of fail2ban if you wish.

Hope that helps,
Mark


It might not run in this case because the brute force attack is using different IP so you can't block by IP because you can have 15 failed logins from 15 diferent IP in one minute.

Re: Cant shake bruteforce monkey

Posted: Wed Oct 24, 2018 3:39 pm
by copowpow
Well it looks like the DoS filter is not working (zcs 8.8.8) , accounts still getting locked out after following Marks blog post,

As per https://wiki.zimbra.com/wiki/DoSFilter, it says to check /opt/zimbra/log/zmmailboxd.out and /opt/zimbra/log/sync.log

/opt/zimbra/log/zmmailboxd.out is packed with new Broken Pipe errors (see pastebin link, too large to post here):

https://paste.ee/p/fbZMZ


/opt/zimbra/log/sync.log shows:

Code: Select all

2018-10-22 18:33:07,219 INFO  [Thread-12] [] sync - ZeXtras Mobile stopped
2018-10-22 18:33:43,495 INFO  [main] [] sync - ZeXtras Mobile started
2018-10-22 18:42:11,002 INFO  [Thread-12] [] sync - ZeXtras Mobile stopped
2018-10-22 18:42:44,183 INFO  [main] [] sync - ZeXtras Mobile started






Not sure what to do now, can i fix these broken pipe errors so the thing actually works? I would really like this to work. Is there a log somewhere that shows what IP's its trying to block or is it supposed to be in the zmmailboxd.out ?