Failed auth attacks locking out users

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
diito
Posts: 3
Joined: Thu May 26, 2016 7:17 pm

Failed auth attacks locking out users

Postby diito » Fri Apr 05, 2019 7:25 pm

Hi,

I've been experiencing an issue were users are routinely locked out of their accounts (several times a day) because of failed login attempts from random bad actors on the internet. It's infuriating because I can't figure out how to deal with it. I allow three failed password attempts in 15 minutes to prevent brute forcing passwords as well as have fail2ban setup to simply drop any traffic after two failed attempts for 15 minutes, and then for a year if that same IP shows up twice more on the block list later on. That works for the people trying to bruce force me from the same IP address. The issue is that I now get a bunch of single failed auth attempts from a different (seemingly unrelated) set of IP addresses. In that situation fail2ban does nothing and the failed login attempt counter reaches the max and the account gets locked out for 15 minutes.

Anyone have any idea on how to deal with this?


User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: Failed auth attacks locking out users

Postby pup_seba » Sat Apr 06, 2019 5:05 pm

Hi,

Nope...sorry. As per Zimbra native ways to deal with these things are based on blocking an specific IP (DoSFilter...no need for fail2ban imho) or an account (jetty filter). For distributed attacks, there is no native Zimbra way to do it that I'm aware of. I guess that if you need to implement something like this, it should be something that upon registering different login attempst to the same account from different IP sources, it could do things like increment delays for each petition. But in order to achieve such functionality, that new element should be the one getting the petition. It does not seems to be an easy thing to do or at least it should need some work done to integrate with zimbra.

You could use this dashboard if you want easily see when an account is being "attacked" from different IPs. https://github.com/Zimbra-Community/zimbra-elasticstack

Also, these kind of attacks, at least the ones I saw for Zimbra, they don't come from "thousands" of IPs...but rather ~100 or less. So it would not be something too "crazy" to block those IPs, which it could be easily done manually or preferably via scripting ( I can actually block such IPs querying elasticsearch and firing a couple of ansible playbooks).

We have some guys in this community that are pretty good when it comes to code and security...maybe they could throw us some bone here :D
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 512
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 8.7.11_P14 RHEL6 Network Edition
Contact:

Re: Failed auth attacks locking out users

Postby JDunphy » Sun Apr 07, 2019 7:18 pm

This is a hard problem to solve with the tools zimbra provides IMO.

Just that ip4 attack surface is currently 2**32 which is 4B potential ip addresses. That seems like an excessive number of hosts to grant access to any authentication services given your users incoming address ranges could be represented by a small list of cidrs. So a few ideas given that blocking past ip's won't help with future brute force attacks to reduce this attack surface. This is only a Band-Aid unless you want to restrict access to trusted ip space.

Have you determined the origin of these attacking ip's or is there a pattern... geo, tor, etc. If they are tor exit nodes, https://2019.www.torproject.org/docs/faq-abuse.html.en#Bans shows how to ban access to your service. Geo blocking is pretty easy to do with an ipset and 1 line iptable command. I do something similar for cloudflare where I only allow access from their cidr's to our origin servers for one of our web farms.

Here is an example of using an ipset with iptables for a trusted cidr ip range. You could change this up and block a list of regions from certain ports... 443, 80, 995, etc, etc using the same ipset.

Code: Select all

# cloudflare access
-A Input -m state --state NEW -m tcp -p tcp --dport 8080 -m set --match-set trusted_hosts src -j ACCEPT

Here is a site to get you started if geo blocking is a direction that makes sense. http://www.ipdeny.com/ipblocks/

As a gross example - USA has about 18K aggregated cidrs and Canada where I am from is only 4.5K. This is an efficient lookup with an ipset. I have scripts in my github to build these type of aggregate cidrs because we currently geo tag incoming mail with a milter to provide our reputation engines with a little more information. Our normal email mix is about 5 countries so everything else is considered foreign to that engine where those foreign ip's get a lot more rules and reputation checks applied to them. One needs to be a little careful here for access because false positives could happen as that aggregate range is quite fluid and sometimes not SWIP'd immediately by some NSP's so the data can be slightly out of date.

For my own Zimbra server which saw 1000's of unique ip's attack it, I removed that attack surface completely. I added a few openvpn access servers in the cloud (for use when out of the office or mobile platforms) and locked those access servers ip's along with the office cidr's so only those trusted ip addresses have access to zimbra service ports. We keep our out facing MX's external which have a lot more edge detection rules using some custom milters and we deploy some netfilters features like recent, hitcount and automatic blacklisting into a realtime ipset for excessive connections per minute for our incoming rule for port 25. The goal is to not allow a single ip address hammer any MX's with too many connections per ip over a 60 second interval. Should they try, they go into a 4 hr ipset that expires automatically before they can connect for re-delivery again. No code other than iptables rules to make this work.

A better solution is something I plan to work on and to give me an edge with future XSS bugs that will continue to crop up. I am adding a modsecurity nginx connector to our zimbra proxies. That will allow additional capability to observe oddities for future attacks and would allow the creation of rules to track access anywhere in the http pipeline... for example, the response header from zimbra. In theory, one could use modsecurity's collection type to map users login/failures/successes and count the misses... after some number of misses, one could throw up a captcha or other for a period of time for that account to slow down the attacks that DosFilter would normally see to reduce or prevent this lockout.

This problem is best described as a nuisance; although it can be a real DoS for a user by any determined attacker. Temporarily locking access is prudent but I wish Zimbra would provide another security layer such as throwing up a captcha to help with this type of nuisance.

I would be interested in what others have tried. It is a complex subject without any perfect one fits all strategy IMO.
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: Failed auth attacks locking out users

Postby pup_seba » Mon Apr 08, 2019 7:16 am

I would be interested in what others have tried

I wish I had something to share :/ What you are saying about modsecurity sounds promizing, I wish you very good luck with that!!! It could also be feeded by other things like elastic for what I can read here https://karlstoney.com/2018/02/23/nginx ... ecchatops/
copowpow
Posts: 19
Joined: Mon Mar 26, 2018 3:34 pm

Re: Failed auth attacks locking out users

Postby copowpow » Fri Apr 12, 2019 7:05 pm

diito wrote:Anyone have any idea on how to deal with this?


I was in your exact situation, and it was about 5 users that were being targeted. I have fail2ban, geo blocking most of EU on the firewall and also the default failed login attempts zimbra lockout method and I was still getting attacked and the users would be locked out daily.

What I did was rename the accounts to something else, and create a new email account with the old name that forewarded the received email to the re-named account. That way the account with the original name would get locked out and it didnt matter because the user was now logging into the re-named account. After about 3 months with the accounts locked the attacks eventually stopped and I was able to rename the original accounts back to the original name.

Its not a pretty work around, and you have to have your users log in with another account name for a while (you also have to enable sending as another user so the recipient only saw the original account name) but it worked for us. I guess 3 months of failed attempts on the accounts led to the attackers moving on.

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 8 guests