Fail2ban is a great little tool for system administration. In a nutshell, it watches your log files for pre-defined patterns and then executes actions if it sees them. Ususally, this is of the form "if there are more than X failed authentication attempts in Y minutes from a single IP address, update the firewall to block the offending IP for Z minutes". The actions are not restricted to updating the firewall with iptables - you can also configure fail2ban to send notification emails, for example. This is useful for protecting against brute force attacks against services like:

• SSH (you should be using Publickey authentication if possible which will stop them ever guessing a password, but allowing them to try is still a waste of resources)
• SASL authentication attempts (Postfix and Dovecot)
• Login forms for web based services like Roundcube webmail.

It is also useful for dealing with certain kinds of denial of service attacks: for example, ModSecurity is good at blocking requests that take up lots of bandwidth, but it won't stop bots making loads of requests that get served a small 301 response page, even though dealing with all of these connections will slow down your web server.

Installation

First, install fail2ban:

sudo apt-get update
sudo apt-get install fail2ban

Configuration

Configuration files are found in /etc/fail2ban. The subdirectory /etc/fail2ban/filter.d/ is where you define the patterns that fail2ban will look for, and /etc/fail2ban/action.d/ contains all of the possible actions that will be executed if the pattern is spotted. However, the most important file is /etc/fail2ban/jail.conf: this is where you turn jails on/off, define which actions go with each jail, and set timing limits. Because new versions of fail2ban may overwrite this file, we create a local version of it which will override the defaults:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

The .local extension is not arbitrary - fail2ban expects you to put configuration here. Now we need to edit it to make some changes:

Whitelist your LAN:

Find this line:

ignoreip = 127.0.0.1/8

...and change it to this to include all of your LAN, not just localhost (assumes your router is at 192.168.1.1):

ignoreip = 127.0.0.1/8 192.168.1.0/24

Don't forget to restart fail2ban after making changes:

sudo service fail2ban restart

A few defaults

The first part of the file sets the defaults for each pattern/jail. The bantime is the duration of the ban, maxretry is the number of times the pattern can be matched, and the findtime is the length of time that they can be matched in. I left these at their default values: if the pattern is matched 3 times in 10 minutes, the IP address is banned for 10 minutes. After 10 minutes, the firewall rule is removed again.

Enabling some pre-installed rules

I'm going to use the postfix jail as an example. Find this part of the file:

[postfix]
enabled = false
port     = smtp,ssmtp,submission
filter   = postfix
logpath  = /var/log/mail.log

Enable it by editing the relevant line to this:

enabled = true

This enables the jail /etc/fail2ban/filter.d/postfix.conf:

[Definition]

_daemon = postfix/smtpd

failregex = ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[\]: 554 5\.7\.1 .*$
            ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[\]: 450 4\.7\.1 : Helo command rejected: Host not found; from=<> to=<> proto=ESMTP helo= *$
            ^%(__prefix_line)sNOQUEUE: reject: VRFY from \S+\[\]: 550 5\.1\.1 .*$

ignoreregex =

As you can see, the file contains regular expressions that match entries in /var/log/mail.log where your mail server is rejecting incoming emails (presumably from spammers). The logpath is the path to the log that fail2ban is watching, and the filter relates to the name of the filter in /etc/fail2ban/filter.d/ with the .conf extension removed. The port section decides which ports will be blocked when the iptables-multiport action is executed. You can use numbers or names (i.e. 22 and ssh are interchangeable). We already covered setting default values at the start of jail.local; if you want to change the defaults then put them under the header for the rule you want to change, e.g. if we wanted the postfix rule to kick in if we matched 10 log entries in 10 minutes, and then ban the IP for an hour we could do this:

[postfix]
enabled = false
port     = smtp,ssmtp,submission
filter   = postfix
logpath  = /var/log/mail.log
findtime = 600
maxretry = 10
bantime = 3600

Writing a custom filter

As I mentioned in the intro, I have recently had problems with Denial of Service (DoS) attacks against my server that were either trying to overwhelm the number of concurrent connections that Apache could handle... or the script/person carrying out the attack was an idiot. Here's what my logs were showing:

59.58.137.162 - - [10/Aug/2014:11:48:25 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:25 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:26 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:26 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:27 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:27 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:28 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:28 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"

And here's a graph showing the huge peak in traffic: My server re-writes http://samhobbs.co.uk to http://www.samhobbs.co.uk, so each hit was served a small 301 "moved permanently" page. If the attacker had requested the www. version they would have got a much larger page (better for choking up resources, from their point of view) but ModSecurity's DoS rules would have kicked in. To give Apache a bit of a break I created a custom filter like this:

sudo nano /etc/fail2ban/filter.d/apache-301-DoS.conf

# ModSecurity deals with most denial of service attacks by monitoring how many images and
# other high-bandwidth resources are sent per unit time. It doesn't block requests that
# result in multiple 301 redirects or errors

# example:
# 59.58.137.162 - - [10/Aug/2014:07:42:25 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"

[Definition]
failregex  = ^<HOST> - - (?:\[[^]]*\] )+\"GET .* HTTP/1.1\" 301
ignoreregex =

And then appended this to /etc/fail2ban/jail.local:

# "home made" filter to stop 301 denial of service attacks
[apache-301-DoS]
enabled = true
port = http,https
filter = apache-301-DoS
action = iptables-multiport[name=apache301DoS]
        sendmail-whois[name=Apache-301-DoS, dest=root@yourdomain.com, sender=fail2ban@yourdomain.com]
logpath = /var/log/apache2/*/access.log
bantime = 600
findtime = 60
maxretry = 20

...so if an IP address gets 20 301 redirects in a minute, they get banned for ten minutes, and an email is sent to the root email address using fail2ban's premade sendmail-whois action (which also does a whois lookup on the IP address and includes the information in the email). Legitimate users of the site may get a few 301s, but never that many.

Checking a filter's regex

To check a filter's regex, we can use this command:

fail2ban-regex /path/to/logfile.log /path/to/filter.conf

e.g:

sam@samhobbs:/etc/fail2ban$ fail2ban-regex /var/log/apache2/samhobbs/access.log /etc/fail2ban/filter.d/apache-301-DoS.conf

Running tests
=============

Use   failregex file : /etc/fail2ban/filter.d/apache-301-DoS.conf
Use         log file : /var/log/apache2/samhobbs/access.log


Results
=======

Failregex: 16860 total
|-  #) [# of hits] regular expression
|   1) [16860] ^ - - (?:\[[^]]*\] )+\"GET .* HTTP/1.1\" 301
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [27757] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 27757 lines, 0 ignored, 16860 matched, 10897 missed
Missed line(s):: too many to print.  Use --print-all-missed to print all 10897 lines

Notice that this would have counted the number of entries that were ignored due to the ignoreregex, if I had defined ignoreregex in the filter. For more verbose output, you can use the -v option which will give you the timestamps that the rules were matched at. This command is really useful for checking regex when you are writing a custom filter - you don't have to sit there and watch it in action to check for false positives etc, just run the command to see what the server would have done if the rule had been on.

Modifying Default Filters

I found that I needed to make a change to the postfix-sasl.conf filter because I changed my postfix configuration - the logging format for connecitons on 465 appears in the logs with the tag "smtps" instead of "smtp" so I can distinguish between the two. Failed logins on port 25 look like this:

Apr 28 17:55:02 samhobbs postfix/smtpd[29656]: warning: agreeabledismiss.com[130.185.150.126]: SASL LOGIN authentication failed: UGFzc3dvcmQ6

...and failed logins on port 465 look like this (the difference is the postfix/smtps/smtpd part):

Apr 28 15:49:29 samhobbs postfix/smtps/smtpd[29077]: warning: unknown[177.244.148.77]: SASL LOGIN authentication failed: UGFzc3dvcmQ6

This breaks the default rule. I wondered if I could overwrite this rule by creating a filter called postfix-sasl.local, but my experiments have shown that it doesn't work. So, if you want to change a default rule, I'd recommend you make a copy and edit that instead of changing the original file, or your changes could be lost when fail2ban is updated. In this case, I copied the file to postfix-sasl-SWH.conf and then edited it to look like this:

# Fail2Ban filter for postfix authentication failures
#

[INCLUDES]

before = common.conf

[Definition]

#_daemon = postfix/smtpd
_daemon = postfix/smtps/smtpd

failregex = ^%(__prefix_line)swarning: [-._\w]+\[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed(: [ A-Za-z0-9+/]*={0,2})?\s*$
ignoreregex =
# Author: Yaroslav Halchenko

The updated _daemon line means the new log messages will match. Since I wanted to run this rule in addition to the original one, I added this to my jail.local:

[sasl-SWH]
# cutsom filter to match updated logging tag for port 465
enabled = true
port = smtp,ssmtp,submission,imap2,imap3,imaps,pop3,pop3s
filter = postfix-sasl-SWH
logpath = /var/log/mail.log

Of course, if I had wanted to run the modified filter instead of the original one then I could have just change the filter line in the original rule.

Permanently banning persistent abusers

There's a great ruleset written up at this site describing how to ban repeat offenders. Fail2ban writes its own log to /var/log/fail2ban.log, and the filter monitors this logfile for IP addresses that get banned multiple times and bans them for ever. These firewall rules will be re-created when you reboot the system or restart fail2ban. It's really handy and I highly recommend making use of it!

Checking your firewall rules

To list the current state of your firewall, you can use this command:

sudo iptables -L

You will see output similar to this:

sam@samhobbs:/etc/fail2ban$ sudo iptables -L
Chain INPUT (policy ACCEPT)                                                                                                                                                                   
target     prot opt source               destination         
fail2ban-apache301DoS  tcp  --  anywhere             anywhere             multiport dports ssh
fail2ban-ip-blocklist  tcp  --  anywhere             anywhere            
fail2ban-repeatoffender  tcp  --  anywhere             anywhere            
fail2ban-dovecot  tcp  --  anywhere             anywhere             multiport dports smtp,urd,submission,imap2,imap3,imaps,pop3,pop3s
fail2ban-sasl  tcp  --  anywhere             anywhere             multiport dports smtp,urd,submission,imap2,imap3,imaps,pop3,pop3s
fail2ban-postfix  tcp  --  anywhere             anywhere             multiport dports smtp,urd,submission
fail2ban-roundcube-auth  tcp  --  anywhere             anywhere             multiport dports http,https
fail2ban-ssh-ddos  tcp  --  anywhere             anywhere             multiport dports ssh
fail2ban-ssh  tcp  --  anywhere             anywhere             multiport dports ssh

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain fail2ban-apache301DoS (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-dovecot (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-ip-blocklist (1 references)
target     prot opt source               destination         
REJECT     all  --  48-255-144-216.static.reverse.lstn.net  anywhere             reject-with icmp-port-unreachable
REJECT     all  --  star28.auto.ru       anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-postfix (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-repeatoffender (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-roundcube-auth (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-sasl (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-ssh (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-ssh-ddos (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

This command can be quite slow because it does reverse DNS lookups on the IP addresses listed in your firewall. If you don't need that information and don't want to wait, you can use:

sudo iptables -L -n

Useful links

Fail2ban's manual