Introduction to Fail2ban

fail2ban logo

Fail2ban is a great little tool for system administration. In a nutshell, it watches your log files for pre-defined patterns and then executes actions if it sees them. Ususally, this is of the form "if there are more than X failed authentication attempts in Y minutes from a single IP address, update the firewall to block the offending IP for Z minutes". The actions are not restricted to updating the firewall with iptables - you can also configure fail2ban to send notification emails, for example.

This is useful for protecting against brute force attacks against services like:

  • SSH (you should be using Publickey authentication if possible which will stop them ever guessing a password, but allowing them to try is still a waste of resources)
  • SASL authentication attempts (Postfix and Dovecot)
  • Login forms for web based services like Roundcube webmail.

It is also useful for dealing with certain kinds of denial of service attacks: for example, ModSecurity is good at blocking requests that take up lots of bandwidth, but it won't stop bots making loads of requests that get served a small 301 response page, even though dealing with all of these connections will slow down your web server.

Installation

First, install fail2ban:

sudo apt-get update
sudo apt-get install fail2ban

Configuration

Configuration files are found in /etc/fail2ban. The subdirectory /etc/fail2ban/filter.d/ is where you define the patterns that fail2ban will look for, and /etc/fail2ban/action.d/ contains all of the possible actions that will be executed if the pattern is spotted.

However, the most important file is /etc/fail2ban/jail.conf: this is where you turn jails on/off, define which actions go with each jail, and set timing limits. Because new versions of fail2ban may overwrite this file, we create a local version of it which will override the defaults:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

The .local extension is not arbitrary - fail2ban expects you to put configuration here. Now we need to edit it to make some changes:

Whitelist your LAN:

Find this line:

ignoreip = 127.0.0.1/8

...and change it to this to include all of your LAN, not just localhost (assumes your router is at 192.168.1.1):

ignoreip = 127.0.0.1/8 192.168.1.0/24

Don't forget to restart fail2ban after making changes:

sudo service fail2ban restart

A few defaults

The first part of the file sets the defaults for each pattern/jail. The bantime is the duration of the ban, maxretry is the number of times the pattern can be matched, and the findtime is the length of time that they can be matched in. I left these at their default values: if the pattern is matched 3 times in 10 minutes, the IP address is banned for 10 minutes. After 10 minutes, the firewall rule is removed again.

Enabling some pre-installed rules

I'm going to use the postfix jail as an example. Find this part of the file:

[postfix]
enabled = false
port     = smtp,ssmtp,submission
filter   = postfix
logpath  = /var/log/mail.log

Enable it by editing the relevant line to this:

enabled = true

This enables the jail /etc/fail2ban/filter.d/postfix.conf:

[Definition]

_daemon = postfix/smtpd

failregex = ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[\]: 554 5\.7\.1 .*$
            ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[\]: 450 4\.7\.1 : Helo command rejected: Host not found; from= to= proto=ESMTP helo= *$
            ^%(__prefix_line)sNOQUEUE: reject: VRFY from \S+\[\]: 550 5\.1\.1 .*$

ignoreregex =

As you can see, the file contains regular expressions that match entries in /var/log/mail.log where your mail server is rejecting incoming emails (presumably from spammers).

The logpath is the path to the log that fail2ban is watching, and the filter relates to the name of the filter in /etc/fail2ban/filter.d/ with the .conf extension removed. The port section decides which ports will be blocked when the iptables-multiport action is executed. You can use numbers or names (i.e. 22 and ssh are interchangeable).

We already covered setting default values at the start of jail.local; if you want to change the defaults then put them under the header for the rule you want to change, e.g. if we wanted the postfix rule to kick in if we matched 10 log entries in 10 minutes, and then ban the IP for an hour we could do this:

[postfix]
enabled = false
port     = smtp,ssmtp,submission
filter   = postfix
logpath  = /var/log/mail.log
findtime = 600
maxretry = 10
bantime = 3600

Writing a custom filter

As I mentioned in the intro, I have recently had problems with Denial of Service (DoS) attacks against my server that were either trying to overwhelm the number of concurrent connections that Apache could handle... or the script/person carrying out the attack was an idiot. Here's what my logs were showing:

59.58.137.162 - - [10/Aug/2014:11:48:25 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:25 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:26 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:26 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:27 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:27 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:28 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"
59.58.137.162 - - [10/Aug/2014:11:48:28 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"

And here's a graph showing the huge peak in traffic:

DoS_stats.png

My server re-writes http://samhobbs.co.uk to http://www.samhobbs.co.uk, so each hit was served a small 301 "moved permanently" page. If the attacker had requested the www. version they would have got a much larger page (better for choking up resources, from their point of view) but ModSecurity's DoS rules would have kicked in. To give Apache a bit of a break I created a custom filter like this:

sudo nano /etc/fail2ban/filter.d/apache-301-DoS.conf
# ModSecurity deals with most denial of service attacks by monitoring how many images and
# other high-bandwidth resources are sent per unit time. It doesn't block requests that
# result in multiple 301 redirects or errors

# example:
# 59.58.137.162 - - [10/Aug/2014:07:42:25 +0100] "GET /2013/ HTTP/1.1" 301 478 "-" "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"

[Definition]
failregex  = ^<HOST> - - (?:\[[^]]*\] )+\"GET .* HTTP/1.1\" 301
ignoreregex =

And then appended this to /etc/fail2ban/jail.local:

# "home made" filter to stop 301 denial of service attacks
[apache-301-DoS]
enabled = true
port = http,https
filter = apache-301-DoS
action = iptables-multiport[name=apache301DoS]
        sendmail-whois[name=Apache-301-DoS, dest=root@yourdomain.com, sender=fail2ban@yourdomain.com]
logpath = /var/log/apache2/*/access.log
bantime = 600
findtime = 60
maxretry = 20

...so if an IP address gets 20 301 redirects in a minute, they get banned for ten minutes, and an email is sent to the root email address using fail2ban's premade sendmail-whois action (which also does a whois lookup on the IP address and includes the information in the email). Legitimate users of the site may get a few 301s, but never that many.

Checking a filter's regex

To check a filter's regex, we can use this command:

fail2ban-regex /path/to/logfile.log /path/to/filter.conf

e.g:

sam@samhobbs:/etc/fail2ban$ fail2ban-regex /var/log/apache2/samhobbs/access.log /etc/fail2ban/filter.d/apache-301-DoS.conf

Running tests
=============

Use   failregex file : /etc/fail2ban/filter.d/apache-301-DoS.conf
Use         log file : /var/log/apache2/samhobbs/access.log


Results
=======

Failregex: 16860 total
|-  #) [# of hits] regular expression
|   1) [16860] ^ - - (?:\[[^]]*\] )+\"GET .* HTTP/1.1\" 301
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [27757] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 27757 lines, 0 ignored, 16860 matched, 10897 missed
Missed line(s):: too many to print.  Use --print-all-missed to print all 10897 lines

Notice that this would have counted the number of entries that were ignored due to the ignoreregex, if I had defined ignoreregex in the filter.

For more verbose output, you can use the -v option which will give you the timestamps that the rules were matched at. This command is really useful for checking regex when you are writing a custom filter - you don't have to sit there and watch it in action to check for false positives etc, just run the command to see what the server would have done if the rule had been on.

Modifying Default Filters

I found that I needed to make a change to the postfix-sasl.conf filter because I changed my postfix configuration - the logging format for connecitons on 465 appears in the logs with the tag "smtps" instead of "smtp" so I can distinguish between the two. Failed logins on port 25 look like this:

Apr 28 17:55:02 samhobbs postfix/smtpd[29656]: warning: agreeabledismiss.com[130.185.150.126]: SASL LOGIN authentication failed: UGFzc3dvcmQ6

...and failed logins on port 465 look like this (the difference is the postfix/smtps/smtpd part):

Apr 28 15:49:29 samhobbs postfix/smtps/smtpd[29077]: warning: unknown[177.244.148.77]: SASL LOGIN authentication failed: UGFzc3dvcmQ6

This breaks the default rule. I wondered if I could overwrite this rule by creating a filter called postfix-sasl.local, but my experiments have shown that it doesn't work. So, if you want to change a default rule, I'd recommend you make a copy and edit that instead of changing the original file, or your changes could be lost when fail2ban is updated. In this case, I copied the file to postfix-sasl-SWH.conf and then edited it to look like this:

# Fail2Ban filter for postfix authentication failures
#

[INCLUDES]

before = common.conf

[Definition]

#_daemon = postfix/smtpd
_daemon = postfix/smtps/smtpd

failregex = ^%(__prefix_line)swarning: [-._\w]+\[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed(: [ A-Za-z0-9+/]*={0,2})?\s*$
ignoreregex =
# Author: Yaroslav Halchenko

The updated _daemon line means the new log messages will match.

Since I wanted to run this rule in addition to the original one, I added this to my jail.local:

[sasl-SWH]
# cutsom filter to match updated logging tag for port 465
enabled = true
port = smtp,ssmtp,submission,imap2,imap3,imaps,pop3,pop3s
filter = postfix-sasl-SWH
logpath = /var/log/mail.log

Of course, if I had wanted to run the modified filter instead of the original one then I could have just change the filter line in the original rule.

Permanently banning persistent abusers

There's a great ruleset written up at this site describing how to ban repeat offenders. Fail2ban writes its own log to /var/log/fail2ban.log, and the filter monitors this logfile for IP addresses that get banned multiple times and bans them for ever. These firewall rules will be re-created when you reboot the system or restart fail2ban.

It's really handy and I highly recommend making use of it!

Checking your firewall rules

To list the current state of your firewall, you can use this command:

sudo iptables -L

You will see output similar to this:

sam@samhobbs:/etc/fail2ban$ sudo iptables -L
Chain INPUT (policy ACCEPT)                                                                                                                                                                   
target     prot opt source               destination         
fail2ban-apache301DoS  tcp  --  anywhere             anywhere             multiport dports ssh
fail2ban-ip-blocklist  tcp  --  anywhere             anywhere            
fail2ban-repeatoffender  tcp  --  anywhere             anywhere            
fail2ban-dovecot  tcp  --  anywhere             anywhere             multiport dports smtp,urd,submission,imap2,imap3,imaps,pop3,pop3s
fail2ban-sasl  tcp  --  anywhere             anywhere             multiport dports smtp,urd,submission,imap2,imap3,imaps,pop3,pop3s
fail2ban-postfix  tcp  --  anywhere             anywhere             multiport dports smtp,urd,submission
fail2ban-roundcube-auth  tcp  --  anywhere             anywhere             multiport dports http,https
fail2ban-ssh-ddos  tcp  --  anywhere             anywhere             multiport dports ssh
fail2ban-ssh  tcp  --  anywhere             anywhere             multiport dports ssh

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain fail2ban-apache301DoS (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-dovecot (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-ip-blocklist (1 references)
target     prot opt source               destination         
REJECT     all  --  48-255-144-216.static.reverse.lstn.net  anywhere             reject-with icmp-port-unreachable
REJECT     all  --  star28.auto.ru       anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-postfix (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-repeatoffender (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-roundcube-auth (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-sasl (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-ssh (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-ssh-ddos (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

This command can be quite slow because it does reverse DNS lookups on the IP addresses listed in your firewall. If you don't need that information and don't want to wait, you can use:

sudo iptables -L -n

Useful links

Fail2ban's manual

Type: 

Comments

Thank you for tutorial.

Just one question. Whenever I receive a mail from Fail2Ban, the date displayed in the email is wrong. Like for today it says 28.1.2014, but everything is OK in the list where e-mails are shown (left panel in Outlook).

I checked the logs and discovered that Fail2Ban sets the date with the Date header in SMTP communication. Is this a bug? Thank you.

Logs from my mail:

Return-Path:
From: "Fail2Ban"
To:
Subject: [Fail2Ban] Dovecot: started
Date: Tue, 28 Jan 2014 18:54:11 +0100

That's odd!

Use the date command on the server to check if the system time is correct.

Also, can you check if the time is correct in your log files?

Sam

The date command returns todays date and my mail log contains the correct date.

For other mails, everything is working fine, the date is correct.

I don't know what could be causing that... out of interest, which OS are you using?

Sam

EDIT: my google searches have turned up some bugs related to locale settings. What's your locale set to? E.g. use the locale command.

I am running normal Debian release on Raspberry PI and on my computer Windows 7.

My locale setting are:

LANG=sl_SI.UTF-8
LANGUAGE=
LC_CTYPE="sl_SI.UTF-8"
LC_NUMERIC="sl_SI.UTF-8"
LC_TIME="sl_SI.UTF-8"
LC_COLLATE="sl_SI.UTF-8"
LC_MONETARY="sl_SI.UTF-8"
LC_MESSAGES="sl_SI.UTF-8"
LC_PAPER="sl_SI.UTF-8"
LC_NAME="sl_SI.UTF-8"
LC_ADDRESS="sl_SI.UTF-8"
LC_TELEPHONE="sl_SI.UTF-8"
LC_MEASUREMENT="sl_SI.UTF-8"
LC_IDENTIFICATION="sl_SI.UTF-8"
LC_ALL=

Hi Sam,

I got this email this morning:

/etc/cron.daily/ddclient:
SUCCESS: updating @: good: IP address set to 2.103.246.251
/etc/cron.daily/logrotate:
ERROR Unable to contact server. Is it running?
error: error running non-shared postrotate script for /var/log/fail2ban.log of
'/var/log/fail2ban.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

It shows the email server id ok, but not fail2ban (I think)

How do I know if the server is running? Is it talking about the email server? How do I know if the fail2ban service is running?

When i run the "sudo service fail2ban restart" command I get no errors...

When I followed the instructions on the wireflare and ran the "/usr/bin/fail2ban-client reload" command, it wouldn't connect to the server...

I also checked and fund the fail2ban.log to be empty if that helps.

Please advise.

Thx,

Jo

Test if fail2ban is running with:

sudo service fail2ban status

Logrotate is the service that rotates your logs at specified intervals (e.g. monthly) or when they reach a certain size. Looks like there's an issue with the script it is running after the logs are rotated (the "postrotate script", from the site I linked to for banning persistent abusers).

Possibly something to do with the switch from traditional UNIX logging in flat text files, to systemd's journald, where logs are stored in a database and queries are made with journalctl. Have a look at the journalctl manpage for more information.

Unfortunately, I can't be of much help here because my server runs ubuntu 14.04, and the first Ubuntu release with systemd was 15.04, so I haven't had to work through all the quirks yet.

Sam

Hi Sam,

Following on from what you've mentioned, I did a bit of digging...but first here's my fail2ban status:

pi@raspberrypi ~ $ sudo service fail2ban status
● fail2ban.service - LSB: Start/stop fail2ban
Loaded: loaded (/etc/init.d/fail2ban)
Active: active (exited) since Sun 2015-11-01 08:28:50 GMT; 10h ago
Process: 24511 ExecStop=/etc/init.d/fail2ban stop (code=exited, status=0/SUCCESS)
Process: 24521 ExecStart=/etc/init.d/fail2ban start (code=exited, status=0/SUCCESS)

Nov 01 08:28:50 raspberrypi fail2ban[24521]: Starting authentication failure monitor: fail2banERROR No file(s) foun...s.log
Nov 01 08:28:50 raspberrypi fail2ban[24521]: ERROR Failed during configuration: Have not found any log file for apa... jail
Nov 01 08:28:50 raspberrypi fail2ban[24521]: failed!
Nov 01 08:28:50 raspberrypi systemd[1]: Started LSB: Start/stop fail2ban.
Nov 01 17:39:21 raspberrypi systemd[1]: Started LSB: Start/stop fail2ban.
Hint: Some lines were ellipsized, use -l to show in full.

.........
It seems that there are files missing and I did a search on google and came across two websites:

Links

https://www.howtoforge.com/community/threads/fail2ban-is-no-working.53686/
At the bottom of this page is a guy who had the same problem and fixed it...somehow, but pointed to two possible files that may not exist...
I can't see the files for me that don't exist because of the ellipsized...

This all led me to the following website where there was a similar discussion...

http://sourceforge.net/p/fail2ban/mailman/message/31069999/
They were going on about something to do with the /dev/null redirection which I see in /etc/init.d/fail2ban, but don't know anything about it.

How can I see the ellipsized data?
Does the info given make any sense to you? I've learnt a lot through this process over the last thre.four weeks, but am still way behind in understanding how all the linux files and commands fit together...

Regards,

Jo

Sounds like you did some good research there, well done.

The status command is printing some lines from the journal, so try this (I think it will work from my limited journalctl usage):

sudo journalctl -u fail2ban -l

I think the problem is probably that you recreated the apache-301-dos filter of mine, and your logs are in a different place (the apache logs on my server are split up into subdirs for each site, e.g. the logs for this site are at /var/log/apache2/samhobbs/ssl_access.log).

You could either change the path like this:

logpath = /var/log/apache2/access.log

or delete it altogether - it's just an example I wrote to fix a specific problem of mine, you may not even need it.

BTW, /dev/null is a location in the filesystem that is basically a black hole, you can write information into it and it just "disappears", so scripts that don't want any lines of output sometimes redirect the output there.

Sam

Hi Sam,

Thanks again...you did it once more!

I ran sudo journalctl -u fail2ban -l and found the following which made things clearer:

Nov 01 08:27:52 raspberrypi systemd[1]: Started LSB: Start/stop fail2ban.
Nov 01 08:28:45 raspberrypi systemd[1]: Stopping LSB: Start/stop fail2ban...
Nov 01 08:28:45 raspberrypi fail2ban[24511]: Stopping authentication failure monitor: fail2ban.
Nov 01 08:28:45 raspberrypi systemd[1]: Starting LSB: Start/stop fail2ban...
Nov 01 08:28:50 raspberrypi fail2ban[24521]: Starting authentication failure monitor: fail2banERROR No file(s) found for glob /var/log/apache2/*/access.log
Nov 01 08:28:50 raspberrypi fail2ban[24521]: ERROR Failed during configuration: Have not found any log file for apache-301-DoS jail
Nov 01 08:28:50 raspberrypi fail2ban[24521]: failed!
Nov 01 08:28:50 raspberrypi systemd[1]: Started LSB: Start/stop fail2ban.
Nov 01 17:39:21 raspberrypi systemd[1]: Started LSB: Start/stop fail2ban.
Nov 02 20:01:12 raspberrypi systemd[1]: Started LSB: Start/stop fail2ban.
Nov 02 20:04:37 raspberrypi systemd[1]: Started LSB: Start/stop fail2ban.
Nov 02 20:09:55 raspberrypi systemd[1]: Stopping LSB: Start/stop fail2ban...
Nov 02 20:09:55 raspberrypi fail2ban[12539]: Stopping authentication failure monitor: fail2ban.
Nov 02 20:09:55 raspberrypi systemd[1]: Stopped LSB: Start/stop fail2ban.
Nov 02 20:10:01 raspberrypi systemd[1]: Starting LSB: Start/stop fail2ban...

I then changed the log path as suggested and now I think it's working properly:

pi@raspberrypi ~ $ sudo service fail2ban status
● fail2ban.service - LSB: Start/stop fail2ban
Loaded: loaded (/etc/init.d/fail2ban)
Active: active (running) since Mon 2015-11-02 20:10:07 GMT; 8min ago
Process: 12539 ExecStop=/etc/init.d/fail2ban stop (code=exited, status=0/SUCCESS)
Process: 12580 ExecStart=/etc/init.d/fail2ban start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/fail2ban.service
└─12591 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail...

Nov 02 20:10:07 raspberrypi fail2ban[12580]: Starting authentication failure monitor: fail2ban.
Nov 02 20:10:07 raspberrypi systemd[1]: Started LSB: Start/stop fail2ban.

You're the best!

Regards,

Jo

You're welcome :)

The next LTS release of Ubuntu will be 16.04 in April next year, I plan on upgrading as soon as it's released. I should be better at asking jessie/systemd related questions after that!

Sam

Add new comment