Getting Started with Apache ModSecurity on Debian and Ubuntu

ModSecurityLogo.png

ModSecurity is a Web Application Firewall: a program that can be used to inspect information as it passes through your web server, intercepting malicious requests before they are processed by your web application.

This tutorial will show you how to install ModSecurity on Apache, and configure it with some sensible rules provided by the Open Web Application Security Project's Core Rule Set (OWASP CRS), which will help to protect your server against SQL injection, denial of service attacks, malformed requests, cross site scripting attacks, and more.

And yes, you can use this guide with your Raspberry Pi if you're running Raspbian, Ubuntu, or another Debian derivative on it.

The Open Web Application Security Project's Core Rule Set (OWASP CRS)

ModSecurity is configured using a set of rules that provide the logic used to decide which requests to block. Probably the most common set of rules is the Open Web Application Security Project's Core Rule Set (OWASP CRS), which is in the repos as modsecurity-crs.

The CRS can run in two modes: traditional and anomaly scoring. In traditional mode, the first rule that matches will block the request; in anomaly scoring mode the rules increment counters that "enumerate badness", and if the rule exceeds a threshold then the request is blocked.

In this tutorial, we will configure Apache to run the core rule set in anomaly scoring mode, since it allows for a more intelligent approach to blocking.

Installing ModSecurity and the CRS

I'm assuming at this point that you already have Apache2 installed and some kind of web application (Wordpress, ownCloud etc.) running. If you don't, running the following commands will install apache2 as a dependency anyway, but it's a good idea to get something up and running on the server before you install ModSecurity.

ModSecurity is a module for Apache2, so the name of the package follows the apache module conventions: libapache2-mod-security2. You can install it like this:

sudo apt-get update
sudo apt-get install libapache2-mod-security2

On most systems, this will pull in the CRS too, which is packaged as modsecurity-crs, as it is marked as a suggested package for modsecurity. This command will make sure it is set to manually installed so it won't get auto-removed:

sudo apt-get install modsecurity-crs

You can download new versions of the CRS directly from the project's GitHub repo, but don't! Some newer versions of the CRS make use of new features and directives in ModSecurity, so trying to run the newest rules with an "old" (read: not bleeding edge) version of ModSecurity will not work. I found this out the hard way...

Configuration

ModSecurity creates a directory at /etc/modsecurity during installation. This is the directory you should use to store all of your ModSecurity rules and configuration.

The module configuration file that comes with ModSecurity will read any files in this directory that end in .conf when apache starts up. This is what the module config file looks like:

<IfModule security2_module>
        # Default Debian dir for modsecurity's persistent data
        SecDataDir /var/cache/modsecurity

        # Include all the *.conf files in /etc/modsecurity.
        # Keeping your local configuration in that directory
        # will allow for an easy upgrade of THIS file and
        # make your life easier
        IncludeOptional /etc/modsecurity/*.conf
</IfModule>

This file should be at /etc/apache2/mods-available/security2.conf; there is also a file called security2.load that loads the module itself. We need to make sure that these two files are read by apache2 when it starts up, by creating symlinks into the /etc/apache2/mods-enabled directory. You could create the symlinks manually with ln -s, but the easiest and most reliable way is to use the a2enmod command:

sudo a2enmod security2

ModSecurity also makes use of the headers module, which we can enable like this:

sudo a2enmod headers

Don't reload apache2 yet!

ModSecurity Configuration File

At the moment, you should have two files in /etc/modsecurity: unicode.mapping and modsecurity.conf-recommended. The latter is a template config file. First we need to copy it so that it has a name that will be read by apache:

sudo cp /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

You shouldn't have to change much in this configuration file, but I'd like to bring your attention to the SecRuleEngine parameter, which decides whether ModSecurity is On, Off, or in DetectionOnly mode where it will process requests and write the audit log but not actually block anything:

SecRuleEngine DetectionOnly

You can leave this at DetectionOnly for now: there will be tons of false positives to begin with, and until you have written a whitelist (more on that later), turning the engine on will likely just break your site! When you have written the whitelist, you can change the parameter to On.

Filesystem Configuration

As can be seen in the module configuration file (/etc/apache2/mods-available/security2.conf above), ModSecurity creates a directory for storing temporary files at /var/cache/modsecurity during installation and makes it writable by Apache. The default configuration file specifies that temporary files should be stored in /tmp/, but this data is cleared after a reboot, whereas the data in /var/cache really is persistent.

This is important because some of the session hijacking rules in the CRS check to see if secure session ID cookies set by web apps and PHP are ones that ModSecurity has seen before, and block the request if they are "new"... so if you log in to a web app like ownCloud and then reboot the web server and visit again later, the request will be blocked because the old cookie is "new" to ModSecurity.

To fix this, navigate to the "Filesystem configuration" section in /etc/modsecurity/modsecurity.conf and change both SecTmpDir and SecDataDir from /tmp/ to /var/cache/modsecurity, i.e.

SecDataDir /var/cache/modsecurity

Now the session hijacking rules should work as intended (if you choose to enable them!).

Loading the CRS Config Files

The Core Rule Set is installed into /usr/share/modsecurity-crs/, and is separated into several directories such as base_rules, experimental_rules, optional_rules and slr_rules. SLR stands for SpiderLabs Research, which is the research team of the company that currently maintains the CRS.

We are going to create symbolic links in the /etc/modsecurity directory that point to the rule files we want to load.

You can ignore the activated_rules directory - on other distros or if you compiled everything from source, you might unpack the CRS tarball directly into /etc/modsecurity and then use the /etc/apache2/mods-enabled/security2.conf to make Apache read files in /etc/modsecurity/activated_rules instead of everything in the /etc/modsecurity directory.

It's entirely up to you which rule files you choose to load, but at a minimum you should be using the base_rules. I would also recommend loading all of the optional_rules and then choosing a few of the experimental rules you think might be useful for your site or app (for example, I chose to enable the brute force and DoS protection rule files). You probably don't want to enable the appsensor_detection_point_* files (if you're curious what these do, refer to the OWASP documentation). Bear in mind that the experimental rules are likely to require more work tweaking/whitelisting to remove false positives for your site.

Now, you could create each symlink individually, but that would be extremely tedious. If you want to use all of the rule files in one or more of those directories, this is much faster:

For the base_rules:

cd /usr/share/modsecurity-crs/base_rules/
for ruleFile in * ; do sudo ln -s /usr/share/modsecurity-crs/base_rules/$ruleFile /etc/modsecurity/$ruleFile ; done

For the optional_rules:

cd /usr/share/modsecurity-crs/optional_rules/
for ruleFile in * ; do sudo ln -s /usr/share/modsecurity-crs/optional_rules/$ruleFile /etc/modsecurity/$ruleFile ; done

You probably don't want to enable all of the experimental_rules, but if you did you could use this command:

cd /usr/share/modsecurity-crs/experimental_rules/
for ruleFile in * ; do sudo ln -s /usr/share/modsecurity-crs/experimental_rules/$ruleFile /etc/modsecurity/$ruleFile ; done

For the slr_rules, I would recommend yo.u only load rule files when they are relevant to the type of site you run, e.g. if you have a wordpress site, you could run these two commands to load the configuration and data files:

sudo ln -s /usr/share/modsecurity-crs/slr_rules/modsecurity_crs_46_slr_et_wordpress_attacks.conf /etc/modsecurity/modsecurity_crs_46_slr_et_wordpress_attacks.conf
sudo ln -s /usr/share/modsecurity-crs/slr_rules/modsecurity_46_slr_et_wordpress_attacks.data /etc/modsecurity/modsecurity_46_slr_et_wordpress_attacks.data

You should now have a bunch of symlinks in the /etc/modsecurity directory pointing to the rule files, but we still need to copy across the most important file: modsecurity_crs_10_setup.conf, which contains configuration for the CRS itself.

We symlinked the rule files so that when the package manager updates modsecurity-crs, a new version of the rule files will be automatically loaded by apache, meaning we will always have the latest version of the CRS for our version of ModSecurity. We will never edit the files directly to modify the rules, because those changes would be lost when the package is updated, instead we will create new rules in our own files that will override the defaults (more on that later). Since we intend to change the configuration in the modsecurity_crs_10_setup.conf file, we must copy it instead of symlinking so that we can make permanent changes:

sudo cp /usr/share/modsecurity-crs/modsecurity_crs_10_setup.conf /etc/modsecurity/modsecurity_crs_10_setup.conf

Configuring the CRS

Turn On Anomaly Scoring

The configuration in modsecurity_crs_10_setup.conf determines the blocking mode that the core rule set will use - the traditional vs anomaly scoring modes I mentioned in the introduction.

SecDefaultAction determines the action taken by ModSecurity when a rule's action list contains the block keyword. This is unfortunate choice of language in my opinion, because the word "block" implies that the request will be intercepted, but that isn't necessarily the case depending on what SecDefaultAction is set to. I think the language is probably a relic from the way the CRS worked in early versions (traditional mode), where individual rules determined whether to intercept requests, and this parameter was primarily used to decide which phase to intercept the request in, and how to log it (i.e. it always blocked, but the details could be configured differently).

Anyway, here's what the keywords mean:

  • The pass keyword means that ModSecurity will continue to process rules even though this rule matched.
  • To stop processing the transaction and intercept the request, rules can use the keyword deny.
  • In rare cases such as the Denial of Service rules, the rule may specify the drop action, which is even more severe: it closes the TCP connection completely by sending a FIN packet.

The default configuration is:

SecDefaultAction "phase:1,deny,log"

This means that the default block action intercepts the request in the first phase, and writes messages to both the error log and audit log. Since we don't want ModSecurity to decide whether to intercept requests until until all of the rules have been evaluated, locate that line in the file, and change it to this:

SecDefaultAction "phase:2,pass,log"

We also need to turn anomaly_score_blocking on by locating rule 900004 in the modsecurity_crs_10_setup.conf file under the "Collaborative Detection Blocking" heading and uncommenting the first line (the one with SecAction):

SecAction \
  "id:'900004', \
  phase:1, \
  t:none, \
  setvar:tx.anomaly_score_blocking=on, \
  nolog, \
  pass"

So, how does this result in blocking? If you look carefully at any of the rule files (modsecurity_crs_20_protocol_violations.conf would be a good place to start), you will notice that the action section of each rule sets transaction variables like this:

setvar:tx.anomaly_score=+%{tx.notice_anomaly_score},\

And messages like this:

setvar:'tx.%{rule.id}-OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ-%{matched_var_name}=%{matched_var}'

If the rule matches, the anomaly_score variable is being incremented based on the severity of the rule. In this case the anomaly score was incremented by two points, in accordance with the variables set at the start of the transaction by a rule in modsecurity_crs_10_setup.conf under the "Collaborative Detection Severity Levels" heading:

SecAction \
  "id:'900001', \
  phase:1, \
  t:none, \
  setvar:tx.critical_anomaly_score=5, \
  setvar:tx.error_anomaly_score=4, \
  setvar:tx.warning_anomaly_score=3, \
  setvar:tx.notice_anomaly_score=2, \
  nolog, \
  pass"

The rule files are processed in alphanumerical order, so near the end of the chain in modsecurity_crs_49_inbound_blocking.conf you will find this rule, which uses the anomaly_score transaction variable:

# Alert and Block based on Anomaly Scores
#
SecRule TX:ANOMALY_SCORE "@gt 0" \
    "chain,phase:2,id:'981176',t:none,deny,log,msg:'Inbound Anomaly Score Exceeded (Total Score: %{TX.ANOMALY_SCORE}, SQLi=%{TX.SQL_INJECTION_SCORE}, XSS=%{TX.XSS_SCORE}): Last Matched Message: %{tx.msg}',logdata:'Last Matched Data: %{matched_var}',setvar:tx.inbound_tx_msg=%{tx.msg},setvar:tx.inbound_anomaly_score=%{tx.anomaly_score}"
        SecRule TX:ANOMALY_SCORE "@ge %{tx.inbound_anomaly_score_level}" chain
                SecRule TX:ANOMALY_SCORE_BLOCKING "@streq on" chain
                        SecRule TX:/^\d+\-/ "(.*)"

Looks a little scary, but roughly translated this means:

If the anomaly_score is greater than or equal to the inbound_anomaly_score_level, and anomaly_score_blocking is turned on, and there is a transaction variable that looks something like 000001-FOO/BAR, write a log message containing the score and variables that were matched and intercept the request

There's a similar rule for outbound blocking, which runs in a later phase and can intercept requests based on the outbound data to prevent information leakage. With a couple of exceptions such as the Denial of Service protection rules, when the CRS is run in anomaly scoring mode these two rules handle all of the interceptions, and the rest simply modify transaction variables to help them make a decision.

Use Brute Force and DoS Protection

If you enabled the brute force and DoS rule files, you need to modify some SecAction statements in modsecurity_crs_10_setup.conf that are used for these rules.

This SecAction sets some transaction variables that are used in denial of service rules. The rules work by dropping new connections for a period of time (dos_block_timeout) if the number of requests exceeds a limit (dos_counter_thrreshold) in a certain amount of time (dos_burst_time_slice). You can probably leave the variables as they are, just uncomment the SecAction:

SecAction \
  "id:'900015', \
  phase:1, \
  t:none, \
  setvar:'tx.dos_burst_time_slice=60', \
  setvar:'tx.dos_counter_threshold=100', \
  setvar:'tx.dos_block_timeout=600', \
  nolog, \
  pass"

The brute force protection rules work in a similar way. This time you should edit the paths in brute_force_protected_urls so that the login urls for your server are protected, as well as uncommenting SecAction:

SecAction \
  "id:'900014', \
  phase:1, \
  t:none, \
  setvar:'tx.brute_force_protected_urls=#/user/login# #/wp_login.php#', \
  setvar:'tx.brute_force_burst_time_slice=60', \
  setvar:'tx.brute_force_counter_threshold=10', \
  setvar:'tx.brute_force_block_timeout=300', \
  nolog, \
  pass"

GeoIP

Under the GeoIP Database section, there is a SecGetoLookupDb parameter that is used to provide modsecurity with a database it can use to guess the country of clients by their IP addresses.

If you want to use rules that make use of the database, make sure you have the GeoIP database installed:

sudo apt-get update
sudo apt-get install geoip-database

Then uncomment the line and update the path like this:

SecGeoLookupDb /usr/share/GeoIP/GeoIP.dat

Finishing Up

Now we've turned on ModSecurity, configured it to use the CRS, and set some sensible options, all that is left is to reload apache to read the new configuration:

sudo service apache2 reload

If you didn't get any errors, congratulations! ModSecurity is running in DetectionOnly mode. Now comes the difficult part: whitelisting to remove false positives, before you turn ModSecurity on for real.

Before we dive into whitelisting, let's make sure you know how to read the three types of log file that are written by ModSecurity...

Reading the Logs (Error, Audit and Debug)

There are three types of log file that are written by ModSecurity:

  • The error log is the same log file that is used by Apache to write error messages, normally stored at /var/log/apache2/error.log.
  • The audit log is modsecurity's own log file, normally stored at /var/log/apache2/modsec_audit.log, which can contain a complete record of all the data in a transaction, and how ModSecurity processed it.
  • The debug log produces a huge amount of information about how each rule is evaluated and how transaction variables are set and incremented. As a result, it can grow very large very quickly, so you should only turn it on when you have to. Even then, it's a good idea to turn it on selectively for the specific types of request you are interested in. I won't spend any time explaining the debug log, because you hopefully won't have to use it!

Error log

Typically, each rule that matches will write a message to the error log, although this behaviour is controlled by the log or nolog keywords specified in the rule's action list, or in the default action we configured earlier. log means a message will be written to the error log and the audit log, and auditlog means the message will only be written to the auditlog, so if you only want to log in the audit log you must specify nolog,auditlog.

Here's an example of a typical error log message from rule 960911, which verifies that the request line sent by the client follows the format specified in the RFC.

[Sun Mar 13 15:54:12.675099 2016] [:error] [pid 22832] [client 68.224.235.71] ModSecurity: Warning. Match of "rx ^(?i:(?:[a-z]{3,10}\\\\s+(?:\\\\w{3,7}?://[\\\\w\\\\-\\\\./]*(?::\\\\d+)?)?/[^?#]*(?:\\\\?[^#\\\\s]*)?(?:#[\\\\S]*)?|connect (?:\\\\d{1,3}\\\\.){3}\\\\d{1,3}\\\\.?(?::\\\\d+)?|options \\\\*)\\\\s+[\\\\w\\\\./]+|get /[^?#]*(?:\\\\?[^#\\\\s]*)?(?:#[\\\\S]*)?)$" against "REQUEST_LINE" required. [file "/etc/modsecurity/modsecurity_crs_20_protocol_violations.conf"] [line "52"] [id "960911"] [rev "2"] [msg "Invalid HTTP Request Line"] [data "CONNECT login.skype.com:443 HTTP/1.1"] [severity "WARNING"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "login.skype.com"] [uri "/"] [unique_id "VuWNJH8AAQEAAFkwANMAAAAI"]

Let's break this down a bit:

  • The first part of the log entry is a timestamp in the Apache format, followed by the process ID of the Apache handler that handled the request, and the client IP address.
  • Next is the test that was used in the matching rule, and the data that matched it.
  • The third part gives some information about the rule: which file it was loaded from, its position in the file, the unique ID of the rule, and an optional revision of the rule.
  • Following this is some data chosen by the rule. The message is set with the msg parameter (msg:'Invalid HTTP Request Line'); the data part is set with the logdata parameter, and is used to include the relevant part of the request that triggered the rule (logdata:'%{request_line}').
  • A series of CRS specific information follows this: the severity, version of the CRS that the rule belongs to, the rule "maturity" (how heavily tested the rule is, where 10 is the highest), the "accuracy" of the rule (where 10 is highest and scores below 5 indicate that there have been some false positives reported), and finally some tags that can be used to identify the category of attack that the rule was written to catch
  • The hostname and URI sent to the server are the domain name of the site the client wanted the information from, and the unique resource identifier (e.g. the URL) of the resource it wanted.
  • Finally, the unique_id is a unique identifier for that specific request, which can be used to identify the same request in the audit and debug logs.

When a transaction matches more than one rule, more than one log entry may be written, and the rules that do the collaborative blocking based on anomaly scores also write their own log messages on top of that.

The error log is often the best place to look to get an overview of why a transaction has been blocked, but it doesn't contain the whole request sent to the server, and the complete reply. For that, you must look in the audit log.

Audit log

Sometimes, you need more information than the snippets of data selected by the rules that get printed in the error log. The audit log contains a record of complete transactions in text format.

These three lines in /etc/modsecurity/modsecurity.conf set out what gets written to the audit log:

SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ

Lines 1 and 2 mean that only requests with response codes 5XX (server error) or a 4XX (client error), excluding 404 (not found) are written to the audit log. This keeps the log size down, although you will find it still grows pretty quickly!

Line 3 controls which parts of the audit log are written by default for matching requests. The parts available are:

  • A - audit log header
  • B - request headers
  • C - request body
  • D - intended response header (not implemented yet)
  • E - intended response body
  • F - response headers
  • G - response body
  • H - audit log trailer
  • I - reduced multipart request body
  • J - multipart files information
  • K - matched rules
  • Z - audit log footer

Some rules add to the default list of parts logged where that information is particularly useful or relevant, for example many of the outbound rules add part E to the log by specifying ctl:auditLogParts=+E in the action list.

Most of the sections are pretty self explanatory, but I find the most useful section to be part H, which contains one message for each rule that was matched. You can get more information about each section of the log in the ModSecurity Documentation.

When investigating intercepted transactions, you will often find yourself trying to find specific requests in the audit log after identifying them in the access and error logs. You may find it useful to copy the unique_id (which is in both of the other logs), and then after opening the audit log with less, type / and then paste the unique id (ctrl+shift+v) and press enter to search for the request. Here's the audit log entry that matches the error log snippet I used earlier:

--09989b78-A--
[13/Mar/2016:15:54:12 +0000] VuWNJH8AAQEAAFkwANMAAAAI 68.224.235.71 61122 192.168.1.2 80
--09989b78-B--
CONNECT login.skype.com:443 HTTP/1.1
Host: login.skype.com
Proxy-Connection: Keep-Alive

--09989b78-F--
HTTP/1.1 403 Forbidden
Content-Length: 283
Content-Type: text/html; charset=iso-8859-1

--09989b78-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at login.skype.com Port 443</address>
</body></html>

--09989b78-H--
Message: Warning. Match of "rx ^(?i:(?:[a-z]{3,10}\\s+(?:\\w{3,7}?://[\\w\\-\\./]*(?::\\d+)?)?/[^?#]*(?:\\?[^#\\s]*)?(?:#[\\S]*)?|connect (?:\\d{1,3}\\.){3}\\d{1,3}\\.?(?::\\d+)?|options \\*)\\s+[\\w\\./]+|get /[^?#]*(?:\\?[^#\\s]*)?(?:#[\\S]*)?)$" against "REQUEST_LINE" required. [file "/etc/modsecurity/modsecurity_crs_20_protocol_violations.conf"] [line "52"] [id "960911"] [rev "2"] [msg "Invalid HTTP Request Line"] [data "CONNECT login.skype.com:443 HTTP/1.1"] [severity "WARNING"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"]
Message: Warning. Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/etc/modsecurity/modsecurity_crs_30_http_policy.conf"] [line "31"] [id "960032"] [rev "2"] [msg "Method is not allowed by policy"] [data "CONNECT"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"]
Apache-Error: [file "mod_authz_core.c"] [line 866] [level 3] AH01630: client denied by server configuration: %s%s
Stopwatch: 1457884452674435 222736 (- - -)
Stopwatch2: 1457884452674435 222736; combined=222248, p1=221271, p2=0, p3=75, p4=416, p5=395, sr=102, sw=91, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.7 (http://www.modsecurity.org/); OWASP_CRS/2.2.8.
Server: Apache/2.4.7 (Ubuntu)
Engine-Mode: "ENABLED"

--09989b78-Z--

In this case, as you can see in the messages it was the base Apache configuration that denied the request, not ModSecurity (i.e. the request wasn't intercepted). However, two rules did match (960911 and 960032), and we can see some information about those. Here's what a request that is intercepted looks like:

--674ace60-A--
[13/Mar/2016:23:43:25 +0000] VuX7HX8AAQEAAGRzdAsAAAAG 192.168.1.1 41386 192.168.1.2 443
--674ace60-B--
GET /?test=test-attack HTTP/1.1
Host: samhobbs.co.uk
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Referer: https://samhobbs.co.uk/2015/09/example-whitelisting-rules-apache-modsecurity-and-owasp-core-rule-set
Cookie: Drupal.toolbar.collapsed=0; SSESSfoobar; has_js=1
Connection: keep-alive

--674ace60-F--
HTTP/1.1 403 Forbidden
X-Content-Type-Options: nosniff
X-Powered-By: PHP/5.5.9-1ubuntu4.14
Strict-Transport-Security: max-age=15768000
X-Clacks-Overhead: GNU Terry Pratchett
Content-Length: 707
Connection: close
Content-Type: text/html; charset=UTF-8

--674ace60-H--
Message: Access denied with code 501 (phase 1). Pattern match "test-attack" at ARGS:test. [file "/etc/modsecurity/modsecurity_crs_15_customrules.conf"] [line "22"] [id "000001"]
Action: Intercepted (phase 1)
Apache-Handler: application/x-httpd-php
Stopwatch: 1457912605362266 957 (- - -)
Stopwatch2: 1457912605362266 957; combined=300, p1=142, p2=0, p3=0, p4=0, p5=129, sr=43, sw=29, l=0, gc=0
Producer: ModSecurity for Apache/2.7.7 (http://www.modsecurity.org/); OWASP_CRS/2.2.8.
Server: Apache/2.4.7 (Ubuntu)
Engine-Mode: "ENABLED"

--674ace60-Z--

When I started whitelisting, I found that I wanted an easy way to sort through audit logs and inspect groups of requests (e.g. all of the requests from a certain IP address, or all requests that tripped a certain rule). This isn't something that is easily achieved using a text file where the information is split across multiple lines, so I wrote a commandline utility that can read the audit log into a sqlite database. At the time of writing it is in dire need of a rewrite because it was the first thing I wrote in C++, but it does work and it may be useful to you, so feel free to give it a go.

Whitelisting

Now that you have ModSecurity up and running, and you know how to inspect requests, it's time to write a whitelist.

Whitelisting is not easy, but I hope my extensive guide with example whitelisting rules for ModSecurity will prove useful to you. You should also refer to the ModSecurity Reference Manual on GitHub, which is full of useful information.

Once you have written your whitelisting rules, remember to set the SecRuleEngine On and reload apache!

If you run more than one site or web app on your server, you might want to run ModSecurity in two different modes on different parts (e.g. DetectionOnly on one part while you whitelist, On on another part that you have finished whitelisting). You can do this, as well as setting custom log locations, by changing the settings in your virtualhost file. Make sure you use an IfModule guard so that the config is only loaded if ModSecurity is installed/running:

<IfModule mod_security2.c>
        SecRuleEngine On
        SecAuditLog     ${APACHE_LOG_DIR}/samhobbs/modsec_audit.log
        SecDebugLog ${APACHE_LOG_DIR}/samhobbs/modsec_debug.log
</IfModule>

If you have a bit of cash to spare, I would highly recommend buying the ModSecurity Handbook, which you can buy quite cheaply directly from the independent publisher.

Again, you may find my commandline utility that can read the audit log into a sqlite database useful.

If you want to learn more about the types of attack that the CRS protects against, I would recommend hacksplaining.com for a nice easy overview, followed by the OWASP website for articles with more technical detail.

Type: 

Comments

Hi Sam,

Many thanks for the tutorial. I am starting to work through it but will be interrupted shortly for work reasons. Two comments:

1. you write:
sudo cp /etc/apache2/modsecurity.conf-recommended /etc/apache2/modsecurity.conf
but don't you mean:
sudo cp /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

2. in the section about symlinking rules, you write: "If you want to use all of the rule files .."
For a startup installation, which rule file is needed first? Just the "base-rules"? In what circumstances does one need all three (experimental as well as optional)?

John

John,

Thanks for the comments, both very useful. You're right, that was an error (I type some paths so often that my fingers move automatically!). I've added a little bit about choosing rule files. It's up to you really - the more rules you add the longer it will take to whitelist, so what's best for you depends on what your risk tolerance is and which web app(s) you have running.

Sam

Hi Sam,

I have re-read everything and will now start watching the logs before moving on to the whitelisting stage. There is still one uncorrected error:

sudo cp /etc/apache2/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

needs a change in the second argument.

Thanks for the tutorial and the reference to the book. This will keep me quiet for a while...

John

Well spotted :)

Good luck whitelisting. It won't be a quick process, but you're likely to learn lots. Enjoy!

Sam

Hi Sam,

I had a communication from root this morning as follows:

/etc/cron.daily/logrotate:
Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.
error: error running shared postrotate script for '/var/log/apache2/*.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

Here is the output from systemctl:

jmnpi2:~> sudo systemctl -l status apache2.service
● apache2.service - LSB: Apache2 web server
Loaded: loaded (/etc/init.d/apache2)
Active: active (running) (Result: exit-code) since Mon 2016-03-14 08:38:31 UTC; 1 day 1h ago
Process: 22630 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS)
Process: 30188 ExecReload=/etc/init.d/apache2 reload (code=exited, status=1/FAILURE)
Process: 22653 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/apache2.service
├─22668 /usr/sbin/apache2 -k start
├─24389 /usr/sbin/apache2 -k start
└─24390 /usr/sbin/apache2 -k start

Mar 15 06:25:06 jmnpi2 systemd[1]: Reloading LSB: Apache2 web server.
Mar 15 06:25:06 jmnpi2 apache2[30188]: Reloading web server: apache2 failed!
Mar 15 06:25:06 jmnpi2 apache2[30188]: The apache2 configtest failed. Not doing anything. ... (warning).
Mar 15 06:25:06 jmnpi2 apache2[30188]: Output of config test was:
Mar 15 06:25:06 jmnpi2 apache2[30188]: AH00526: Syntax error on line 51 of /etc/modsecurity/modsecurity_crs_16_session_hijacking.conf:
Mar 15 06:25:06 jmnpi2 apache2[30188]: ModSecurity: Disruptive actions can only be specified by chain starter rules.
Mar 15 06:25:06 jmnpi2 apache2[30188]: Action 'configtest' failed.
Mar 15 06:25:06 jmnpi2 apache2[30188]: The Apache error log may have more information.
Mar 15 06:25:06 jmnpi2 systemd[1]: apache2.service: control process exited, code=exited status=1
Mar 15 06:25:06 jmnpi2 systemd[1]: Reload failed for LSB: Apache2 web server.

and as a check, here is the output from configtest:

jmnpi2:~> sudo apache2ctl configtest
AH00526: Syntax error on line 51 of /etc/modsecurity/modsecurity_crs_16_session_hijacking.conf:
ModSecurity: Disruptive actions can only be specified by chain starter rules.
Action 'configtest' failed.
The Apache error log may have more information.

These are the last two lines in that .conf file (lines 51 and 52)

SecRule &SESSION:SESSIONID "@eq 1" "chain,phase:5,id:'981064',nolog,pass,t:none"
SecRule REQUEST_HEADERS:User-Agent ".*" "t:none,t:sha1,t:hexEncode,nolog,setvar:session.ua_hash=%{matched_var}"

which I cannot interpret. Have I messed up something? I followed your suggestion and symlinked optional_rules as well as base_rules.
The Apache error log does not have any information.

Could you advise please?

John

Hi John,

Strange that you didn't get that error when you reloaded apache after setting up modsecurity?

I think you've stumbled upon a bug that was fixed by this commit in the CRS.

For now, I would remove the symlink to the session hijacking file and reload apache. I found the session hijacking rules have a lot of false positives anyway.

Out of interest, which version of the CRS are you using, and which OS/release?

Sam

Hi Sam,

I looked at the commit and will implement that to see it if works. Otherwise, I will remove the symlink as you suggest.

modsecurity version: I followed the second line of your tutorial to install modsecurity. sudo apt-get install libapache2-mod-security2
dpkg-query -l gives modsecurity_crs 2.2.9-1 However, if I visit modsecurity.org they refer to v2.9.1. Is this a variant of 2.2.9-1?

OS version: this is 2015-11-21 raspbian Jessie. This is a Pi 2 +.

John

Hi Sam,

I removed "chain" from /usr/share/....hijacking.conf as indicated in the commit; restarted apache2; apache2ctl configtest no longer complains about a syntax error. I'm a bit worried that this commit is dated Mar 19 2014 which makes the version I have installed more than 2 years old.

Hmmmmm

John

The versions of the CRS and ModSecurity you get with Ubuntu 14.04 are 2.2.8 and 2.7.7, respectively, so you have a newer version than me. Mine doesn't have the chain statement either, so it might even be a bug that was introduced in 2.2.9.

2.9.1 isn't a variant of 2.2.9, I think you're confusing the CRS version and ModSecurity version (modsecurity.org is advertising ModSecurity 2.9, the latest version of the CRS is 2.2.9).

Anyway, there's nothing wrong with the older releases of the CRS, newer packages mostly just have new versions of the experimental rules (for obvious reasons, these change more frequently).

Also, not all versions of the CRS are compatible with all versions of ModSecurity, so you sometimes can't use the newest CRS unless you have the newest ModSecurity, as I mentioned in the article.

If you're really worried about having outdated packages, you can install something like Debian sid (unstable), so you always have the latest stable versions of all of the software. Things will break more often though, because the packages aren't tested together.

Sam

Hi Sam,

Thanks for bringing me uptodate with the versions, especially the distinction between modsecurity and owasp. That was helpful. I visited github and owasp and noticed that the commit date Mar 2014 was the second latest with 18 Feb 2016 being the latest. This is a fix to modsecurity_crs_41_sql_injection _attacks.conf in base_rules.

I am content that everything is now uptodate so I can relax. I am not going to start updating crs packages without your advice to do so.

There have been no further unusual errors after the "wrongful chain action" fix.

John

Sure, there have been many more commits on other branches too - they are working on a 3.0 dev branch, where they have introduced some new methods to catch sql injection attacks (@detectSQLi) and removed some of the special character rules that have so many false positives - that should be nice when it's done.

However, until they tag a new release, it's unlikely that any distros will package any of that (nor should they!).

Sam

Hi Sam,

Hi Sam,

Progress update after monitoring for several days.

After I had started apache2 initially I found a lot of “noise” in modsec_audit.log from 55_application_defects.conf regarding headers, no definition of charset and clickjacking protection. Eventually installed this at the end of apache2.conf:

<IfModule mod_headers.c>
Header set MyHeader "%D %t"
Header always append X-Frame-Options SAMEORIGIN
Header set Content-Type: charset=UTF-8
</IfModule>

Now modsecurity logs are quiet about reports from 55_application_defects.conf

One question about the tutorial and changes to SecDefaultAction "phase:1,deny,log" in modsecurity_crs_10_setup.conf.
I found not one but two SecDefaultAction lines:

SecDefaultAction "phase:1,deny,log"
SecDefaultAction "phase:2,deny,log"

and changed these to:

#SecDefaultAction "phase:1,deny,log"
SecDefaultAction "phase:2,pass,log"

Top tip (amongst many others on your site):

Tutorial on VirtualHost files: Definition of Anti Proxy SPAM virtual host to block script attacks using an IP address instead of FQDN.

(Current records: a visit (from North Africa) that was blocked 62 times in 285 secs. An average rate of one attempt every 4.6secs. Another from Japan saw 101 attempts in 5 secs - stopped by DoS counter? or router’s DoS threshold)

Great stuff…Thanks

John

PS: “Modsecurity getting started” free online pdf is helpful in explaining what “phase” means, and t:none as well as a lot of other interesting material. Apache Security is also a good read - (revealing…)

Thanks for the update :)

That's a good tip with the headers, I think I fixed mine some other way (maybe editing php config?) but then I think the changes I made were clobbered by an update, so I don't think I made the changes to the right file. Your way seems elegant, so I'll give it a go!

Yeah I really like using a default virtualhost to catch spam like that, it takes a lot of load off your server - it won't increase security against targeted attacks, though (that's where ModSecurity is useful!).

You would see something like this in section H of the audit log if the denial of service rules were blocking anyone:

Message: Access denied with connection close (phase 1). Operator EQ matched 0 at IP. [file "/etc/modsecurity/modsecurity_crs_11_dos_protection.conf"] [line "11"] [id "981044"] [msg "Denial of Service (DoS) Attack Identified from 54.171.234.177 (1 hits since last alert)"]
Action: Intercepted (phase 1)

These rules didn't block anyone for a long time on my server, but now there's a comment spam bot going mad and it's getting blocked. I guess that's a good sign, since I haven't seen any legit looking traffic that has been intercepted by these rules :).

Sam

Hi !
Please can someone help me ! I have installed apache2 and modsecurity.I had this error : Failed to start LSB : Apache2 web server
How can I fix that please
Thank you very much

Hi Sam,
See you again. There are one question and two feedbacks.
One question is about brute force protection. According to the code, how to properly set up the address. My code is as below. Isn't it to point to the right entry page? I contrast to yours and feel confused about the "login page" meaning. For example, I have one page to set up on squirrelmail. In the future, I want to install drupal on my RPI. I get a little bit confused on this step. If I use virtual host, it means I will have many host page. How to set up them right on position?

setvar:'tx.brute_force_protected_urls=#/login.jsp# #/partner_login.php#', \

The two matter feedback on the "slr_rules." There are two symbolic links. The directory should be on "sir_rules" not on the "base_rules." Just a feedback. Gratefully thanks your tutorial.

Jeff

Hi Jeff,

Thanks for the corrections.

Since the CRS configration is global, it applies to every virtualhost so you can just list all of the login URLs for every virtualhost.

Wordpress is /wp-login.php
Drupal is /user/login
Squirrelmail is /squirrelmail/src/login.php

Sam

Jeff,

You need the hashes and no commas between urls like this... the backslash at the end escapes the line break and continues the rest of that config on the next line:

setvar:'tx.brute_force_protected_urls=#/squirrelmail/src/login.php# #/user/login# #/wp-login.php#, \

Sam

Hi Sam,
I follow your tutorial to build the mail server and web server~Really gratefully thanks.
Here I want flock log files for monitoring and analyzing them. Meanwhile, your BASH tutorial I haven't read yet due to your abundant and brilliant knowledge existed on there and I need time to digest them. I have been seeing some weird sources coming to my domain. It is including some hash contents existence and even the computer can't read. (There are not plain words.) I check the ip address there are from everywhere globally. The modsecurity is a very discreet tool compared with Fail2ban. It has many complete designs so that you recommend a book for us. ^_^
I may feedback my modsecurity's user report in here. Perhaps, I may bother your brain. Here really thank you.

Hi Sam,

After installing and configuring modsecurity, I attempted to reload apache2, got an error. Here's the log files:

admin@pi-box:~ $ systemctl status apache2.service
● apache2.service - LSB: Apache2 web server
Loaded: loaded (/etc/init.d/apache2)
Active: active (exited) (Result: exit-code) since Wed 2016-04-13 20:50:05 BST; 49min ago
Process: 28828 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS)
Process: 30476 ExecReload=/etc/init.d/apache2 reload (code=exited, status=1/FAILURE)
Process: 28853 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)
admin@pi-box:~ $ journalctl -xn
No journal files were found.

I thought it might be the "chain" thingy in the "optional_rules/modsecurity_crs_16_session_hijacking.conf" file as described in earlier posts, but it didn't help...

Any ideas?

Regards,

Jo

You need to use sudo with journalctl.

You just asked me about a different apache problem...don't try and do too many things at once - I can't help you with two different things if you're changing both at the same time. I would disable modsecurity for now.

Sam

Hi Sam,

Apologies for doing too many things at the same time...got carried away with your tutorials!

How do I disable modsecurity please?

Hi Sam,

Thanks for that...will disable and focus on the other issue...

Regards,

Jo

hi sam...sorry to distub u, i have follow u advice from zero about getting started apache modsecurity debian and ubuntu.when i reload my apache2 server, i have error like below :

apache2.service - LSB: Apache2 web server
Loaded: loaded (/etc/init.d/apache2)
Active: failed (Result: exit-code) since Mon 2016-05-16 17:17:57 WIB; 18s ago
Process: 4065 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)

May 16 17:17:57 fortunaacc apache2[4065]: The apache2 configtest failed. ... (warning).
May 16 17:17:57 fortunaacc apache2[4065]: Output of config test was:
May 16 17:17:57 fortunaacc apache2[4065]: SecReadStateLimit is depricated, use SecConnReadStateLimit instead.
May 16 17:17:57 fortunaacc apache2[4065]: AH00526: Syntax error on line 51 of /etc/modsecurity/modsecurity_crs_16_session_hijacking.conf:
May 16 17:17:57 fortunaacc apache2[4065]: ModSecurity: Disruptive actions can only be specified by chain starter rules.
May 16 17:17:57 fortunaacc apache2[4065]: Action 'configtest' failed.
May 16 17:17:57 fortunaacc apache2[4065]: The Apache error log may have more information.
May 16 17:17:57 fortunaacc systemd[1]: apache2.service: control process exited, code=exited status=1
May 16 17:17:57 fortunaacc systemd[1]: Failed to start LSB: Apache2 web server.
May 16 17:17:57 fortunaacc systemd[1]: Unit apache2.service entered failed state.

what can i do sam?

NB : sorry my english isn't very good

Hi Sam,

I thought it would be better to specify X-FRAME_OPTIONS=DENY as a security measure. Later I discovered that this prevented Squirrelmail from producing a page. Replacing DENY with SAMEORIGIN fixed this problem. But...

While searching for the problem I noticed in modsec_audit.log that although SecAuditLogParts = ABIJDEFHZ, that is, C = Request body is excluded, nevertheless "SecRequestBodyAccess On" is present in modsecurity.conf. As a result, the request body IS included in the log file which clearly shows the user name and password entered in the squirrelmail login page. As follows:

--1d4bb771-C--
login_username=jmn&secretkey=xxxxxxxxxxx&js_autodetect_results=0&just_logged_in=1

Of course, one hopes that modsec_audit.log cannot be read by an intruder, but it is worrying that passwords appear in plain text in this log file as part of the request body.

I changed "SecRequestBodyAccess On" to Off which stopped this. It leaves me the choice to change this if specific debugging is needed.

I thought I should mention this.

John

Hi John,

The modsec_audit.log is owned by root:root with strong file permissions 640, so it's not really an issue unless you use change the permissions.

Part I is a replacement for part C, I think you are seeing C in the log because you have specified part I should be included (see audit log parts in the reference manual):

"I: This part is a replacement for part C. It will log the same data as C in all cases except when multipart/form-data encoding in used. In this case, it will log a fake application/x-www-form-urlencoded body that contains the information about parameters but not about the files. This is handy if you don’t want to have (often large) files stored in your audit logs. "

You can add audit log parts or remove them using ctl:auditlogparts=+I or similar - lots of the CRS rules already do this to add part E when it's relevant:

for file in /etc/modsecurity/* ; do echo $file && cat $file | grep -i ctl:auditlogparts=+ ; done

Changing SecRequestBodyAccess On to Off prevents modsecurity from even reading the request body, which means a lot of the rules will be ineffective. You could change your configuration to SecAuditLogParts ABJDEFHZ (no I) but if you do that the log will be missing a lot of information that is required for debugging.

If you don't want part I for the login page, I suggest you write a rule to remove it at that specific location rather than removing it everywhere. Maybe something like this in modsecurity_crs_15_customrules.conf:

SecRule REQUEST_URI "@eq /squirrelmail/src/login.php" \
  "id:'000005', \
  phase:2, \
  t:none, \
  ctl:auditLogParts=-I, \
  nolog, \
  pass"

Not tested, so you may need to tweak it slightly!

Sam

Pages

Add new comment