Example Whitelisting Rules for Apache ModSecurity and the OWASP Core Rule Set

ModSecurityLogo.png

Recently, I've spent a lot of time tweaking my ModSecurity configuration to remove some false positives.

This tutorial will:

  • Explain the the various methods of altering ModSecurity rules starting with the crudest and working up to the more specific techniques
  • Give some varied examples of custom rules written for exception handling, with a particular focus on the rules distributed by the OWASP Core Rule Set team.

I am calling the process of removing false positives "whitelisting", but technically I should be calling it "exception handling". However, I think more people looking for this information will find it by searching for "whitelisting".

Basic Whitelisting

My previous articles about ModSecurity have discussed some of the cruder methods of removing false positives. When I was new to ModSecurity, I even wrote a BASH script to create a simple whitelist based on ModSecurity messages in the apache error log.

The script made use of the crudest (and easiest) whitelisting technique: removing rules entirely with SecRuleRemoveById, for example this configuration disables rule 981143:

SecRuleRemoveById 981143

Something that wasn't immediately obvious to me is that these are Apache directives and can go in any Apache configuration file. You can put them inside the main Apache config file or a specific VirtualHost file. You can also put them in a separate file entirely, and Include them, like this:

# Include personalised whitelist file for ModSecurity
Include /etc/modsecurity/whitelists/samhobbs.co.uk.conf

One of the features the BASH script I wrote made use of was the ability to combine SecRuleRemoveById with Location and LocationMatch statements to remove rules at specific locations, e.g.:

<LocationMatch "^/comment/[0-9]+/approve$">
SecRuleRemoveById 981143
</LocationMatch>

The difference between Location and LocationMatch is that the latter supports regular expressions, like in the example above.

In addition to SecRuleRemoveById, there are two other directives available, SecRuleRemoveByMsg and SecRuleRemoveByTag
. These do the same thing, but match messages and tags instead of rule IDs, which can be especially handy when you want to do something like remove all SQLi rules in the Core Rule Set.

More Specific Whitelisting

The methods of whitelisting above work well enough for simple cases, but if you over-use them you'll soon end up with the whole of your CRS turned off, or broken! The next level of complexity involves some new directives, discussed below.

SecRuleUpdateActionById

The first of these directives is SecRuleUpdateActionById, which allows us to update the action performed by a rule.

The Core Rule Set is best used in anomaly scoring mode, where the complete chain of rules is evaluated during each phase of request processing, and an overall score is generated to decide whether to block the request or not. However, if you are running ModSecurity in traditional mode then the first rule that matches with the block action will execute the default action, which is normally deny. In this situation, if you wanted to prevent a rule from causing a request to be denied, but still wanted it to be logged, you could do so like this:

SecRuleUpdateActionById 981143 "pass"

Any of the parameters that are part of the action list in a SecRule statement can be updated here. If you want to specify a list, you can do it inside a set of quotation marks. The old list is inherited and overwritten by any new parameters you specify, so you don't need to re-type the whole thing. The one exception I have found is chain - if you are updating a rule that is starting or continuing a chain, you must specify chain again or the subsequent rules will not be treated as part of the chain.

SecRuleUpdateTargetById, SecRuleUpdateTargetByMsg and SecRuleUpdateTargetByTag

Each ModSecurity rule specifies a list of variables to be tested against the operator (e.g. @rx, @beginsWith, @streq etc.). If you want to add additional variables to the list to be inspected, or remove a particular one that is causing a problem, you can use one of these parameters.

The following rule would remove the comment_body argument from the list inspected by rule 981143:

SecRuleUpdateTargetById 981143 !ARGS:comment_body

Note that the arguments you add will be processed on top of the existing list, so if the original list contained ARGS and it was updated by the modification above, the complete list would be ARGS !ARGS:comment_body or in words "all arguments apart from comment body".

Chaining SecRule statements

If you want to use one of the directives above, but only apply it in specific situations, you can create chains of statements. This (made up example) chain removes request cookies from the target list of 981231 at yourdomain.com/admin :

SecRule REQUEST_URI "@beginsWith /admin" \
  "chain, \
  id:'000001', \
  phase:1, \
  t:none, \
  nolog, \
  pass"
  SecRuleUpdateTargetById 981231 !REQUEST_COOKIES

To create a logical "and", just increase the length of the chain - this extended example only updates the target list if the URL begins with /admin and the IP address of the client is 192.168.1.1:

SecRule REQUEST_URI "@beginsWith /admin" \
  "chain, \
  id:'000001', \
  phase:1, \
  t:none, \
  nolog, \
  pass"
  SecRule REMOTE_ADDR "@ipMatch 192.168.1.1" chain
  SecRuleUpdateTargetById 981231 !ARGS:REQUEST_COOKIES

Whitelisting with "ctl" actions

Although it is possible to amend the rule set selectively by creating chains with SecRuleUpdateTargetById and similar parameters, these chains can quickly become difficult to read. My preferred method of mitigating false positives is to use the newer ctl versions of those actions:

  • ctl:ruleRemoveById (or Msg or Tag)
  • ctl:ruleRemoveTargetById (or Msg or Tag)
  • ctl:ruleEngine On|Off|DetectionOnly

I find this method easier on the eye, since everything that starts with SecRule is a condition, and the actions (updating target lists, removing rules etc.) are placed in the action list. If you have two conditions (logical "and") then you have two SecRule statements in the chain, and the ctl action goes in the action list of the last one. The rule below is equivalent to the previous example:

SecRule REQUEST_URI "@beginsWith /admin" \
  "chain, \
  id:'000001', \
  phase:1, \
  t:none, \
  nolog, \
  pass"
  SecRule REMOTE_ADDR "@ipMatch 192.168.1.1" \
    ctl:ruleRemoveTargetById=981173;ARGS:REQUEST_COOKIES

Note that you don't need the ! in front of the argument you want to remove when using ctl:ruleRemoveById.

Remember to put the ctl actions in the last rule in a chain if you want them to be executed when the whole chain matches - the example code below would result in the target list being updated for all IP addresses, because the action is fired before the @ipMatch condition is evaluated:

SecRule REQUEST_URI "@beginsWith /admin" \
  "chain, \
  id:'000001', \
  phase:1, \
  t:none, \
  nolog, \
  pass, \
  ctl:ruleRemoveTargetById=981173;ARGS:REQUEST_COOKIES"
  SecRule REMOTE_ADDR "@ipMatch 192.168.1.1"

Since ctl: actions are evaluated at runtime, these rules should generally go in modsecurity_crs_15_customrules.conf so they are processed before the rule they are amending.

Examples

This section lists some real world examples of rules I wrote for my setup.

Use a custom error file

No matter how careful and thorough you are when creating your whitelist, it is inevitable that some users will be blocked when trying to do legitimate things. Luckily, it is possible to serve clients a custom error document when they are blocked, which is much less frustrating for them than seeing a standard "500: Internal Server Error" with no explanation of what happened.

There are three rules in the CRS that do all the blocking when ModSecurity is deployed in anomaly scoring mode. First, update the actions of these rules in modsecurity_crs_60_customrules.conf:

# update action for inbound blocking rules in modsecurity_crs_49_inbound_blocking.conf
# so that they use the Security Error script
SecRuleUpdateActionById 981175 "chain,deny,status:501"
SecRuleUpdateActionById 981176 "chain,deny,status:501"

# also update the outbound blocking rules
SecRuleUpdateActionById 981200 "chain,deny,status:501"

Next, tell Apache to serve a custom error page for error 501:

ErrorDocument 501 /security-error.php

This directive could go in the same rule file, or if you want to serve different pages for different domains (e.g. with different contact details), you could place it in your VirtualHost file instead. The path is relative to the DocumentRoot defined in the VirtualHost; my site files are stored in /var/www/samhobbs so I created the custom error page at /var/www/samhobbs/security-error.php.

Here's my file:

<?php header("HTTP/1.0 403 Forbidden"); ?>

<html>

<head>
<title>Security Error</title>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
</head>

<body>
<h1>Security Error</h1>
<pre>
Your request has been blocked for security reasons. Please try again in a few minutes.
Should the problem persist, please email me (see the contact page for details) with the
following unique ID, which will help to identify your request:

<?php echo htmlspecialchars($_SERVER["REDIRECT_UNIQUE_ID"], ENT_QUOTES, 'UTF-8'); ?>


Sorry for any inconvenience.

<!--
This comment is here to increase the page size and prevent Internet Explorer from masking
the message. More information is available at the following address:

        http://support.microsoft.com/default.aspx?scid=kb;en-us;Q294807
-->

</body>
</html>

You'll notice that the start of the script sets the status back to 403 - the 501 status is just for internal use/convenience (you only want to serve this page for security errors, not for actual internal server errors as well).

This file has to be readable by Apache, but it does not have to be executable. Mine has ownership root:root and permissions 644.

You will notice that the variables ModSecurity uses for processing rules are made available to the PHP script, prefixed with REDIRECT_, so REDIRECT_UNIQUE_ID is the unique ID that appears in the Apache logs. If someone sends you that ID, it's simple to look it up in the audit log and determine why the request was blocked.

You can see what this page looks like to users by clicking this link: https://samhobbs.co.uk/?test=test-attack

The rule causing the request to be blocked when you click that URL is this:

SecRule ARGS "test-attack"\
  "id:'000001', \
  phase:1, \
  log, \
  deny, \
  status:501"

This is one of the many cool tips I discovered by reading the ModSecurity handbook, although I had to modify the example slightly as the HTTP response codes allowed by default in apache2 seem to have changed since the book was written (the book suggests using 509, but the configuration checking tool throws errors if you try and use that code). I also had to add php tags to the script.

Turn off denial of service counter for a specific URI or path

The denial of service rules in modsecurity_crs_11_dos_protection.conf count the number of requests from each IP address and block new requests when the limit has been exceeded. The rule that does the blocking is 981045:

# Block and track # of requests but don't log
SecRule IP:DOS_BLOCK "@eq 1" "phase:1,id:'981045',t:none,drop,nolog,setvar:ip.dos_block_counter=+1"

This rule checks to see if the ip.dos_block is set, but since it doesn't actually set it itself, removing the rule at a specific URL will prevent the client from being blocked for part of the site, but it will be blocked everywhere else. Not a good solution. Here is the rule that counts the number of requests:

#
# DOS Counter
# Count the number of requests to non-static resoures
# 
SecRule REQUEST_BASENAME "!\.(jpe?g|png|gif|js|css|ico)$" "phase:5,id:'981047',t:none,nolog,pass,setvar:ip.dos_counter=+1"

Removing this rule on parts of the site where you expect lots of requests (like ownCloud) prevents the IP block from being created in the first place:

SecRule REQUEST_URI "@beginsWith /owncloud" \
  "id:'000013', \
  phase:5, \
  t:none, \
  nolog, \
  pass, \
  ctl:ruleRemoveById=981047"

This rule needs to be processed before the file modsecurity_crs_11_dos_protection.conf, so place it in a new file named modsecurity_crs_10_customrules.conf.

Enable extra HTTP request methods and content types for specific locations

The CRS sets the allowed HTTP request methods in modsecurity_crs_10_setup.conf rule 900012, and enforces them in file modsecurity_crs_30_http_policy.conf, rule 960032.

Here is the relevant section from the setup file:

SecAction \
  "id:'900012', \
  phase:1, \
  t:none, \
  setvar:'tx.allowed_methods=GET HEAD POST OPTIONS', \
  setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json', \
  setvar:'tx.allowed_http_versions=HTTP/0.9 HTTP/1.0 HTTP/1.1', \
  setvar:'tx.restricted_extensions=.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .resources/ .resx/ .sql/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/', \
  setvar:'tx.restricted_headers=/Proxy-Connection/ /Lock-Token/ /Content-Range/ /Translate/ /via/ /if/', \
  nolog, \
  pass"

The methods GET, HEAD, POST, and OPTIONS are usually enough for something like a simple blog, and disabling the less well-known methods is probably quite a good idea, because legitimate users won't be using them (bots might, in an attempt to get your web app to react in a non-standard way and expose some vulnerability or sensitive information).

However, if you run ownCloud or a similar service, you will find that calDAV, cardDAV and webDAV together use the methods PROPFIND, REPORT, PUT and MKCOL, which you need to enable to avoid these requests from being blocked by ModSecurity.

The same is true for standard content types: most requests will be fine with the defaults, but I've noticed ModSecurity uses text/calendar for calDAV and application/octet-stream for webDAV.

The following rule will enable these requests by overriding the allowed methods and content types for ownCloud transactions, without allowing them globally:

SecRule REQUEST_URI "@beginsWith /owncloud/remote.php" \
  "id:'000002', \
  phase:1, \
  t:none, \
  setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PROPFIND REPORT PUT MKCOL', \
  setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json|application/octet-stream|text/calendar', \
  nolog, \
  pass"

I put this rule in modsecurity_crs_15_customrules.conf.

Allow missing accept header for a specific location

Pretty much every browser will send an "accept" header to the server to communicate which content types it is happy to receive. For this reason, requests without accept headers are protocol anomalies (not strictly a violation, but a good sign it's not a genuine request) and are flagged by the rules in modsecurity_crs_21_protocol_anomalies.conf. However, you may find that certain programs (particularly ones that send lots of requests like sync clients) omit the accept header. This rule will remove the rule check for ownCloud:

SecRule REQUEST_URI "@beginsWith /owncloud" \
  "id:'000005', \
  phase:2, \
  t:none, \
  ctl:ruleRemoveById=960015, \
  nolog, \
  pass"

Fight comment spam with hidden fields

There are modules that achieve something similar to this for Drupal and probably WordPress: a field is added to comment forms and hidden with CSS. Humans don't see it but bots do, and the dumb ones fill it in causing the comment to be blocked.

This rule will block those requests before they are even seen by the web app, which is preferable (some of these bots are carrying out SQLi probing attacks).

SecRule ARGS:feed_me "!@rx ^$" \
  "id:'000023', \
  phase:2, \
  log, \
  t:none, \
  block, \
  logdata:'SPAM comment detected - hidden filled in comment form', \
  msg:'SPAM comment detected - hidden field filled in comment form', \
  severity:'2', \
  setvar:'tx.msg=%{rule.msg}', \
  setvar:tx.anomaly_score=+20, \
  setvar:tx.%{rule.id}-CUSTOM_RULE/COMMENT_SPAM"

The regex matches "not empty". This one goes in modsecurity_crs_15_customrules.conf.

Turn on the XML processor for CalDAV/CardDAV/WebDAV requests

The CRS has some rules that enable the XML processor for standard XML content types, but many *DAV requests have custom content types and therefore don't trigger these rules.

If ModSecurity inspects an XML request body without processing it with the XML processor, it will spew out a bunch of false positives due to all of the < and > characters (normally you get a "number of special characters exceeded" message), and there's a good chance the request will be blocked. Turning on the XML request body processor will ensure the XML is valid (this check is done in both the CRS and the ModSecurity setup file, so you don't need to add anything), and it will also strip the XML tags leaving just the data, which is run against the other rules.

# turn on the XML processor for webdav and caldav requests
SecRule REQUEST_URI "@rx ^/owncloud/remote.php/(webdav|caldav|carddav)" \
  "chain,id:'000090',phase:1,t:none,t:lowercase,pass,nolog"
    SecRule REQUEST_METHOD "@rx (PROPFIND|REPORT)" \
      "ctl:requestBodyProcessor=XML"

That rule should be placed in modsecurity_crs_15_customrules.conf.

Log all comments, not just ones with high anomaly scores

If there's a specific type of request that you always want to log regardless of whether the request matched any rules or not, you can force ModSecurity to log it. In my case, I wanted a record of all comments posted on the site:

# log all comments to auditlog
SecRule REQUEST_METHOD "@streq post" \
  "chain, \
  id:'000081', \
  phase:1, \
  t:none, \
  t:lowercase, \
  t:normalisePath, \
  msg:'All comments logged to auditlog'"
    SecRule REQUEST_URI "@rx ^(/comment/reply(/\d+)*|/comment/\d+/edit)$" \
      "setvar:'tx.msg=%{rule.msg}', \
      nolog,auditlog

A couple of things worth noting here: be careful when using transformation functions. At first when I wrote this rule, it started SecRule REQUEST_METHOD "@streq POST", but after transforming the data to lowercase the request method is no longer POST, it's post, so the rule didn't match. Now I have a log of all the legitimate comments on the site as well as the ones that tripped the CRS rules.

Missing HTTP only flag in cookies

You will find that you get lots of misconfiguration messages that fill up your audit log if you haven't configured your site to set the httponly flag in your cookies.

This isn't a modsecurity rule, but it's relevant so I decided to include it. You can solve this issue by changing a parameter in /etc/php5/apache2/php.ini:

session.cookie_httponly = 1

And you should no longer see the messages.

Inspect specific requests with the debug log

If your rules aren't working in the way you expected them to, you might want to see detailed information about how modsecurity is processing the relevant requests in the debug log. However, I quickly learned that you don't want to enable debug logging globally because the amount of output produced is huge! Instead, you can turn it on for specific requests (e.g. comments):

# turn on debug logging for comments
SecRule REQUEST_METHOD "@eq POST" id:'000025',phase:1,chain,t:none,nolog
  SecRule REQUEST_URI "@rx ^(/comment/reply(/\d+)*|/comment/\d+/edit)$" \
    log,ctl:debugLogLevel=5

This rule goes in modsecurity_crs_15_customrules.conf.

Remove parameter names from the target list for a specific rule

I wrote this rule for my brother's WordPress site, which is set up to make changes using FTP. The argument names for the login data was triggering the special character limit, so I made an exception for them:

SecRule REQUEST_URI "@beginsWith /authorize.php" \
  "id:'000028', \
  phase:2, \
  t:none, \
  ctl:ruleRemoveTargetById=981173;ARGS_NAMES:connection_settings[ftp][username], \
  ctl:ruleRemoveTargetById=981173;ARGS_NAMES:connection_settings[ftp][password], \
  ctl:ruleRemoveTargetById=981173;ARGS_NAMES:connection_settings[ftp][advanced][hostname], \
  ctl:ruleRemoveTargetById=981173;ARGS_NAMES:connection_settings[ftp][advanced][port], \
  nolog, \
  pass"

Again, this rule goes in modsecurity_crs_15_customrules.conf. I could also have used a regular expression to match all of the argument names.

Remove a group of rules for a specific field (e.g. free-form fields like comment boxes)

The place you're likely to get most false positives is free-form text fields, for example comment bodies. This rule makes use of the fact that the CRS tags rules by type (e.g. SQL injection), which allows us to update the target list for all SQLi and XSS rules so that it excludes the comment_body field:

SecRule REQUEST_URI "@rx ^(/comment/reply(/\d+)*|/comment/\d+/edit)$" \
  "id:'000029', \
  phase:2, \
  t:none, \
  ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/SQL_INJECTION;ARGS:comment_body[und][0][value], \
  ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/XSS;ARGS:comment_body[und][0][value], \
  nolog, \
  pass"

This rule goes in modsecurity_crs_15_customrules.conf.

Put the rule engine in DetectionOnly mode for new apps or a specific path

Initially, your whole ModSecurity installation is likely to be in DetectionOnly mode while you remove false positives. However, you may want to add a new web app to Apache after you have done your initial whitelisting exercise and turned the engine On. In this case it's much better to put the engine in DetectionOnly mode for just the new web app, leaving the engine On for the rest of your site while you do initial testing. You can then comment the rule when you're done.

SecRule REQUEST_URI "@beginsWith /webapp" \
  "id:'000080', \
  phase:1, \
  t:none, \
  ctl:ruleEngine=DetectionOnly, \
  nolog, \
  pass"

Likewise, you may want to turn the rule engine off for the admin backend for your site:

SecRule REQUEST_URI "@beginsWith /admin" \
  "id:'000003', \
  phase:1, \
  t:none, \
  ctl:RuleEngine=Off, \
  nolog, \
  pass"

These rules go in modsecurity_crs_15_customrules.conf.

Allow multiple URL encoding in comments

One of the protocol violation rules for the CRS was being triggered for URLs submitted in the body of node edits for Drupal, especially when the text was different to the URL like this:

<a href="https://samhobbs.co.uk">my website</a>

...which is something I do quite a lot, so I removed it for the specific field:

SecRule REQUEST_URI "@beginsWith /node" \
  "id:'000006', \
  phase:2, \
  t:none, \
  ctl:ruleRemoveTargetById=950109;ARGS:body, \
  nolog, \
  pass"

I put that rule in modsecurity_crs_60_customrules.conf.

Increase the anomaly score threshold for blocking specific requests

The default inbound_anomaly_score_level (the value compared to the transaction score to decide whether to block a request) is set in modsecurity_crs_10_setup.conf with a value of 5. I wanted to raise the threshold for comments to reduce the chances of false positives blocking people from commenting:

SecRule REQUEST_URI "@rx ^(/comment/reply(/\d+)*|/comment/\d+/edit)$" \
  "id:'000014', \
  phase:1, \
  t:none, \
  nolog, \
  pass, \
  setvar:'tx.inbound_anomaly_score_level=10'"

This rule goes in modsecurity_crs_60_customrules.conf.

Remove rules matching common shell commands

The file modsecurity_crs_40_generic_attacks.conf contains rules that look for common shell commands embedded in requests. For most blogs these rules are useful, but since I'm expecting users to post commands in comments on this site, I removed the comment body from the target list. For example, rule 950907 matches wget, curl and cc:

SecRuleUpdateTargetById 950907 !ARGS:/^comment_body/

The regex matches any argument beginning with comment_body, e.g. comment_body[und][0][value].

This rule goes in modsecurity_crs_15_customrules.conf.

Allow multiple linefeeds in comments

Rule 960024 looks for four (4) non-word characters in a row. Two linefeeds (\r\n) in a row looks like (\x0d\x0a\x0d\x0a) and triggers the rule. For this reason you may want to remove it from comment fields:

SecRuleUpdateTargetById 960024 !ARGS:/^comment_body/

Argument names tripping special character limits

Rule 981173 checks for too many special characters and on Drupal sites may be triggered by the names of comment fields, e.g. ARGS_NAMES:comment_body[und][0][value] and ARGS_NAMES:comment_body[und][0][format]. This rule will remove the names of arguments starting with comment_body from that specific rule:

SecRuleUpdateTargetById 981173 !ARGS_NAMES:/^comment_body/

This rule goes in modsecurity_crs_60_customrules.conf.

Notes, References and Tools

If you want to review a large volume of data at once, you might find my commandline utility for reading a modsecurity audit log file into a sqlite database useful.

I started out by running ModSecurity in DetectionOnly mode for a few months, which produced a huge amount of data in the audit logs. Reading it all into a database made it vastly easier to sort and get a feel for which false positives to prioritise.

The ModSecurity reference manual is hosted on the project's github page.

If you don't have the ModSecurity Handbook, I would highly recommend it. It is available directly from the publisher, in paper and ebook formats often cheaper than Amazon. Quite a cool company really, if you buy a paper copy you get the ebook thrown in (including subsequent revisions if the book is updated).

In the variables list, &VARIABLE means the number of that variable, e.g. &REQUEST_HEADERS is the number of request headers.

When writing rules for ModSecurity that are intended to increase an anomaly score, be aware that rule blocking is part of a complicated chain. One of the tests requires there to be a transaction variable that starts with a rule ID number, so you need to add something similar to this snippet to your rules if you want them to match:

setvar:tx.%{rule.id}-CUSTOM_RULE/COMMENT_SPAM

Hopefully you will find these rules useful. If you have any questions or spot a mistake, please leave a comment. Hopefully you won't be blocked :p

Category: 
Type: 

Comments

Hi there,

I have recently started implementing mod_security and have run into the issue that a "PUT /api/" request throws up rule 960032:


PUT /api/lists/stuff/subscribers/id123 HTTP/1.1

Message: Access denied with code 403 (phase 1). Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/etc/httpd/crs-ruleset/owasp-modsecurity-crs/base_rules/modsecurity_crs_30_http_policy.conf"] [line "31"] [id "960032"] [rev "2"] [msg "Method is not allowed by policy"] [data "PUT"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"]
Action: Intercepted (phase 1)
Stopwatch: 1449836127922164 1129 (- - -)
Stopwatch2: 1449836127922164 1129; combined=314, p1=273, p2=0, p3=0, p4=0, p5=41, sr=67, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache
Engine-Mode: "ENABLED"

In my HTTP config I have the inclusion of the OWASP rules and then my custom overwrite-rules:

Include /etc/httpd/crs-ruleset/owasp-modsecurity-crs/modsecurity_crs_10_setup.conf
Include /etc/httpd/crs-ruleset/owasp-modsecurity-crs/base_rules/*.conf
...
# my rules
SecRuleRemoveById 960032 # works but I ideally want to remove it only for URI /api

# This does not work?
SecRule REQUEST_URI "@beginsWith /api" "id:'000005',phase:1,t:none,t:lowercase,ctl:ruleRemoveById=960032,nolog,pass"

Hi,

It's difficult to be sure without knowing all of the filenames for your rules, but one possible reason why your rule wasn't working could be the order the files are read - ctl:ruleRemoveById is triggered at runtime so it should be specified before the rule it is disabling.

In my setup, modsecurity loads files from the /etc/modsecurity directory. The default rules are symlinked there, and the filenames look like this:

sam@samhobbs:/etc/modsecurity$ l
modsecurity_35_bad_robots.data@
modsecurity_35_scanners.data@
modsecurity_40_generic_attacks.data@
modsecurity_42_comment_spam.data@
modsecurity_42_customrules_comment_spam.data
modsecurity_50_outbound.data@
modsecurity_50_outbound_malware.data@
modsecurity.conf
modsecurity.conf-recommended
modsecurity_crs_10_customrules.conf
modsecurity_crs_10_ignore_static.conf@
modsecurity_crs_10_setup.conf
modsecurity_crs_10_setup.conf.bak
modsecurity_crs_11_avs_traffic.conf@
modsecurity_crs_11_dos_protection.conf@
modsecurity_crs_11_slow_dos_protection.conf@
modsecurity_crs_13_xml_enabler.conf@
modsecurity_crs_15_customrules.conf
modsecurity_crs_16_authentication_tracking.conf@
modsecurity_crs_16_session_hijacking.conf@
modsecurity_crs_16_username_tracking.conf@
modsecurity_crs_20_protocol_violations.conf@
modsecurity_crs_21_protocol_anomalies.conf@
modsecurity_crs_23_request_limits.conf@
modsecurity_crs_25_cc_known.conf@
modsecurity_crs_30_http_policy.conf@
modsecurity_crs_35_bad_robots.conf@
modsecurity_crs_40_generic_attacks.conf@
modsecurity_crs_41_sql_injection_attacks.conf@
modsecurity_crs_41_xss_attacks.conf@
modsecurity_crs_42_comment_spam.conf@
modsecurity_crs_42_customrules_comment_spam.conf
modsecurity_crs_42_tight_security.conf@
modsecurity_crs_45_trojans.conf@
modsecurity_crs_46_av_scanning.conf@
modsecurity_crs_47_common_exceptions.conf@
modsecurity_crs_47_skip_outbound_checks.conf@
modsecurity_crs_48_local_exceptions.conf.example@
modsecurity_crs_49_header_tagging.conf@
modsecurity_crs_49_inbound_blocking.conf@
modsecurity_crs_50_outbound.conf@
modsecurity_crs_55_application_defects.conf@
modsecurity_crs_55_marketing.conf@
modsecurity_crs_59_outbound_blocking.conf@
modsecurity_crs_60_correlation.conf@
modsecurity_crs_60_customrules.conf

Since the rule you are removing is in modsecurity_crs_30_http_policy.conf, the custom rule to remove it would have to be read before that file by apache (e.g. you could use modsecurity_crs_15_customrules.conf).

Instead of removing the rule, I think you should add PUT to the list of allowed methods at that location. Have a look at the example "Enable extra HTTP request methods and content types for specific locations". You could do something like:

SecRule REQUEST_URI "@beginsWith /api" \
  "id:'000002', \
  phase:1, \
  t:none, \
  setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT', \
  nolog, \
  pass"

Sam

Thank you so much. My config is slightly different - i.e. as part of my Apache config, I include a separate config file which has our virtuals defined, in there I have a global section and I just needed to switch the rule before the base includes:

Include /etc/httpd/crs-ruleset/owasp-modsecurity-crs/modsecurity_crs_10_setup.conf

# Allow API access - see https://samhobbs.co.uk/2015/09/example-whitelisting-rules-apache-modsecurity-and-owasp-core-rule-set
# Needs to be before other rules
SecRule REQUEST_URI "@beginsWith /api" "id:'001000',phase:1,t:none,t:lowercase,ctl:ruleRemoveById=960032,nolog,pass"

Include /etc/httpd/crs-ruleset/owasp-modsecurity-crs/base_rules/*.conf

# Use OWASP CRS to detect & block malicious attacks
SecRuleEngine On

Cool, glad it's working now.

Here's some more related info to bear in mind (I did some more digging after replying). I thought your problem might also have been due to having phase 1 rules inside a virtualhost (I wasn't sure if apache has already decided which virtualhost the transaction belongs to before phase 1 is executed) but the documentation on the "request headers" phase (phase 1) says this:

Rules in this phase are processed immediately after Apache completes reading the request headers (post-read-request phase). At this point the request body has not been read yet, meaning not all request arguments are available. Rules should be placed in this phase if you need to have them run early (before Apache does something with the request), to do something before the request body has been read, determine whether or not the request body should be buffered, or decide how you want the request body to be processed (e.g. whether to parse it as XML or not).

Note : Rules in this phase can not leverage Apache scope directives (Directory, Location, LocationMatch, etc...) as the post-read-request hook does not have this information yet. The exception here is the VirtualHost directive. If you want to use ModSecurity rules inside Apache locations, then they should run in Phase 2. Refer to the Apache Request Cycle/ModSecurity Processing Phases diagram.

And an old post on the ModSecurity blog says this:

Configuration contexts other than <VirtualHost> cannot hold phase 1 rules.

Worth remembering when you're adding more rules inside a virtualhost file - don't stick anything like that inside <Location> blocks!

I still think you should consider updating the allowed methods to include PUT instead of removing that rule altogether, at the moment you are allowing all kinds of other methods (PROPFIND, REPORT, PUT, MKCOL etc.) through to the web app if the URI matches.

Sam

My sits outside the VirtualHost definition and the rules I customise are global to all virtuals. I sofar did not have the need to provide custom rules within a VirtualHost.

I do have a SecRule further down in my config where I limit the methods based on the request URI. Initially I hoped that I could just expand the allowed methods, but I also did not want to make changes to files within OWASP files.

Ah sorry, I misunderstood you. I was assuming the config file where you have defined your virtualhosts just contains the virtualhost blocks with no other global directives (e.g. the Debian/Ubuntu default setup).

"Initially I hoped that I could just expand the allowed methods, but I also did not want to make changes to files within OWASP files."

You don't need to make a change to the file directly, the example I pointed you to overwrites the original transaction variables set by the CRS rule, but only for that specific URI. And it leaves the CRS config file untouched (the new custom rule is placed in a separate file).

Sam

Hi Sam,
I tried blocking the uri using ARGS (https://samhobbs.co.uk/?test=test-attack) and its working fine. Thanks.
I wanted to know how to block absolute URL, please let me know.
i tried several ways like HTTP_REFERER, REQUEST_URI and REQUEST_RAW_URI but nothing worked.
Regards,
Muruli

This should work for the URI, you can chain it with a test for the host if you want a logical AND for "yourdomain.com" and "/some-uri":

SecRule REQUEST_URI "@eq /some-uri" \
  "id:'000002', \
  phase:1, \
  log, \
  deny, \
  status:501"

I'm assuming you're just experimenting to learn? If not, you would probably be better off doing this kind of thing in the apache configuration file instead (see the docs on access control).

Sam

Hi Sam,
Thanks for the reply.
I tried it and it is blocking everything in my app path. i want it to block only specific uri like any path to html file or php file, etc.

Hi Sam,
i gave "/test/test.html" then it is blocking and working as expected.
but If i give http://ipaddress/test/test.html, then it is allowing.
Please help me in resolving this, i want it in both ways.

Which did you use, a chain or the rule exactly as I wrote it? It sounds like you don't want a chain (you want it to block that URI regardless of host)?

Try the same rule but with @streq instead of @eq - I should have said @streq first time because we're comparing strings. @eq is for numerical comparisons.

Sam

Hi Sam,
Thanks for your kind reply.
I tried with @streq but no luck. Anyways without https:// it is working fine.
is it a valid request to mention https://IpAddress/path in modsecurity conf files to block ?

I can't think why that wouldn't work - URI matching should strip the protocol and domain parts of the address, so https://foo.com/bar and http://1.2.3.4/bar should both match "REQUEST_URI @streq /bar" (see the documentation for REQUEST_URI).

As the docs say, no transformations are done to prevent evasion by default, so you could use the transformations in their example to prevent people from evading the rule:

t:none,t:urlDecode,t:lowercase,t:normalizePath

How have you enabled modsecurity? If you've done it in your virtualhost configuration, make sure you have the same config for HTTP and HTTPS (is modsecurity enabled on one and disabled on the other?).

Sam

Hi Sam,
I have configured apache to ssl and enabled modsecurity for it.
As i mentioned earlier if i do not mention ipaddress then it is blocking perfectly.
Also as per documentation, if domain name is provided on the request line we have to use REEQUEST_URI_RAW. i tired this option also but still no luck.
let me know if you have used it anywhere.
Thanks

What the docs are saying is that if you want a specific rule for just one domain name, you should use REQUEST_URI_RAW. Since you want the rule to apply regardless of which domain name is used, you should use REQUEST_URI.

What I think is happening is the request sent to the IP address is being served by the default virtualhost, which has different modsecurity configuration.

I tend to create a blocking default virtualhost as described in my virtualhost tutorial, which ensures that people can only access your site with the configuration you expect, and everything that doesn't match one of the virtualhosts is blocked.

Sam

Hi Sam,
I can list out the url in custom rules file to block url which essentially becomes like a black listing.
In order to make a white list which allows only mentioned url. What are the changes required in custom rules file ?

What do you mean? There is no blacklist, requests are all evaluated individually (apart from some rules like the denial of service rules, which store persistent data).

If you want to turn modsecurity off for a certain IP address, try:

SecRule REMOTE_ADDR "@ipMatch 192.168.1.1" \
  "id:'000001', \
  phase:1, \
  t:none, \
  ctl:ruleEngine=Off, \
  nolog, \
  pass"

Writing a rule that would do this based on an IP address submitted via HTTP seems like a bad idea - what happens when someone figures out they can whitelist themselves by doing that?

You could probably build in some password authentication, but a better way to do this is to write a rule to correct whichever false positive causes the request to be blocked in the first place.

Sam

Hi Sam,
Is it possible to have port # whitelist so that no one can access the webserver in other ports. This is the case if we are using proxy server so I tried "SecRule SERVER_PORT "^80$" "id:69, deny, status:403" but still i am able to access the original server with redirected port.
Please let me know if i can block the other ports using Modsecurity rule.

That's a bit of a complicated way of doing things, I would do this in the apache virtualhost file instead - only have the virtualhost serve content on the port you want people to connect to by changing this:

<VirtualHost *:80>

To this:

<VirtualHost *:port>

Sam

Add new comment