Beware Apache2 mod_proxy

While tinkering with the settings for my site, I discovered an Apache module called mod_proxy.

I was interested in it because I am running two webservers – one for www.samhobbs.co.uk and one for webmail, and I wanted to redirect traffic from one part of the site to the webmail server using ProxyPass.

Unfortunately, I was over-enthusiastic in my explorations and made an error: I enabled my server to be used as an open proxy, and attracted thousands and thousands of dodgy requests from around the world.

What this meant is that anyone could connect to my server and use it to visit web pages whilst concealing their true identity: the pages visited would only see my IP, not theirs.

What I did wrong.

What I did was stupid, embarrassingly so. It wouldn’t have happened if I had read the manual, or even just paid attention to the useful information in the config files.

On Debian and Ubuntu systems, the settings for mod_proxy are located here: /etc/apache2/mods-enabled/proxy.conf, and look something like this:

<IfModule mod_proxy.c>

# If you want to use apache2 as a forward proxy, uncomment the
# 'ProxyRequests On' line and the <Proxy *> block below.
# WARNING: Be careful to restrict access inside the <Proxy *> block.
# Open proxy servers are dangerous both to your network and to the
# Internet at large.
#
# If you only want to use apache2 as a reverse proxy/gateway in
# front of some web application server, you DON'T need
# 'ProxyRequests On'.

ProxyRequests off
# <Proxy *>
#        AddDefaultCharset off
#        Order deny,allow
#        Deny from all
#        #Allow from .example.com
#</Proxy>

# Enable/disable the handling of HTTP/1.1 "Via:" headers.
# ("Full" adds the server version; "Block" removes all outgoing Via: headers)
# Set to one of: Off | On | Full | Block
#ProxyVia Off

</IfModule>

Line 13 was the unlucky one here. You don’t need to enable ProxyRequests to use ProxyPass, but I naively changed line 13 to ProxyRequests on, and didn’t even restrict its usage. Note that the config file actually warns against doing this – it was really stupid.

Anddd… hey presto: my RasPi was now an open proxy ready to be exploited by the world’s tech savvy miscreants and script kiddies.

Some Signs that your server is being used as an open proxy:

The time it took me from changing the setting to realising something was wrong was about a week. The number of requests was small at first, and at some point someone must have realised the server could be used as a proxy because the number of requests increased dramatically, which is when I noticed.

There were a few things that tipped me off. The first was that my router slowed down dramatically. A quick look at the admin page showed that the number of active IP connections was maxed out.

You can check the number of active IP connections on a DD-WRT router on the web interface under Status --> Router. The screenshot below is my router during normal activity. When it was an open proxy, the number of IP connections was about 4090.

Active IP connections during normal use

The image above shows the number of active connections during normal use. Other routers should have a similar function to this. If you are renting hosting space from somewhere else then you won’t be able to use this an indicator because your server isn’t behind your router.

The second thing that tipped me off was that when I did a google search I got this message: “Our systems have detected unusual traffic from your computer network” and a CAPTCHA box to confirm that I was a human. Presumably, this is because some kind of automated service was using the open proxy to perform google searches.

By this time I was very concerned, and had shut off port forwarding on my router while I investigated.

I took a look at the apache access logs confirmed the worst: thousands and thousands of requests had been made to the server in the last few days, some of them to really dodgy porn sites and other pages you really wouldn’t want people to think you had visited. Just look at the size of the logs and the dates – the requests to the server for hosts that weren’t defined as virtual hosts in /etc/apache2/sites-enabled/000-default had exploded over the last couple of days.

Apache access logs

The content of the log files sheds more light on the requests the server was receiving:

Contents of the Apache access logs

Notice the two numbers after the GET address. The first one is a code that will tell you whether or not the request succeeded. If the number is 200, the request was successful. Server error is 500. Other codes can be checked here: http://www.w3.org/Protocols/rfc2616/rfc2616.txt.

What to do if you’ve made the same mistake I did

While you sort out the server configuration, turn off port forwarding on your router’s admin page. While there were 4000 active IP connections to the Pi I couldn’t even open a SSH session, either to the local IP address or the global one. I’m not sure whether this is because the active IP sessions were maxed out, or because the server was overloaded, but either way it’ll stop those unwanted connections until you can sort out the configuration.

First, change the setting “ProxyRequests” to “off” in /etc/apache2/mods-enabled/proxy.conf.

This will stop your server being used as an open proxy, but when someone requests a page on some other site (www.someotherdomain.com/home) your site will look for a corresponding page on your website, and serve www.yourdomain.com/home. This is obviously a waste of your resources, as whoever was trying to use your server as a proxy has no interest in your site at all.

The way to get around this is to add to your VirtualHosts in /etc/apache2/sites-enabled/000-default.

<VirtualHost *:80>
        ServerName default.only
        <Location />
                Order allow,deny
                Deny from all
        </Location>
</VirtualHost>

<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        ServerName www.samhobbs.co.uk:80

        DocumentRoot /var/www
</VirtualHost>

The first virtual host (default.only) must go before your other VirtualHosts, and means that if anyone requests a domain that is not defined as a VirtualHost, the request will be rejected instead of sending the corresponding page for your actual site.

Now reload the apache configuration:

sudo service apache2 reload

…and enable port forwarding on your router again. Because proxy requests now return an error code, requests from automated services soon drop off, and anyone trying to connect manually will also lose interest. The number of active IP connections started out around 300, and have now dropped off to below 100.

This should solve your problem. I feel really stupid for making the mistake in the first place, but at least I’ve learned something about Apache. Next time I’ll RTFM!

Type: 

Comments

Hi Sam,

Looking through this tutorial...I don't seem to have this module (I assume it has to be installed?)

dmin@pi-box:/etc/apache2/mods-enabled $ ls
access_compat.load authn_core.load authz_user.load deflate.load filter.load mpm_prefork.conf php5.conf setenvif.load status.conf alias.conf authn_file.load autoindex.conf dir.conf headers.load mpm_prefork.load php5.load socache_shmcb.load status.load alias.load authz_core.load autoindex.load dir.load mime.conf negotiation.conf rewrite.load ssl.conf
auth_basic.load authz_host.load deflate.conf env.load mime.load negotiation.load setenvif.conf ssl.load

Is this something for me to be concerned about?

Please advise.

Thx,

Jo

You almost certainly have the module, because it's bundled with apache2 (you can use dpkg's -S flag to search for the package that provides a file on your system):

$ dpkg -S /etc/apache2/mods-available/proxy.load
apache2: /etc/apache2/mods-available/proxy.load

But you don't need to worry about it, because it's not enabled by default. To enable it, you would have to run:

sudo a2enmod proxy

which would create a symlink in the mods-enabled folder, and load the module.

Sam

Add new comment