If you have a homelab or a small web server for your tool, you probably know the Fail2ban software. It’s a tool capable of taking input from another software (usually log) to flag the IP doing bad stuff and block them for a defined amount of time (usually using iptables). It’s really not a tool for big production-grade projects, but for small personal stuff, it does the job quite well!

But as long as you have multiple VMs running for different tools or projects, having all of them individually fully exposed to the internet isn’t a good idea. It costs some extra IPv4 allocation (assuming you aren’t IPv6 only), and securing them may require more time and effort. Most people end up with some kind of proxy. It can be a single Nginx/Apache2, a haproxy, or a traefik… it doesn’t really matter. What does matter is that now, all of your requests will come from the IP of that proxy and no longer the one trying to mess with your stuff. Does this make Fail2ban useless? Dropping the ban hammer on your own proxy IP isn’t a good idea, but how can you selectively ban something that comes from a single server?

Requirement and Disclaimer

In order to have this to work, you’ll need:

• a proxy server who support adding the Real IP/Forwarded For header.
• an application server with:
  • Iptables.
  • an application that support trusted proxy and log the real ip when something goes wrong.
  • Nginx or Apache2 as a front to that app.

I would like to push on the fact that this is some “clever hack” i found while digging and in no way the “proper way”. In a production environment, i would rely on some WAF, or adding the troublesome ip to a KV datastore and dynamically update multiple haproxy to drop those at the gate of our network.

Step 1: Getting the Real IP

So, first thing first, you need to get the offending ip back to your application server. For this, there are multiple different standards using HTTP header. In my case i’ll use the X-Forwarded-For and X-Real-IP header. On your proxy server, you first want to make sure those aren’t already set. Someone could trick you into banning other ip if they could set them themselves. You then set them correctly to contain the correct ip. Here is an example for Haproxy, but i’ll leave you to the doc of your Proxy of choice on best practice to securely manage those header.

frontend web_plain
    mode http
  ...
    http-request del-header X-Forwarded-For
    http-request del-header X-Real-IP
    option forwardfor except 127.0.0.0/8
    acl from_local src 127.0.0.0/8
    http-request set-header X-Real-IP %[src] unless from_local
  ...

Step 2: Recover that IP with your Web server

Now that the IP of your enduser is sent in the HTTP request header, you will have to retrieve it before you can work with it. We will look at Apache2 and Nginx, but most other web apps have a way to do this as well. To do so we need to edit the log format to include/replace the remote IP with the one from the header. This way, Fail2Ban can still perform it’s log-parsing job.

Apache 2

With Apache 2, the easiest way is to enable the mod_remoteip module.

➜  ~ sudo a2enmod remoteip
Enabling module remoteip.
To activate the new configuration, you need to run:
  systemctl restart apache2

Then you can add in your vhost which header to use and what proxy to trust.

<VirtualHost *:80>
  ...
    RemoteIPHeader X-Forwarded-For
    RemoteIPInternalProxy 10.10.1.25
    RemoteIPInternalProxy 10.20.1.25
  ...
</VirtualHost>

After a restart, you should start seeing the real IP in your log.

Nginx

With Nginx, the module http_realip needs to be compiled and enabled. On most distribution it is by default but your mileage may vary.

Then, just add the following line in your vhost

...
set_real_ip_from    10.10.1.25;
set_real_ip_from    10.20.1.25;
real_ip_header      X-Forwarded-For;
...

Step 3: Setup Fail2ban

Great ! You are now generating log that contains accurate ip data. You can now enable Fail2ban as you normally would, right? Well … no.

Creating customs Fail2ban action

You see, Fail2ban is composed of 2 important part in our cases, the filter, and the action. The default filter should work fine for now, but the action is usually to add the offending IP to an Iptables drop list, and that IP never reach the firewall. Unless you wants to install a layer 7 Firewall (or a WAF) locally, you will need to change the way you “ban” your IP.

I’ve created 2 action files you can get from the following Gist:

• apache2-map.conf
• nginx-map.conf

The goal is to create a list of ip that each web server can understand and block at its level. You may need to change the user and/or path of the file depending on your distribution default.

On Apache2 it’s a simple list of <ip> - we save to /etc/apache2/banned-hosts with the www-data user and using 0640 permission (see the actionstart of the file). We use a printf to add line to it (actionban) and sed to remove line from it (actionunban).

On Nginx, it’s a bit more complicated, we need to create a geo map, we use the actionstart to create a template that we will fill with ip down the line.

geo $bad_user {
  default 0;
}

We save that file to /etc/nginx/conf.d/banned-hosts.conf and every time it is changed, Nginx will have to be reloaded. Then, adding and removing lines is done with simple sed command with line looking like <ip> 1;

Creating the Jail

This step is quite up to you, actually. If you know Fail2ban, you don’t need me to tell you there is no best config, it all depends on your use-case and need. But for the sake of completeness, i will provide an example jail that worked well for me.

[<Apache or Ningx>_RealIP]
enabled = true
filter = <Creating a filter for your app is out of the scope of this post. See https://fail2ban.readthedocs.io/en/latest/filters.html>
logpath = /var/log/apache2/access.log

sender = fail2ban@example.com
destemail = your-mail-here@example.com
mta = sendmail

action = apache2-map[name=apache2-map] # or nginx-map[name=nginx-map]
         %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s"]

bantime = 30d

# Maximum amount of login attempts before IP is blocked
maxretry = 3

Step 4: Blocking the ip

Last step, we now need to use the web server to block the user. This time it’s Apache2 that is a bit more complicated than Nginx.

Apache 2

We want to use the Rewrite Engine to redirect the user to a Forbidden error. Enable mod_rewrite like so:

➜  ~ sudo a2enmod remoteip
Enabling module rewrite.
To activate the new configuration, you need to run:
  systemctl restart apache2

Then add the following to your vhost:

<VirtualHost *:80>
  ...
    RewriteEngine On
    RewriteMap hosts-deny txt:/etc/apache2/banned-hosts
    RewriteCond ${hosts-deny:%{REMOTE_ADDR}|NOT-FOUND} !=NOT-FOUND [OR]
    RewriteCond ${hosts-deny:%{REMOTE_HOST}|NOT-FOUND} !=NOT-FOUND
    RewriteRule ^(.*)$ /error.html [F,L]
  ...
</VirtualHost>

Using the Flag in in the RewriteRule we redirect the user to a 403. Sadly, as far as I know, there is not a simple way to just drop the packet silently with Apache2.

Nginx

Here, you will just have 3 lines to add in your Nginx server:

server {
  ...
    if ($bad_user) {
        return 444; #Close connection silently
    }
  ...
}

Wrap-up

Hopefully, by following this post you managed to setup the 3 crucial step to make Fail2ban work behind a web proxy

1. Collecting the real user IP in your log
2. Configuring Fail2ban to build a list of abuse IP
3. Creating a filter to block those IP

If you have anything you think i should add, edit or correct, don’t hesitate to reach out !