Rate limiting with HAProxy: slowing bots down before they reach PHP

A look at the access log of a shop: a single IP sent a little over 300,000 requests in six and a half hours. That is about 13 per second, through the night and without a break. A person loads a page, reads it and clicks on. On average that is one page every 10 to 20 seconds.

What helps against this kind of non-stop traffic is a rate limit: at most X requests per IP in a time window, and everything above that is turned away.

Where does the limit belong?

There are three places it could go. Only one really fits:

• Firewall: It sees connections, not individual HTTP requests. Under HTTP/2, thousands of requests travel over a handful of connections. To the firewall, that looks harmless.
• Application (PHP): This is too late. By the time PHP rejects the request, a web server worker and a PHP process are already taken. On top of that, the application never sees images, CSS or JS files.
• HAProxy: It sees every request with its path and sender. It can count per IP and answer with 429 Too Many Requests before the web server does anything at all.

If there is an HAProxy in front of the web servers anyway, that is the right place.

What to count?

Not every request costs the same. An image comes ready-made off the disk. A page starts PHP and queries the database. On top of that, a single page view easily loads 50 images at once.

So there are two separate counters: a strict one for pages and PHP, and a generous one for static files.

The counting tables

HAProxy comes with what it calls stick tables for this. They are small key-value stores in the proxy's memory:

backend ratelimit_table_page_rate
    stick-table type ipv6 size 1m expire 1m store http_req_rate(10s)

backend ratelimit_table_static_rate
    stick-table type ipv6 size 1m expire 1m store http_req_rate(10s)

The backend is only a container here. HAProxy allows exactly one stick table per section, so two tables need two sections. There are no servers in them, and no request is ever sent there.

• type ipv6: The key is an IP address. IPv4 fits as well and is stored as ::ffff:1.2.3.4.
• size 1m: at most one million IPs at the same time.
• expire 1m: An IP that sends nothing for a minute is dropped. That is just housekeeping.
• store http_req_rate(10s): the value per IP, namely the number of requests in the last 10 seconds.

http_req_rate is built in. Nothing is programmed here; you only set the window it counts over.

The rules in the frontend

The actual flow sits in the frontend that takes the requests:

frontend https
    # ... bind, certificate and so on

    acl is_static_request path_end -i .jpg .jpeg .png .gif .webp .svg .ico .css .js .woff .woff2 .mp3
    acl is_ratelimit_allowlisted src -f /usr/local/etc/haproxy/ratelimit-allow.lst
    acl is_ratelimit_excluded_path path_beg -f /usr/local/etc/haproxy/ratelimit-excluded-paths.lst

    http-request track-sc0 src table ratelimit_table_page_rate if !is_static_request !is_ratelimit_allowlisted !is_ratelimit_excluded_path
    http-request track-sc1 src table ratelimit_table_static_rate if is_static_request !is_ratelimit_allowlisted !is_ratelimit_excluded_path

    acl is_page_rate_over_limit sc_http_req_rate(0) gt 50
    acl is_static_rate_over_limit sc_http_req_rate(1) gt 300

    http-request capture str(page) len 6 if is_page_rate_over_limit
    http-request capture str(static) len 6 if is_static_rate_over_limit

    http-request deny deny_status 429 if is_page_rate_over_limit || is_static_rate_over_limit

    default_backend webservers

That looks like a lot. But it is only two building blocks taking turns.

acl: a named condition

An acl line creates a kind of variable that is true or false. It is only evaluated when a rule uses it, and then anew for every request. Like a function definition, it does not matter where it stands.

The structure is always the same: first fetch a value, then compare it.

• path_end -i .jpg …: The path of the request (without the query string) ends in one of these extensions. -i ignores upper and lower case.
• src -f file: The sender's IP is in the file. Networks such as 10.0.0.0/8 work too.
• path_beg -f file: The path starts with an entry from the file.

The list files hold one entry per line. Lines that start with # are comments.

http-request: a hook into the request

http-request hooks into the handling of every incoming request before it goes to the web server. The lines run from top to bottom. After http-request comes the action, and optionally an if with conditions at the end. Several ACLs in a row are an AND, ! negates, || is an OR.

The counterpart is called http-response and runs on the web server's answer.

track-sc0: count and remember

http-request track-sc0 src table ratelimit_table_page_rate if !is_static_request ...

This reads as: for a request that is not a static file, add 1 in ratelimit_table_page_rate under the sender's IP. The table row then ends up in sc0.

sc0, sc1 and sc2 are fixed names you cannot make up yourself. By default there are exactly three ("sticky counters"). They only apply to the current request. An image fills sc1, a page fills sc0, and the other slot stays empty.

The value in sc0 is the count after the +1. So the current request is already included.

Two more conditions

acl is_page_rate_over_limit sc_http_req_rate(0) gt 50

sc_http_req_rate(0) reads the count from sc0. The 0 is the slot number, not a table name. gt stands for "greater than". There are also ge, lt and le. With gt 50, the first 50 requests in 10 seconds go through, and from the 51st on the condition is true.

capture: a marker in the log

http-request capture str(page) len 6 if is_page_rate_over_limit

capture does not write a log entry of its own. It appends a text to the request's normal log line, in curly braces:

... 200 78439 - - --NI 8/8/7/7/0 0/0 {page|} "GET /some/page HTTP/2.0"

Each capture line takes a fixed field, separated by |. If the condition does not apply, the field stays empty. A normal request therefore has {|}, a page that came too fast {page|}, and a static file that came too fast {|static}. grep -F '{page|}' finds every hit.

deny: turn it away

http-request deny deny_status 429 if is_page_rate_over_limit || is_static_rate_over_limit

Above either limit, HAProxy answers itself with 429 Too Many Requests. The web server never sees the request. Without deny_status it would be a 403. But 429 is the right code: well-behaved crawlers read it as "slower" rather than "forbidden".

Exceptions

Not everyone who sends a lot of requests is a problem:

• Internal networks, for example the office, the warehouse or a PDF generator that fetches pages, belong in the IP allowlist.
• Webhooks from payment providers belong in the path exceptions. A 429 there means, at worst, that a payment is not booked.
• Search engine crawlers publish their IP ranges. You can add those to the allowlist as well.

Both lists are read at startup. After a change, HAProxy needs a reload.

Count first, block later

Before the deny line goes live, it should stay commented out for a few days. The capture markers then show whom the limit would hit, without a real visitor seeing an error page.

For the limits, it is worth looking at the access log you already have: how many page views does one IP make at most in a 10-second window? In projects where this was measured, real visitors came to a little over 30 for pages and PHP. The reason is the AJAX calls every page loads afterwards. A limit of 30 would have hit people, and above 50 there were only bots. For static files, people briefly reach 200 requests too.

Watching it live

Through the stats socket, HAProxy shows the table with every IP and its count:

echo "show table ratelimit_table_page_rate" | nc 127.0.0.1 9999

# table: ratelimit_table_page_rate, type: ipv6, size:1048576, used:1
0x7ec2431594a0: key=::ffff:172.19.0.1 use=0 exp=57506 shard=0 http_req_rate(10000)=74

You can test it with parallel requests:

seq 80 | xargs -P 20 -I{} curl -sk -o /dev/null -w '%{http_code}\n' https://example.test/ | sort | uniq -c

     50 200
     30 429

Blocked requests show up in the log with <NOSRV>, status 429 and the flag PR--. HAProxy answered itself.

What a 10-second limit cannot do

A rate limit caps the pace, not the amount. Anyone who keeps producing 50 requests with F5 swings back and forth at the limit: one page, one 429, and after 10 seconds the count starts over.

More annoying are bots that stay evenly just below the limit. A bot with 43 requests per 10 seconds slips through and still makes more than 15,000 PHP requests an hour.

What helps against that is a second, long window in another table, for example http_req_rate(10m). A person who reads a page every 10 to 20 seconds comes, with AJAX calls, to a few hundred requests in 10 minutes. Non-stop traffic comes to many times that. Important: expire then has to be at least as long as the window.

If you want to do more than slow outliers down and lock them out completely for a few minutes, you take a third table with a gpc0 counter. That is a general-purpose counter that is raised when the limit is crossed and blocks the IP until it expires. But that is a topic of its own.