Re: Correctly dealing with bots and scrapers.

[Robert L Mathews via NANOG <nanog () lists nanog org>]

On Jul 16, 2025, at 9:48 AM, Andrew Latham via NANOG <nanog () lists nanog org> wrote:
> 2. What tools for response rate limiting deal with bots/scrapers that
> cycle over a large variety of IPs with the exact same user agent?


If the bots are impersonating real browser User-Agents, and you use something like ModSecurity that can examine HTTP 
headers, you can look at a few requests and probably find that they send or omit things compared to real browsers.

Today, for example, I blocked some of the requests from a botnet that often sends this pair of headers:

 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 
Safari/537.36
 Sec-Ch-Ua-Platform: "macOS"

Note the mismatch of "Windows NT" vs. "macOS": it appears the bot randomizes "Sec-Ch-Ua-Platform" but not the 
"User-Agent", so a good percentage of their requests show this mismatch.

Another recent high volume botnet impersonating Chrome/134 is sending this header:

 Referrer: https://www.google.com/

[sic]: They forgot to misspell "Referer".

Most botnets I look at have multiple "tells" like this in the HTTP headers. You have to be mindful to avoid false 
positives from proxies that mess with headers, but it's otherwise an effective way to block them and stop them from 
consuming CPU time.

Whether this is worth your time is a different matter. It's worth mine because we host thousands of sites, but I 
probably wouldn't waste the effort on it if it was just my own site, unless the botnet was making the site not work.

-- 
Robert L Mathews

_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/DXWVYDR47HOLJBQQNZQKAK242ASV2SWY/