Re: Correctly dealing with bots and scrapers.

[William Herrin via NANOG <nanog () lists nanog org>]

On Wed, Jul 16, 2025 at 11:57 AM Chris Adams via NANOG
<nanog () lists nanog org> wrote:
> The problem with a lot of the "AI" scrapers is that they're apparently
> using botnets and will often only make a single request from a given IP
> address, so reactive blocking doesn't work

So do reactive un-blocking instead. The first request from any IP
address gets an empty web page with, "<meta http-equiv=refresh
content=1>," and triggers a permit rule which allows subsequent
requests to reach the actual web server and content. That way the
one-shot sources get an empty page no matter what URL they request.

Regards,
Bill Herrin


-- 
William Herrin
bill () herrin us
https://bill.herrin.us/
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/IDU4FU7HZVRHUZWJ6SLQ7UH64JQ4IYD3/