Re: Correctly dealing with bots and scrapers.

[Jay Acuna via NANOG <nanog () lists nanog org>]

On Wed, Jul 16, 2025 at 1:57 PM Chris Adams via NANOG
<nanog () lists nanog org> wrote:

> The problem with a lot of the "AI" scrapers is that they're apparently
> using botnets and will often only make a single request from a given IP
> address, so reactive blocking doesn't work (and can cause other issues,

Append a canary ID to all URLs displayed on the page.
For example:
https://example.com/example.html?visitor=1234ABCDEF&signature=XYZ

Upon receiving a page request that is missing a valid "Visitor="  tag,
or in case the visitor's IP address  does not match the correct IP
address linked to
that visitor tag: create a new visitor tag in the database and Return
an empty page
with a 302 redirect  redirecting the visitor back to the homepage with
the new tag added
and refusing to display the individual page requested, until they
click a link provided by
the website.

Do the same if the signature=  attribute is missing or fails to verify.

The signature attribute is a HMAC which authenticates that the combination
of the URL path and visitor ID are from a page displayed by the web server
and have not been altered by the client.   For example, they cannot simply
learn their client ID and append it on their own to
https://example.com/example.html
they need the unique  signature=   added to the link by the web server
in order to have access to the example.html page.


--
-JA
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/FN2CUVJAOPGYQ7OTXBGHF7P2HC6QCTTY/