Re: Correctly dealing with bots and scrapers.

[Marco Moock via NANOG <nanog () lists nanog org>]

Am 16.07.2025 um 10:48:39 Uhr schrieb Andrew Latham via NANOG:

> 1. What tools or setups have people found most successful for dealing
> with bots/scrapers that do not respect robots.txt for example?

Place a link to a file that is hidden to normal people. Exclude the
directory via robots.txt.

Then use fail2ban to block all IP addresses that poll the file.

-- 
Gruß
Marco

Send unsolicited bulk mail to 1752655719muell () cartoonies org

Attachment: _bin
Description: OpenPGP digital signature

_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/WX65VXNA5F2CMOZLFCXGNPVLMNIY5QV5/