Re: Correctly dealing with bots and scrapers.

[Andrew Latham via NANOG <nanog () lists nanog org>]

Yes Marco that is one of the tactics I see to deal with this. Fail2ban
is a great piece of software that I use. This reminded me of another
related topic. Security Scans. Any requests for wordpress could be an
easy way to flag and block with fail2ban when wordpress is not in use.

On Wed, Jul 16, 2025 at 12:36 PM Marco Moock via NANOG
<nanog () lists nanog org> wrote:
> Am 16.07.2025 um 10:48:39 Uhr schrieb Andrew Latham via NANOG:
>
> > 1. What tools or setups have people found most successful for dealing
> > with bots/scrapers that do not respect robots.txt for example?
>
> Place a link to a file that is hidden to normal people. Exclude the
> directory via robots.txt.
>
> Then use fail2ban to block all IP addresses that poll the file.
>
> --
> Gruß
> Marco
>
> Send unsolicited bulk mail to 1752655719muell () cartoonies org
> _______________________________________________
> NANOG mailing list
> https://lists.nanog.org/archives/list/nanog () lists nanog org/message/WX65VXNA5F2CMOZLFCXGNPVLMNIY5QV5/



-- 
- Andrew "lathama" Latham -
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/W7F25UAXWPJYKLVFTGU7HFYSAMRP4Y54/