Re: Correctly dealing with bots and scrapers.

[Eric Kuhnke via NANOG <nanog () lists nanog org>]

There is what we could call a "growth market" of grey market organizations
and individuals selling residential proxies that route traffic through
actual residential cablemodem, DSL, FTTH connections at peoples' houses.
Usually this is implemented one of two ways, a router/home gateway device
that's been pwned and is part of a botnet, or by tricking people into
installing some form of proxy software on one of their persistently
connected home computers.

Used for both bot scraping and also more mundane credit card fraud and
various scams, providing people access to georestricted online casinos,
weird hawala-adjacent online money/cryptocurrency exchange and transfer
platforms. Just a vast array of shady stuff.

Google "residential proxies for sale" to see the tip of the iceberg.

On Wed, Jul 16, 2025 at 4:38 PM Andrew Latham via NANOG <
nanog () lists nanog org> wrote:

> Chris
>
> Spot on, and I am getting the feeling this is where the value to a
> geo-ip service comes to play that offers defined "eyeball networks" to
> allow.
>
> On Wed, Jul 16, 2025 at 12:57 PM Chris Adams via NANOG
> <nanog () lists nanog org> wrote:
> > Once upon a time, Marco Moock <mm () dorfdsl de> said:
> > > Place a link to a file that is hidden to normal people. Exclude the
> > > directory via robots.txt.
> > >
> > > Then use fail2ban to block all IP addresses that poll the file.
> >
> > The problem with a lot of the "AI" scrapers is that they're apparently
> > using botnets and will often only make a single request from a given IP
> > address, so reactive blocking doesn't work (and can cause other issues,
> > like trying to block 100,000 IPs, which fail2ban for example doesn't
> > really handle well).
> > --
> > Chris Adams <cma () cmadams net>
> > _______________________________________________
> > NANOG mailing list
> https://lists.nanog.org/archives/list/nanog () lists nanog org/message/AFJF4UQJZW6ALTY6SA7OHBN2AZC72SZQ/
>
>
>
> --
> - Andrew "lathama" Latham -
> _______________________________________________
> NANOG mailing list
>
> https://lists.nanog.org/archives/list/nanog () lists nanog org/message/DHUYTBIXFMWE2KWC5NKCR7AJIWPYUL4E/
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/YF2DNATKWKI6JF47F5OZZCYGBXOXMODX/