Correctly dealing with bots and scrapers.

[Andrew Latham via NANOG <nanog () lists nanog org>]

I just had an issue with a web-server where I had to block a /18 of a
large scraper. I have some topics I could use some input on.

1. What tools or setups have people found most successful for dealing
with bots/scrapers that do not respect robots.txt for example?

2. What tools for response rate limiting deal with bots/scrapers that
cycle over a large variety of IPs with the exact same user agent?

3.  Has anyone written or found a tool to concentrate IP addresses
into networks for IPTABLES or NFT? (60% of IPs for network X in list
so add network X and remove individual IP entries.)

-- 
- Andrew "lathama" Latham -
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/Z2J6CFBKU3ARYSQYKBRDCV6ZFUOS3WMU/