# We usually don't block any particular robot software in robots.txt, # as we judge them by behavior rather than name. Abusive IPs which # make hundreds of requests in a short period of time will be banned. # Block http://www.dotnetdotcom.org/ User-agent: dotbot Disallow: / # Block commoncrawl.org crawler which makes a huge number of requests User-agent: CCBot Disallow: / User-agent: * # Disallow crawling search results (search page itself is ok) Disallow: /search.html?* Disallow: /search/?* Sitemap: https://seclists.org/seclists_index.xml