Home page logo

nmap-dev logo Nmap Development mailing list archives

[NSE] Robots rethink
From: "Eddie Bell" <ejlbell () gmail com>
Date: Wed, 4 Jun 2008 12:14:50 +0100

Hi all,

I've had this new robot script lying around for a while. The current
version prints out all disallowed entries in a site's robots.txt file
which takes up a lot of screen estate[1]. The attached version just
prints a message if a site has a robots.txt file with disallowed

What do you think?

 - eddie

[1] 80/tcp open  http
     |  robots.txt: /news?output=xhtml& /search /groups /images
     |  /catalogs /catalogues /news /nwshp /? /addurl/image?
     |  /pagead/ /relpage/ /relcontent /sorry/ /imgres
     < ... this goes on for another 26 lines ... >

[2] 80/tcp open  http
     |_ robots.txt: has disallowed entries

Attachment: robots.nse

Sent through the nmap-dev mailing list
Archived at http://SecLists.Org

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]