robots.txt NSE script

[doug () hcsw org]

Hi Eddie!

Just thought I'd send over this article about a
large-scale robots.txt survey in case you hadn't
already seen it:

http://www.nextthing.org/archives/2007/03/12/robotstxt-adventure

Lots of interesting tidbits, as well as some ideas
for fingerprinting web frameworks and add-on packages
by commonly excluded path-names.

Also there's apparently a very common file called
info.txt that Alexa uses for site info:

http://www.varisearch.com/Webmasters/PromotingYourSite/TipSheets/Alexa.htm

Best,

Doug

Attachment: signature.asc
Description: Digital signature

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org