Thoughts about http-spider.nse

[Ron <ron () skullsecurity net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey all,

I've been putting some work this week into a spider script. Right now it works with very basic functionality (basically 
finds all the href='..' and src='...' arguments, parses them, and stores them in the registry), but I'm hoping to get 
some comments. 

First, at the moment, the script has no output of its own. I think that's a good thing, because of the amount of 
information that people may or may not want to see, having other scripts that display the results might be a better 
plan. This leads us to required dependencies, though -- right now, dependencies only modify the order that requests 
happen, but for this to work we need dependencies that actually turn on other scripts. 

Second, I'm making heavy use of the registry to store information. I'm trying to give later scripts as much information 
as possible. So, I'm storing the same data in many different ways to make it easy to find exactly what you want. Here 
are some types of data, and some short examples (mostly from nmap.org). I've attached a full registry dump of scanning 
nmap.org at a depth of 2. 

* All pages, and all pages with their full querystring
|       all_pages:
|         1: "/"
|         2: "/shared/css/insecdb.css"
|         3: "/book/man.html"
|         4: "/book/install.html"
|         5: "/download.html"

* All directories, and all files
|       directories:
|         1: "/"
|         2: "/book/"
|         3: "/presentations/BHDC10/"
|         4: "/nsedoc/"
|       files:
|         1: "/shared/css/insecdb.css"
|         2: "/book/man.html"
|         3: "/book/install.html"
|         4: "/download.html"

* All files indexed by extension
|       extensions:
|         html:
|           1: "/book/man.html"
|           2: "/book/install.html"
|           3: "/download.html"
|           4: "/changelog.html"
|           5: "/docs.html"
|           6: "/book/nse.html"
|           7: "/movies.html"
|           8: "/book/man-bugs.html"
|         css:
|           1: "/shared/css/insecdb.css"

* All pages that have arguments, as well as their arguments (can have multiple copies for pages we see linked with 
different arguments)
|       cgi_args:
|         /index.cfm:
|           1:
|             pageID: "12"
|           2:
|             pageID: "13"
|           3:
|             pageID: "249"
|           4:
|             pageID: "1"
|       cgi_querystring:
|         /index.cfm:
|           1: "pageID=12"
|           2: "pageID=13"
|           3: "pageID=249"
|           4: "pageID=1"
|           5: "pageID=2"
|       cgi_full_query:
|         1: "/index.cfm?pageID=12"
|         2: "/index.cfm?pageID=13"
|         3: "/index.cfm?pageID=249"
|         4: "/index.cfm?pageID=1"
|       cgi:
|         1: "/index.cfm"
|         2: "/display.cfm"

* All pages we've seen a specific page link to (or linked from)
|       links_to:
|         /docs.html:
|           1: "/shared/css/insecdb.css"
|           2: "http://g.adspeed.net/ad.php?do=clk&zid=14678&wd=728&ht=90&pair=as";
|           3: "http://nmap.org/";
|           4: "http://nmap.org/book/man.html";
|           5: "http://nmap.org/book/install.html";
|           6: "http://nmap.org/download.html";
|           7: "http://nmap.org/changelog.html";
|         /book/nse.html:
|           1: "/shared/css/insecdb.css"
|           2: "/book/toc.html"
|           3: "/book/osdetect-unidentified.html"
|           4: "/book/nse-usage.html"
|           5: "/book/preface.html"
|           6: "/book/intro.html"

* All pages indexed by content-type 
|       content-type:
|         text/html; charset=UTF-8:
|           1: "/"
|           2: "/book/man.html"
|           3: "/book/install.html"
|           4: "/download.html"
|           5: "/changelog.html"
|           6: "/book/"
|         text/css:
|           1: "/shared/css/insecdb.css"
|         text/html; charset=iso-8859-1:
|           1: "/favicon"
|           2: "/data/COPYING"
|           3: "/fb"

Opinions would be great! 

Ron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)

iEYEARECAAYFAky9DuIACgkQ2t2zxlt4g/RPRACfSfK2Kgh4zRLsjmTNu+LGDxn9
+F0AoLBkFZ2EzOnW+BXuSndp8zP0N1A3
=t0DG
-----END PGP SIGNATURE-----

Attachment: nmap-spider.txt
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/