Home page logo

nmap-dev logo Nmap Development mailing list archives

Re: [RFC] Improve NSE HTTP architecture.
From: Ron <ron () skullsecurity net>
Date: Thu, 16 Jun 2011 18:02:02 -0500

On Thu, 16 Jun 2011 08:06:30 +0200 Patrik Karlsson <patrik () cqure net> wrote:
1. In my experience it's kind of difficult to write a good
spider/crawler. Today it's a lot more complex than using regexp to
discover all <a href tags or stuff that looks like an url due to
javascript, flash, silverlight, etc ... That said, I think a decent
spider/crawler could still be written for NSE. What I also think
could be a good idea is to allow the user to "import" a file
containing the URLs to process. This way you could manually cover
most parts of a site using a local proxy, extract the urls and feed
them to NSE.
Something else to keep in mind - http-fingerprints.nse can seed http-spider.nse. So in addition to the URLs to process, 
we can use a whole pile of known URLs.

Djalal - Great work on this writeup! I replied to a thread about this from Patrik yesterday, but it seems like you were 
wayyy ahead of me. In nmap-exp/ron, I have a mostly working (but not really tested) http-spider.nse script. If you're 
planning on working on a spider, that might be a decent starting point (or not.. :) )

Sent through the nmap-dev mailing list
Archived at http://seclists.org/nmap-dev/

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]