Home page logo

nmap-dev logo Nmap Development mailing list archives

Re: [RFC] Improve NSE HTTP architecture.
From: Djalal Harouni <tixxdz () opendz org>
Date: Sat, 18 Jun 2011 09:47:31 +0100

On Thu, Jun 16, 2011 at 06:02:02PM -0500, Ron wrote:
On Thu, 16 Jun 2011 08:06:30 +0200 Patrik Karlsson <patrik () cqure net> wrote:
1. In my experience it's kind of difficult to write a good
spider/crawler. Today it's a lot more complex than using regexp to
discover all <a href tags or stuff that looks like an url due to
javascript, flash, silverlight, etc ... That said, I think a decent
spider/crawler could still be written for NSE. What I also think
could be a good idea is to allow the user to "import" a file
containing the URLs to process. This way you could manually cover
most parts of a site using a local proxy, extract the urls and feed
them to NSE.
Something else to keep in mind - http-fingerprints.nse can seed http-spider.nse. So in addition to the URLs to 
process, we can use a whole pile of known URLs.

Djalal - Great work on this writeup! I replied to a thread about this from Patrik yesterday, but it seems like you 
were wayyy ahead of me. In nmap-exp/ron, I have a mostly working (but not really tested) http-spider.nse script. If 
you're planning on working on a spider, that might be a decent starting point (or not.. :) )
I'll check that thread.

I didn't go deep in the study of the spider, since it will be Paulino's
job, and he already has a proposal draft in his nmap-exp/ branch.

And yes we must check your http-spider script, it really can be a good
starting point. Our proposal is based on your http-enum and
http-fingerprints work, and also Patrik's work.

Just to let you know that currently I'm not working on implementing
the proposal ideas, feel free to experiment them.

Thanks Ron.

Sent through the nmap-dev mailing list
Archived at http://seclists.org/nmap-dev/

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]