Home page logo

nmap-dev logo Nmap Development mailing list archives

Re: Web crawling library proposal
From: Paulino Calderon <paulino () calderonpale com>
Date: Mon, 07 Nov 2011 11:16:47 -0800

Hi list,

Thank you all for the feedback. You've pointed out some very important issues. I've started working on the following changes:
* OO design.
* crawl() must return a iterator.
* Allow multiple crawlers to run at the same time.
* Remove anchors from links by default instead of having to set the variable httpspider.ignoreParams.
* Queue will be removed.
* Implement a better uri filter.


Paulino Calderón Pale
Web: http://calderonpale.com
Twitter: http://www.twitter.com/paulinocaIderon

Sent through the nmap-dev mailing list
Archived at http://seclists.org/nmap-dev/

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]