Re: Proposed improvements on httpspider.lua

[David Fifield <david () bamsoftware com>]

On Sun, Jun 23, 2013 at 09:47:02PM +0300, George Chatzisofroniou wrote:
> To summarize the two blacklists:
>
> blacklist for spidering - The crawler will not return the resources
> defined in this blacklist at all. Currently supported by "withinhost",
> "withindomain", "noblacklist" options.
> blacklist for scraping - The crawler will return the resources defined
> in this blacklist but it won't scrape them (meaning it won't search
> for links within them). Currently supported by "blacklistforscraping"
> option.

Even better, can't you use a design where you provide a callback
function that is called for each resource? The function returns true if
spidering should continue or false if not. A script that just wants to
list all the external resources can make note of them in the callback,
and return false for them. The default callback would just exclude other
domains, the way it works now.

The redirect_ok option in http.lua works like this.

David Fifield
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/