Nmap Development: Re: New script - http-favicon.nse

[Vlatko Kosturjak <kost () linux hr>]

Brandon Enright wrote:
> On Fri, 07 Nov 2008 00:05:57 +0100
> Vlatko Kosturjak <kost () linux hr> wrote:
> > Thanks on suggestion! Started doing the random scan. Actually, two of
> > them on two different machines/IPs. One scans for fingerprints on port
> > 80 and the other on port 443. Hopefully, database will be populated
> > very soon.
> A good way to get info from a boatload of webservers is to pull down the
> Wikipedia database (available in XML for you xquery/xpath/grep fiends)
> and harvest URLs/domain/webservers from there.
> I think using something actually linked in Wikipedia is much better
> than "-iR awholeheckofalot -p 80,443" of the Internet.
Actually, Wikipedia is not good source. They started to remove http://
references from the usual articles and only top 5 (or some other number)
links they put on External references in articles:
http://en.wikipedia.org/wiki/Wikipedia:EL

I did small research and I found out that the best source for this would
be Open Directory Project (DMOZ):
http://rdf.dmoz.org/

Direct link is (don't just click on it, it's big):
http://rdf.dmoz.org/rdf/content.rdf.u8.gz

They even have nice format to do so:
http://rdf.dmoz.org/rdf/content.example.txt

Will leave nmap -iR do its job while I'm writting parser for this one.
Let's see what results will got!

Kost

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org