Home page logo
/

pauldotcom logo PaulDotCom mailing list archives

Re: Looking for a good web spider
From: Dennis Lavrinenko <dennis.lavrinenko () gmail com>
Date: Sat, 25 Sep 2010 19:27:22 -0500

Although I've used it mostly for mirroring sites, I believe httrack has
--spider option.

On Sat, Sep 25, 2010 at 7:46 AM, Robin Wood <robin () digininja org> wrote:

On 25 September 2010 02:46, Adrian Crenshaw <irongeek () irongeek com> wrote:
Hi all,
    I'm looking at some of the tools in BT4R1, and will be looking at
what
Samurai WTF has to offer once I finish downloading the latest version.
I'm
looking for some sort of spider that lets me do the following:

1. Follow every link on a page, even onto other domains, as long as the
top
level domain name is the same (edu, com, cn, whatever)
2. For every page it visits, it collect the file names of all resources.
3. The headers so I can see the server version.
4. Grab the robots .txt if possible.

Any ideas on the best tool for the job, or do I need to roll my own?

If you want to roll your own you can take my CeWL code and check the
spider, I do a full spider and check whether you are on the same site
or off and grab all the documents, you should easily be able to modify
this to do what you want.

Robin
_______________________________________________
Pauldotcom mailing list
Pauldotcom () mail pauldotcom com
http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom
Main Web Site: http://pauldotcom.com

_______________________________________________
Pauldotcom mailing list
Pauldotcom () mail pauldotcom com
http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom
Main Web Site: http://pauldotcom.com

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]
AlienVault