Home page logo

bugtraq logo Bugtraq mailing list archives

Re: WebVulnCrawl searching excluded directories for hackable web servers
From: "Dennis Brown" <dennis.brown () gmail com>
Date: Fri, 31 Mar 2006 16:04:20 -0500

Hi Michael, thanks for noticing my project.  You've pretty much made
my point about why I'm doing this, as robots.txt shouldn't be used as
an ACL.  That's exactly the reason why I'm doing this crawling.  I'm
trying to find out how widespread of an issue this is, and to see how
this is being misused, as I've stated in my blog.

In both your posts, you've claimed you have evidence of port scanning
and, in this recent post, hitting .org and .gov sites, both of which
are false.  As you've made no attempt to contact me directly about
what I'm doing or to verify that I am indeed doing either of these
things, I'd like to ask you to share this information so we can
determine what is going on.  The data you've linked to shows mostly
port 80 access, as well as some port 9999 access which I suspect is
from a redirect, probably the main reason you may believe I'm doing
things like intentionally hitting .org sites, but there's nothing
there to verify your claims.  If you have other data to back up these
allegations, please share it.

Dennis Brown

On 3/29/06, Michael Scheidell <scheidell () secnap net> wrote:
Just a quick followup and clarification:

-----Original Message-----
From: Michael Scheidell
Sent: Wednesday, March 15, 2006 8:38 AM
To: bugtraq () securityfocus com
Subject: WebVulnCrawl searching excluded directories for
hackable web servers

What he is doing is a violation of the RFC's (governing
robots.txt.. Yes, hackers do that also)

There was an RFC proposed and looked at in 1996, but never adopted.

The robots.txt file is NOT AN ACCESS CONTROL LIST, and SHOULD
PROTECTED AGAINST Directory listing.

Someone mentioned that sometimes you want directory listings.
That should have suggested turning off directory listing for any
directories you don't want listed.
(I don't know why you would put them in robots.txt)

WebVuln Blog stated he was only hitting .com sites.
I have evidence he has moved to .org sites, and in fact, has hit a US
government site as well.
I would hope this US government IT security folks would know not to use
robots.txt as an ACL, the web folks aren't always security folks (web
aplications themselves are sometimes prone to SQL injextion, XSS
attacks, PHP coding errors) and since there is a large gap between
applications and web development, the chances of accidentially gathering
information that should not be gathered is huge.

Every security person should review the robots.txt file on their web
site for implications.

Further, dshield shows them portscanning the net also,
looking for unpublished information on unpublished servers.

So does mynetwatchman:


Michael Scheidell, CTO
561-999-5000, ext 1131
SECNAP Network Security Corporation

  By Date           By Thread  

Current thread:
  • Re: WebVulnCrawl searching excluded directories for hackable web servers Dennis Brown (Apr 03)
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]