Home page logo
/

fulldisclosure logo Full Disclosure mailing list archives

Re: Google's robots.txt handling
From: "Gildseth, Tommy" <tommy.gildseth () aller no>
Date: Mon, 10 Dec 2012 21:01:37 +0000

On 2012-12-10 12:25, Hurgel Bumpf wrote:
Hi list,


i tried to contact google, but as they didn't answer my email,  i do
forward this to FD.
This "security" feature is not cleary a google vulnerability, but
exposes websites informations that are not really intended to be
public.

(Additionally i have to say that i advocate robots.txt files without
sensitive content and working security mechanisms.)

Here is an example:

An admin has a public webservice running with folders containing
sensitive informations. Enter these folders in his robots.txt and
"protect" them from the indexing process of spiders. As he doesn't
want the /admin/ gui to appear in the search results he also puts his
/admin in the robots text and finaly makes a backup to the folder
/backup.

Nevertheless these folders arent browsable but they might contain
f(a)iles with easy to guess namestructures, non-encrypted
authentications (simple AUTH) , you name it...

Without a robots.txt nobody would know about the existance of these
folders, but as some folders might be linked somewhere, these folders
might appear in search results when not defined in the robots.txt
The
admin finds himself in a catch-22 situation where he seems to prefer
the robots.txt file.

Long story short.

Although google accepts and respects the directives of the robots.txt
file, google INDEXES these files.

This my concern.


http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fadmin

http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fbackup

http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fpassword

As these searches can be used less for targeted attacks, they more
can be used to find victims.


http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+%2FDisallow%3A+wp-admin

http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+%2FDisallow%3A+typo3
<Just be creative>

This shouldn't be a discussion about bad practice but the google
feature itself.

Indexing a file which is used to prevent indexing.. isn't that just
paradox and hypocrite?

If we ignore the obvious security by obscurity aspect of this discussion, there are alternatives to putting "secret" 
paths in the robots.txt file. One is f.ex to tell your webserver to serve these "secret" paths with a X-robots-tag HTTP 
header, thus not exposing anything before it's actually been requested.

-- 
TommyG

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/


  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]