more on AFP Sues Google News

[Nexus <dave () farber net>]

------- Original message -------
From: Carl Malamud  <carl () media org>
Sent: 20/3/'05,  12:02

> ------ Forwarded Message
> From: Dana Blankenhorn <dana () a-clue com>
> Date: Sun, 20 Mar 2005 14:08:04 -0500
> To: <dave () farber net>
> Subject: Re: [IP] AFP Sues Google News
<snip>

> Now  to Google's defense.
>
> Exhibit A for the defense. This is an Agence France-Presse story published
> on its customer site, Velo News. It has been spidered by Google News,
> obviously without the express written permission of Agence France-Presse.
>
> But is it possible for Google News not to spider this story? Yes, it is.
> That would require only AFP to include a robots.txt file on stories it sends
> affiliates, instructing those pages not to allow spiders or robots to see
> them.

Their site shows a robots.txt file in place for at least a month:

wget --save-headers http://www.afp.com/robots.txt
HTTP/1.1 200 OK
Date: Sun, 20 Mar 2005 19:58:28 GMT
Server: Apache/1.3.27 (Unix)
Cache-Control: max-age=300
Expires: Sun, 20 Mar 2005 20:03:28 GMT
Last-Modified: Wed, 23 Feb 2005 10:54:38 GMT
ETag: "761b2-4f-421c60ee"
Accept-Ranges: bytes
Content-Length: 79
Connection: close
Content-Type: text/plain

User-Agent: *
Disallow: /beta
Disallow: /francais/news
Disallow: /english/news

And, Archive.org shows that they've had that in place for a long
time before:

http://web.archive.org/web/*/http://www.afp.com/robots.txt

Regards,

Carl

-------------------------------------
You are subscribed as lists-ip () insecure org
To manage your subscription, go to
  http://v2.listbox.com/member/?listname=ip

Archives at: http://www.interesting-people.org/archives/interesting-people/