IP: a bit more Every 30 minute spiders

[Dave Farber <farber () cis upenn edu>]

Date: Sat, 02 Nov 1996 17:31:27 -0500
To: farber () central cis upenn edu (David Farber)
From: Jock Gill <jgill () penfield-gill com>
Subject: Every 30 minute spiders


Dave,


Brad Cox wrote:


> Apart from the "few weeks old" bit, this sounds routine and innocuous.
> Search engines (Yahoo, AltaVista) routinely rebuild their indices with
> spiders just as you describe.


These new spiders are visiting some locations twice per hour and snarfing
every page each time they visit.  This is not required for index re-building.


Perhaps it is time to think about micro-payments for transactions between
machines, cyber-serfs, and cyber-serfs and machines.  Perhaps to only think
of micro-payments when humans are involved is too human-centric.


Perhaps also a system such as Agoric Inc.'s Digital Silk Road would be an
appropriate response - or part of one?  See www.agorics.com/%7Eagorics/


Send your spiders as often as you see fit, but pay as you go.  Turn spider
visits into profitable exercises.  Financial incentives and penalties have
worked well in the history of the West.  They form a very interesting
constraint system.


Regards,


Jock
 ____________________________________________________________________________
                                Jock Gill
                            Penfield Gill, Inc.
                               Boston,  MA
                          jgill () penfield-gill com           
                    <http://www.penfield-gill.com/gill>
 ____________________________________________________________________________