29 Jun
2004
29 Jun
'04
5:58 a.m.
On Tuesday 29 June 2004 12:16, Paul Williams wrote:
Al Kossow wrote:
Grr. Don't do this. I really hate it when people disallow google to
index content. It always makes it harder to find stuff. The only time
I'd consider doing it is if the "webserver" is on a dialup connection
or something that won't stay at the same IP address.
Pat
--
Purdue University ITAP/RCS --- http://www.itap.purdue.edu/rcs/
The Computer Refuge --- http://computer-refuge.org
As soon as bitsavers came on line again, google
crawlers started
downloading EVERYTHING from multiple IP adrs.
Put this in your robots.txt:
User-agent: Googlebot
Disallow: /*.pdf$