29 Jun
2004
29 Jun
'04
3:20 a.m.
Several things are going on right now. At the same time that we were
moving bitsavers to its own machine, script kiddies totally FU spies.com
We've been dealing with that (little things like not having mail working
for most of the weekend)
As soon as bitsavers came on line again, google crawlers started downloading
EVERYTHING from multiple IP adrs. There is now bandwidth limiting by host so
I'm trying to keep them from consuming all of the bandwidth.
FTP for mirroring isn't up yet, I've told Jay that I'll email when it works.
Haven't heard any more from Patrick about the mirror he was starting.
29 Jun
29 Jun
3:42 a.m.
On Tuesday 29 June 2004 10:20, Al Kossow wrote:
FTP for mirroring isn't up yet, I've told Jay
that I'll email when it
works. Haven't heard any more from Patrick about the mirror he was
starting.
"I'm working on it". Actually, it's the 2nd item on my to-do list.
First, I've gotta move computer-refuge to a new machine, though.
Pat
--
Purdue University ITAP/RCS --- http://www.itap.purdue.edu/rcs/
The Computer Refuge --- http://computer-refuge.org
4:16 a.m.
On Tue, 2004-06-29 at 15:42, Patrick Finnegan wrote:
On Tuesday 29 June 2004 10:20, Al Kossow wrote:
Second from top, presumably ;-)
cheers,
Jules
FTP for mirroring isn't up yet, I've told
Jay that I'll email when it
works. Haven't heard any more from Patrick about the mirror he was
starting.
"I'm working on it". Actually, it's the 2nd item on my to-do list.
5:16 a.m.
Al Kossow wrote:
As soon as bitsavers came on line again, google crawlers started downloading
EVERYTHING from multiple IP adrs.
Put this in your robots.txt:
User-agent: Googlebot
Disallow: /*.pdf$
FTP for mirroring isn't up yet, I've told Jay
that I'll email when it works.
Haven't heard any more from Patrick about the mirror he was starting.
Mirrors are clearly like buses. I'm one of the "dozens of people"[1] who
has a private mirror, for maintaining Manx and OCRing, but that is now
going online at VT100.net as I've moved to a dedicated server. Bandwidth
is, as Rolls-Royce would say, "adequate". However, I'm populating the
mirror from home at only 100 MiB an hour, so it could take weeks to get
up to speed!
[1] Al, 2004-05-07.
--
Paul
http://vt100.net/manx/ - a catalogue of online computer manuals
5:58 a.m.
On Tuesday 29 June 2004 12:16, Paul Williams wrote:
Al Kossow wrote:
Grr. Don't do this. I really hate it when people disallow google to
index content. It always makes it harder to find stuff. The only time
I'd consider doing it is if the "webserver" is on a dialup connection
or something that won't stay at the same IP address.
Pat
--
Purdue University ITAP/RCS --- http://www.itap.purdue.edu/rcs/
The Computer Refuge --- http://computer-refuge.org
As soon as bitsavers came on line again, google
crawlers started
downloading EVERYTHING from multiple IP adrs.
Put this in your robots.txt:
User-agent: Googlebot
Disallow: /*.pdf$ 
6:22 a.m.
Patrick Finnegan wrote:
Grr. Don't do this. I really hate it when people
disallow google to
index content. It always makes it harder to find stuff. The only time
I'd consider doing it is if the "webserver" is on a dialup connection
or something that won't stay at the same IP address.
It would surprise me if Google gets anything out of the .pdf files at
bitsavers. They are all (mostly?)
image scans....
Then again if Google DOES OCR and index the scans all our problems are
solved ;-)
-- HansP
6:30 a.m.
>>>> "Patrick" == Patrick
Finnegan <pat(a)computer-refuge.org> writes:
Patrick> On Tuesday 29 June 2004 12:16, Paul Williams wrote:
> Al Kossow wrote: > As soon as bitsavers came on
line again, google
> crawlers started > downloading EVERYTHING from multiple IP adrs.
>
> Put this in your robots.txt:
>
> User-agent: Googlebot Disallow: /*.pdf$
Patrick> Grr. Don't do this. I really hate it when people disallow
Patrick> google to index content. It always makes it harder to find
Patrick> stuff. The only time I'd consider doing it is if the
Patrick> "webserver" is on a dialup connection or something that
Patrick> won't stay at the same IP address.
There's a second reason, which applies here -- there IS no content
that Google can index, because those files are all bitmap -- no text.
If there were text files or OCRed PDF files, that would be different,
but as it is, Google will find absolutely nothing. So Al might as
well tell it not to try.
paul
6:39 a.m.
On Tuesday 29 June 2004 13:30, Paul Koning wrote:
There's a second reason, which applies here --
there IS no content
that Google can index, because those files are all bitmap -- no text.
If there were text files or OCRed PDF files, that would be different,
but as it is, Google will find absolutely nothing. So Al might as
well tell it not to try.
Oh right. Heh, I've managed to forget about that part.
Pat
--
Purdue University ITAP/RCS --- http://www.itap.purdue.edu/rcs/
The Computer Refuge --- http://computer-refuge.org
7433
days inactive
7433
days old
7 comments
6 participants
participants (6)
-
aek＠spies.com -
hansp＠citem.org -
julesrichardsonuk＠yahoo.co.uk -
pat＠computer-refuge.org -
paul＠frixxon.co.uk -
pkoning＠equallogic.com