24 May
2002
24 May
'02
11:33 p.m.
Sellam Ismail wrote:
On Fri, 24 May 2002, Megan wrote:
This "per the request of the site owner" is nothing more than an
instruction placed in the website's robots.txt file, which can request
that either specific or all crawlers do not retrieve pages from the
site. When archive.org's crawler sees this instruction, it definitely
does not crawl the site, so they don't have a large stash of pages that
are not available for viewing.
- Paul
"Per the request of the site owner, http://www.dec.com
is no longer available in the Wayback Machine.
But I wonder if they do in fact have the pages archived? My guess
is that they do, but they just can't make them available.