20 Sep
2011
20 Sep
'11
11:03 a.m.
On 20 Sep 2011 at 10:38, jim s wrote:
You can poke websites at archive.org to crawl. It
won't help if the
site has robots set to deflect archiving, but sometime it misses all
of a site because the front porch is marked closed. I have entered
all of my sites for crawling. I don't know how good it does archiving
content though. It seems to get images, but not sure about binaries
of various types. I've seen a very few binaries, so they do save
some.
Nope, Micronics has ROBOTS.TXT.
I've never found archive.org to be of much use when looking for
defunct ftp material.
Am I missing something?
--Chuck