19 Mar
2007
19 Mar
'07
12:29 p.m.
From: Jules Richardson <julesrichardsonuk at
yahoo.co.uk>
Dave Dunfield wrote:
OK, a more considered reply now I've got a few minutes...
Hi
I thought I'd add a little here. One thing that is missing. We'd need
some way to keep it running beyond when we are gone. There should
be a trust setup. It should have at least two or three people maintaining
is and willing to deal with the tax issues.
I tend to agree that it shouldn't force format. Dave's open format is
vary well thought out and his tool is good but there is legacy data
that would need to be converted. As an example, the disk image
data I have for the Polymorphic 8813 is in the format used by the
Polymorphic software to move disk images through modems to other
Polymorphic machines. It does currently lack a method to restore it
to a machine that has no boot floppy ( over time I hope to fix that ).
Other images are not even as floppy images. As an example, all the
data that I have for my Nicolet floppy disk system is images on
paper tape data ( in files on my PC ).
And of course, I also have cassette data images.
Some people like to store cassette images as audio files. I like to store
the actual data. I have done this for my Poly88 stuff and have a bootstrap
method to get things started without a first tape.
I also have images in a format that I'm just too lazy to convert. As an
example,
the images I made for my H89. These also have some legacy issues, as
my tool was used to create the images for the hard sectored images
on SEBHC. I've included a method to bootstrap without having a first
disk.
Of course, there should always be some form of readme with each
group of files explaining how to use the files. In the past, this has been
one of the hardest things to figure out on random archives. Many times,
I've seen images with no information on what to do with them. This
has happened because the directory was copied from someplace without
the support. Each directory should have at least a simple readme describing
how the images were made and what tool should be used to recover
them. Just a readme at the top directory is not enough. The top directory
may get separated. A readme is minimum.
These readme's may need updating as time goes by. Things like the fact
that some tool needs to run under a particular OS and not some future OS
that
may be more common, needs to be passed on. It is hard to tell what bit
of information that is assumed today may be key for extracting useful
data in the future.
It must be a living archive.
Just my thoughts
Dwight
_________________________________________________________________
Watch free concerts with Pink, Rod Stewart, Oasis and more. Visit MSN
Presents today.
http://music.msn.com/presents?icid=ncmsnpresentstagline&ocid=T002MSN03A…
= How to go about building it
A tricky question - Who's going to do all this work? The good
news is that it can be shared, but it will require a bit of a
committment. Not everyone is going to want to put up the
investment to acquire the means to process their disk collection.
What I am thinking of is several key volenteers positioned so as
to cover the major geographic areas who would be willing to set
up the necessary software and equipment to handle as many
different media types as possible, and provide a service to
nearby collectors to turn media into images, and to turn images
into media. Working with others in the project, the images
could be transmitted and shared so that any particuylar system
disk can be accessed from anywhere.
That much makes sense to me. I've been steadily collecting various relevant
odds and ends for the past couple of years with a view to doing just that -
various drives, controllers, tape units etc.
In order to spread this around as much as possible, we could probably do
with some "style guide" documents produced by some of the list gurus as to
what hardware works with what etc. (e.g. summarising things like the
responses to my recent question about hooking an 8" floppy drive up to a
PC). That will hopefully encourage more people to participate.
(There's certainly been a lot of knowledge shared on this list in the past
about recovering old tapes too - it'd be nice to see that written up
somewhere "central")
Hmmm, it's not really a FAQ, but there's perhaps scope for something like a
"classiccmp.org knowledge base" containing info like this?
= How to make the archive available
Another tricky question, which has two major components, legality
and accessability.
I still think the key here is distribution. Allow people who have
specialist knowledge in a particular area to run their own little corner of
the web as they see fit - but they also "publish" what they've made
available through some defined mechanism. In addition, they present some
unified way of searching across *all* participating sites for content.
To extend that concept a little further (thinking on my feet here), how's
about we say that each "published" bit of data can be marked with different
levels:
"Offline" (say pending copyright issue resolution!)
Free for download (i.e. available to all users)
Available for mirror (to specific peers)
Available for mirror (to anyone wishing to make a mirror of the item)
Obvious benefits:
1) The "specialists" in any given area retain control over what they
have available, the format that it's in, and the "look and feel" of
whatever frills surround the actual data.
2) There's no unwieldy central repository, with corresponding high cost
of maintenance.
3) Mirroring is more controllable.
4) Searching (from a user POV) can be flexible and tailored to certain
pre-defined "classes" of content.
5) Copyright is at least a little less of an issue; if someone publishes
something that violates copyright, it's far more likely to come down on
their own head than jeaporising the distributed archive as a whole.
6) Dictating a "common archival format" that everyone agrees on probably
isn't possible, for various reasons. Look at what happened with Sellam's
efforts. However, dictating a common set of different content types (disk
image, documentation scan etc.), and the metadata fields that they can be
searched for with, is a heck of a lot easier!
7) Admin is a lot easier, as every participating site owner is an admin
of the stuff that *they* make available, rather than having one person (or
a team) trying to look after everything - Al can probably comment on what a
headache this is!
Downsides:
1) Probably needs a web interface at least for the searching (but that's
not to stop someone making the actual data available over FTP or whatever,
and the search interface code could be produced in a variety of languages -
PHP, ASP etc. - to provide flexibility)
2) The actual database of "what's where" probably still needs to be
central, because distributing that across multiple sites (ala DNS) is
likely impractical, plus the majority of websites probably do have some
server-side scripting ability, but not necessarily any kind of coherent
database support) 
19 Mar
19 Mar
2:18 p.m.
New subject: Don Maslin/Archiving system software (was: ftparchives disappearing?)
dwight elvey wrote:
I thought I'd add a little here. One thing that is
missing. We'd need
some way to keep it running beyond when we are gone. There should
be a trust setup.
Hmmm, I wonder if it can be engineered such that mirroring the information
will do that job? (i.e. sites are free to come and go - providing there's a
mirror active *somewhere* then it might not be a problem). Although yes,
worrying about what happens over time seems sensible - although maybe that's a
task for site owners to sort out individually, rather than there being some
"central policy"?
It should have at least two or three people
maintaining
is and willing to deal with the tax issues.
Tax issues are probably largely a US issue :-) In other words outside the
scope of any archive/collection...
I tend to agree that it shouldn't force format.
Dave's open format is
vary well thought out and his tool is good but there is legacy data
that would need to be converted.
I'd disagree on the *need* to convert. There may be cases where migration's
desirable as/when new formats are available / become popular - but it
shouldn't be a given. If the format does the job and the tools are still
widely available [1] then there's little reason to convert formats.
[1] Actually "archival tool" is probably a class of data that should be made
available / distributed / mirrored in its own right. After all, the people
"owning" the data at participating sites are most likely to be the ones who
recognise the need to make the tools available too.
As an example, the disk image
data I have for the Polymorphic 8813 is in the format used by the
Polymorphic software to move disk images through modems to other
Polymorphic machines. It does currently lack a method to restore it
to a machine that has no boot floppy ( over time I hope to fix that ).
See, that probably is a good candidate for archive format migration - *but*
it's still very useful to have the data out there and available now, even if
it's in a slightly sub-optimal format. Something's better than nothing...
Of course, there should always be some form of readme
with each
group of files explaining how to use the files. In the past, this has been
one of the hardest things to figure out on random archives.
I hear ya. I'm not quite sure how to tackle that one given a distributed
archive approach. Maybe a user should be expected to be able to type in
something like "reading foobar archives" into the search facility and to get a
document describing the format back.
But I guess for any class of data like "floppy disk image" or "magtape
image"
part of the metadata associated with that data (entered when the data is
"published") should be the name / version of the tool used to create that
image. That way at least every file gets tagged, with the assumption that the
documentation for that format is also reachable via the search engine.
These readme's may need updating as time goes by.
Things like the fact
that some tool needs to run under a particular OS and not some future OS
that may be more common, needs to be passed on.
Is that not a problem for whoever's maintaining the tools? They're the ones
who dictate what platforms their tools will run on - it's not necessarily the
job of the archive maintainers to know this; they simply record what tool (and
hence what file format) was used to create their images. The burden's on the
individual users to search for something that will understand that format, and
then filter out results to get something that will work with their particular
hardware combination.
e.g. I might have a site with a bunch of Superbrain floppies on, all in
Imagedisk format. It's my job as maintainer of that archive (and participating
in the "global" distributed archive) to say what tool I used to create those
images when I publish them - but it's Dave's job as Imagedisk maintainer to
dictate what other platforms Imagedisk might run on now or in the future.
Sure, I might be helpful and (with Dave's permission) put a few different
copies of Imagedisk on the site too and publish those so people can find them
- but I don't need to do any maintenance myself every time a new version of
Imagedisk comes out.
It is hard to tell what bit
of information that is assumed today may be key for extracting useful
data in the future.
Indeed - and it's easy to say "record as much as possible". Problem then is
that it discourages archive owners from publishing content simply because it's
time-consuming to enter the metadata for the items that they're making
available. Getting the balance right is probably going to be tricky.
cheers
Jules
4:07 p.m.
On 19 Mar 2007 at 13:18, Jules Richardson wrote:
Indeed - and it's easy to say "record as much
as possible". Problem then is
that it discourages archive owners from publishing content simply because it's
time-consuming to enter the metadata for the items that they're making
available. Getting the balance right is probably going to be tricky.
I thought I'd share a little tidbit here. On floppy conversion jobs,
it's rare for us to use a PC diskette controller. I've long found
that using a catweasel or similar flux-transition capture tool
results in a much higher data recovery rate--and it's rare that I
have to read a diskette more than once in order to determine the
format or recording method. This can be an issue with very old
diskettes, which can degrade with each read.
When we return converted data a customer, we'll also generally
include the flux state map as documentation. The needed space on a
CD or DVD usually comes nearly "free" and there's a permanent record.
I don't know if FTP space will ever be cheap enough that recording
flux maps on them for public access will ever be economical, but it's
certainly something to think about if preserving "images" and not
just converted data becomes important.
FWIW,
Chuck
4:30 p.m.
New subject: Don Maslin/Archiving system software (was: ftparchives disappearing?)
Chuck Guzis wrote:
I don't know if FTP space will ever be cheap
enough that recording
flux maps on them for public access will ever be economical, but it's
certainly something to think about if preserving "images" and not
just converted data becomes important.
I think the key thing to aim for is to record as much as necessary such that
it would be possible to recreate the original media (as far as the original
hardware sees it) if desired.
In other words, record flux traversals where required (because some arcane
copy protection / validation scheme needs it, say, or flux traversals actually
allow easier recreation of the original media) - but in the vast majority of
cases a simple structured bunch of data blocks is good enough for the machine
to see it as though it were an original copy.
Note that I say "aim for" - personally I'd like the data out there sooner
rather than later, even if it's in a sub-optimal format; it can always be
improved later as and when time (and the original media) permits. Something's
generally better than nothing...
Plus of course there's a real danger we'll focus on floppy disks here at the
expense of all else. Just a reminder not to lose sight of all the other forms
of media out there that are equally as deserving of preservation! :-)
cheers
Jules
5:09 p.m.
New subject: Don Maslin/Archiving system software (was: ftparchives disappearing?)
These
readme's may need updating as time goes by. Things like the fact
that some tool needs to run under a particular OS and not some future OS
that may be more common, needs to be passed on.
Is that not a problem for whoever's maintaining the tools? They're the ones
who dictate what platforms their tools will run on - it's not necessarily the
job of the archive maintainers to know this; they simply record what tool (and
hence what file format) was used to create their images. The burden's on the
individual users to search for something that will understand that format, and
then filter out results to get something that will work with their particular
hardware combination. from now i86 PC's have all declared dangerous
materials (brain damage etc.), and
can't be had - you might not be able to run
my tools - but given an incentive to
put some effort into it, the information is there to allow you to use other means.
(how that information will survive is another question - but at least I've made it
available which is the best I can do).
The last thing we want to do is lock up the archive in proprietary undocumented
formats.
e.g. I might have a site with a bunch of Superbrain
floppies on, all in
Imagedisk format. It's my job as maintainer of that archive (and participating
in the "global" distributed archive) to say what tool I used to create those
images when I publish them - but it's Dave's job as Imagedisk maintainer to
dictate what other platforms Imagedisk might run on now or in the future.
I think "dictate" is a string word - If someone wants to develop the tool for
another platform, I would be very supportive - however this hasn't happened
so far.
Sure, I might be helpful and (with Dave's
permission) put a few different
copies of Imagedisk on the site too and publish those so people can find them
- but I don't need to do any maintenance myself every time a new version of
Imagedisk comes out.
The sites should carry what tools and documentation can be made available
(ultimately ability to access the archived data will deternine it's acceptance).
Indeed - and it's easy to say "record as much
as possible". Problem then is
that it discourages archive owners from publishing content simply because it's
time-consuming to enter the metadata for the items that they're making
available. Getting the balance right is probably going to be tricky.
I'm not sure I agree with that. You don't need much metadata to be 1000%
more useful than none. I would not expect complete documentation on the
tool and archive formats to be included in each data image - but the type
of system it is for, the name and version of the tool, and a description of
what is in the image should be enough to get you going.
People pay $$$ for a CD that comes in an empty box with a little paper
saying "This is XXX put the CD in your drive" - surely we can do at least
as good as that.
Dave
--
dave06a (at) Dave Dunfield
dunfield (dot) Firmware development services & tools: www.dunfield.com
com Collector of vintage computing equipment:
http://www.classiccmp.org/dunfield/index.html
6:21 p.m.
New subject: Don Maslin/Archiving system software (was: ftparchives disappearing?)
dave06a at dunfield.com wrote:
Can't resist a chance to climb up on my facvorite soapbox - much more important
than the tools, is details and docuemtnation on the archive format.
Oh, I'm right with you there (it's one of my favourites too ;)
Which actually leads to a related question: do we need a resource documenting
media formats? *file* formats seem to be pretty well covered on the 'net, but
I've not seen anywhere really documenting how floppies, tapes etc. for old
systems are laid out. I'm not really talking about at the filesystem level
either, but lower than that at the raw data stream level.
Doing so would certainly seem a useful resource for anyone wanting to help
contribute to the software repository (in terms or what media they're able to
help archive)
(I think someone else asked the same question on here a month or two back - in
which case apologies for not speaking up and saying that it seemed like a good
idea)
Yep, you're right... I was just (badly) saying that I think it's easy to muddy
the waters between tool maintainer and archive maintainer.
Absolutely. I was just pointing out that it's possible to go overboard - as an
example, if it were to take someone 30 minutes to fill in the metadata for a
single floppy disk image then I suspect the temptation will be there for
people not to bother participating in the repository.
So there is such a thing as "too much". So for any class of data
(documentation, floppy image, tape image, archival tool etc. etc.) it just
needs careful consideration at design time as to what metadata is important
and needs to be mandatory.
These readme's may need updating as time goes by.
Things like the fact
that some tool needs to run under a particular OS and not some future OS
that may be more common, needs to be passed on.
Is that not a problem for
whoever's maintaining the tools? They're the ones
who dictate what platforms their tools will run on - it's not necessarily the
job of the archive maintainers to know this; they simply record what tool (and
hence what file format) was used to create their images. The burden's on the
individual users to search for something that will understand that format, and
then filter out results to get something that will work with their particular
hardware combination. e.g. I might
have a site with a bunch of Superbrain floppies on, all in
Imagedisk format. It's my job as maintainer of that archive (and participating
in the "global" distributed archive) to say what tool I used to create those
images when I publish them - but it's Dave's job as Imagedisk maintainer to
dictate what other platforms Imagedisk might run on now or in the future.
I think "dictate" is a string word - If someone wants to develop the tool for
another platform, I would be very supportive - however this hasn't happened
so far. The sites should carry what tools and documentation
can be made available
(ultimately ability to access the archived data will deternine it's acceptance).
I don't think they *have* to though. It's logical for them to do so certainly,
but but for the purposes of searching / mirroring / storage the tools can be
treated purely as items of data - the critical thing (again) is widespread
availability in sufficient numbers, not where the tools available for download
happen to be located. But any road, it's all just semantics :-)
Indeed - and
it's easy to say "record as much as possible". Problem then is
that it discourages archive owners from publishing content simply because it's
time-consuming to enter the metadata for the items that they're making
available. Getting the balance right is probably going to be tricky.
I'm not sure I agree with that. You don't need much metadata to be 1000%
more useful than none. I would not expect complete documentation on the
tool and archive formats to be included in each data image - but the type
of system it is for, the name and version of the tool, and a description of
what is in the image should be enough to get you going. People pay $$$ for a CD that comes in an empty box
with a little paper
saying "This is XXX put the CD in your drive" - surely we can do at least
as good as that.
Oh, we can do a lot better :) It's just a case of reaching some happy medium
that captures as much useful metadata as possible but doesn't put people off
from adding content in the first place.
cheers
Jules
4:51 p.m.
New subject: Don Maslin/Archiving system software (was: ftparchives disappearing?)
Hi
I thought I'd add a little here. One thing that is missing. We'd need
some way to keep it running beyond when we are gone. There should
be a trust setup. It should have at least two or three people maintaining
is and willing to deal with the tax issues.
I don't know about that - we would have to see how it goes, but if enough
people are involved, individuals can come and go but the group continues.
The whole idea is to NOT make it dependant on any individual. I can
only see a problem arising if "everyone loses interest", in which case
we would have to revisit the thinking behind doing it in the first place...
I tend to agree that it shouldn't force format.
Dave's open format is
vary well thought out and his tool is good but there is legacy data
that would need to be converted. As an example, the disk image
data I have for the Polymorphic 8813 is in the format used by the
Polymorphic software to move disk images through modems to other
Polymorphic machines. It does currently lack a method to restore it
to a machine that has no boot floppy ( over time I hope to fix that ).
Other images are not even as floppy images. As an example, all the
data that I have for my Nicolet floppy disk system is images on
paper tape data ( in files on my PC ).
And of course, I also have cassette data images.
Some people like to store cassette images as audio files. I like to store
the actual data. I have done this for my Poly88 stuff and have a bootstrap
method to get things started without a first tape.
I also have images in a format that I'm just too lazy to convert. As an
example,
the images I made for my H89. These also have some legacy issues, as
my tool was used to create the images for the hard sectored images
on SEBHC. I've included a method to bootstrap without having a first
disk.
One concept that I really want to promote is that people involved in the
project can provide material to people who are unable to recreate the
medium themselves - in much the same way Don sent out physical
disks. Theres a lot of people who simply can't deal with how to get an
image back to physical media.
In line with this idea, I would hope that the key people would be fairly
well equiped to handle varous formats. With the equipment available,
we could (over time of course) convert existing data to the more general
formats (where appropriate) - this would not be mandated in any way,
but it would be a something that individuals can contribute that would
benefit the others.
Of course, there should always be some form of readme
with each
group of files explaining how to use the files. In the past, this has been
one of the hardest things to figure out on random archives. Many times,
I've seen images with no information on what to do with them. This
has happened because the directory was copied from someplace without
the support. Each directory should have at least a simple readme describing
how the images were made and what tool should be used to recover
them. Just a readme at the top directory is not enough. The top directory
may get separated. A readme is minimum.
These readme's may need updating as time goes by. Things like the fact
that some tool needs to run under a particular OS and not some future OS
that may be more common, needs to be passed on. It is hard to tell what bit
of information that is assumed today may be key for extracting useful
data in the future.
It must be a living archive.
Absolutely - part of the job will be to document the details. The creation,
collection, archival and documentation of appropriate tools is also key - this
is NOT a small job.
Dave
--
dave06a (at) Dave Dunfield
dunfield (dot) Firmware development services & tools: www.dunfield.com
com Collector of vintage computing equipment:
http://www.classiccmp.org/dunfield/index.html
6330
days inactive
6330
days old
6 comments
4 participants
participants (4)
-
cclist＠sydex.com -
dave06a＠dunfield.com -
dkelvey＠hotmail.com -
julesrichardsonuk＠yahoo.co.uk