11 Aug
2004
11 Aug
'04
4:13 a.m.
as Sellam had suggested, the data size we represent the information in is not that
important. I would encode binary data as hex to keep everything ascii. Data size would
expand, but the data would also be compressable so things could be kept in ZIP files of
whatever choice a person would want to for their archiving purposes.
XML has become a storage format choice for a lot of different commercial packages. My
knowledge is more based on the windows world, but I would doubt that other computer
software houses are avoiding XML. Sun/Java certainly embraces it.
I don't quite understand why representing binary as hex would affect the ability to
have command line utilitities. Certainly more cpu cycles are needed for conversion and
image file size is larger, but we need a readable format and I would think that cpu cycles
is not as much of a concern or file size. A utility to create a disk would only have to
run through the conversion once to buffer a representation of the floppy disk (unless we
are talking about a hard drive image of course). The file size to re-create a floppy disk
is only going to be 2 to 3meg at the most (if thinking about a 1.2meg floppy with
fortmatting info).
The only difference I see in the sections that were described is that the first one
encompasses the format info and the data. My description had the first one as being a big
block that contained the two other sections as well as length and CRC info to verify data
consistency. Adding author, etc to the big block would make perfect sense.
As for GCR, that would have been covered under etc... I am not familiar with GCR, but I
would guess that it has to deal at least with physical tracks and heads. In this case, a
track would consist of whatever the format needed plus the data blocks required for the
track.
best regards, Steve Thatcher
-----Original Message-----
From: Jules Richardson <julesrichardsonuk(a)yahoo.co.uk>
Sent: Aug 11, 2004 8:08 AM
To: "General Discussion: On-Topic and Off-Topic Posts"
<cctalk(a)classiccmp.org>
Subject: Re: Let's develop an open-source media archive standard
On Wed, 2004-08-11 at 10:50, Steve Thatcher wrote:
Hi all, after reading all this morning's posts, I
thought I would throw out some thoughts.
XML as a readable format is a great idea.
I haven't done any serious playing with XML in the last couple of years,
but back when I did, my experience was that XML is not a good format for
mixing human-readable and binary data within the XML structure itself.
To make matters worse, the XML spec (at least at the time) did not
define whether it was possible to pass several XML documents down the
same data stream (or, as we'd likely need for this, XML documents mixed
with raw binary). Typically, parsers of the day expected to take control
of the data stream and expected it to contain one XML document only -
often closing the stream themselves afterwards.
I did end up writing my own parser in a couple of KB of code which was a
little more flexible in data stream handling (so XML's certainly not a
heavyweight format, and could likely be handled on pretty much any
machine), but it would be nice to make use of off-the-shelf parsers for
platforms that have them where possible.
As you've also said, my initial thought for a data format was to keep
human-readable config seperate from binary data. The human-readable
config would contain a table of lengths/offsets for the binary data
giving the actual definition. This does have the advantage that if the
binary data happens to be a linear sequence of blocks (sectors in the
case of a disk image) then the raw image can easily be extracted if
needs be (say, to allow conversion to a different format)
Personally, I'm not a fan of mixing binary data in with the
human-readable parts because then there are issues of character escaping
as well as the structure detracting from the readability. And if encoded
binary data is used instead (say, hexadecimal representation) then
there's still an issue of readability, plus the archive ends up bloated
and extra CPU cycles are needed to decode data. Neither of those two
approaches lend themselves to simply being able to use common
command-line utilities to extract the data, either. I'm prefectly
willing to be convinced, though :)
I looked at the CAPS format and in part that would be
okay. I would like
to throw in an idea of whatever we create as a standard actually have
three sections to it.
So, first section is all the 'fuzzy' data (author, date, version info,
description etc.), second section describes the layout of the binary
data (offsets, surfaces, etc.), and the third section is the raw binary
data itself? If so, I'm certainly happy with that :-)
One aside - what's the natural way of defining data on a GCR floppy? Do
heads/sectors/tracks still make sense as an addressing mode, but it's
just that the number of sectors per track varies according to the track
number? Or isn't it that simple?
cheers
Jules
11 Aug
11 Aug
5:50 a.m.
On Wed, 2004-08-11 at 13:13, Steve Thatcher wrote:
I would encode binary data as hex to keep everything
ascii. Data size would expand,
but the data would also be compressable so things could be kept in ZIP files of
whatever choice a person would want to for their archiving purposes.
"could be kept" in zip files, yes - but then that's no use in 50 years
time if someone stumbles across a compressed file and has no idea how to
decompress it in order to read it and see what it is :-)
Hence keeping the archive small would seem sensible so that it can be
left as-is without any compression. My wild guesstimate on archive size
would be to aim for 110 - 120% of the raw data size if possible...
XML has become a storage format choice for a lot of
different commercial
packages. My knowledge is more based on the windows world, but I would doubt
that other computer software houses are avoiding XML. Sun/Java certainly
embraces it.
My background (in XML terms) is with Java - but I've not come across
software that mixes a human-readable format in with a large amount of
binary data though (whether encoded or not). Typically the metadata's
kept seperate from the binary data itself, either in parallel files (not
suitable in our case) or as seperate sections within the same file.
I don't quite understand why representing binary
as hex would affect the
ability to have command line utilitities.
See my posting the other week when I was trying to convert ASCII-based
hex data back into binary on a Unix platform :-) There's no *standard*
utility to do it (which means there certainly isn't on Windows). If the
data within the file is raw binary, then it's just a case of using dd to
extract it even if there's no high-level utility available to do it.
Certainly more cpu cycles are needed for conversion
and image file size is
larger, but we need a readable format
But the data describing all aspects of disk image would be readable by a
human; it's only the raw data itself that wouldn't be - for both
efficiency and for ease of use. The driving force for having
human-readable data in the archive is so that it can be reconstructed at
a later date, possibly without any reference to any spec, is it not? If
it was guaranteed that a spec was *always* going to be available, having
human-readable data at all wouldn't make much sense as it just
introduces bloat; a pure binary format would be better.
I'm not quite sure what having binary data represented as hex for the
original disk data gives you over having the raw binary data itself -
all it seems to do is make the resultant file bigger and add an extra
conversion step into the decode process.
and I would think that cpu cycles is not as much of a
concern or file size.
In terms of CPU cycles, for me, no - I can't see me ever using the
archive format except on modern hardware. I can't speak for others on
the list though.
As for file size, if encoding as hex that at least doubles the size of
your archive file compared to the original media (whatever it may be).
That's assuming no padding between hex characters. Seems like a big
waste to me :-(
The only difference I see in the sections that were
described is that
the first one encompasses the format info and the data. My description
had the first one as being a big block that contained the two other sections
as well as length and CRC info to verify data consistency.
Adding author, etc to the big block would make perfect sense.
Yep, I'm with you there. CRC's are a nice idea. Question: does it make
sense to make CRC info a compulsory section in the archive file? Does it
make sense to have it always present, given that it's *likely* that
these archive files will only ever be transferred from place to place
using modern hardware? I'm not sure. If you're spitting data across a
buggy serial link, then the CRC info is nice to have - but maybe it
should be an optional inclusion rather than mandatory, so that in a lot
of cases archive size can be kept down? (and the assumption made that
there exists source code / spec for a utility to add CRC info to an
existing archive file if desired)
cheers
Jules
6:56 a.m.
----- Original Message -----
From: "Jules Richardson" <julesrichardsonuk(a)yahoo.co.uk>
To: "General Discussion: On-Topic and Off-Topic Posts"
<cctalk(a)classiccmp.org>
Sent: Wednesday, August 11, 2004 10:50 AM
Subject: Re: Let's develop an open-source media archive standard
"could be kept" in zip files, yes - but then
that's no use in 50 years
time if someone stumbles across a compressed file and has no idea how to
decompress it in order to read it and see what it is :-)
Hence keeping the archive small would seem sensible so that it can be
left as-is without any compression. My wild guesstimate on archive size
would be to aim for 110 - 120% of the raw data size if possible...
I'd say ZIP is so ubiquitous, if the format itself is dead and gone there
should be a pile of documentation on how to extract it, especially if you
keep to the basic "deflate" algorithm, used in everything from PNG to gzip
to Jar.
My background (in XML terms) is with Java - but
I've not come across
software that mixes a human-readable format in with a large amount of
binary data though (whether encoded or not). Typically the metadata's
kept seperate from the binary data itself, either in parallel files (not
suitable in our case) or as seperate sections within the same file.
Fork it. Make it a binary file, but keep the metadata in the header
ASCII/Unicode/XML/Whatever.
That way a simple cat or less or opening in notepad will show you what's in
it.
See my posting the other week when I was trying to
convert ASCII-based
hex data back into binary on a Unix platform :-) There's no *standard*
utility to do it (which means there certainly isn't on Windows). If the
data within the file is raw binary, then it's just a case of using dd to
extract it even if there's no high-level utility available to do it.
On the macs there's Binhex which has been the standard since file transfers
have begun. Also, there are dozens of *nix utilties to deal with Base64
(MIME) encoded data.
But the data describing all aspects of disk image
would be readable by a
human; it's only the raw data itself that wouldn't be - for both
efficiency and for ease of use. The driving force for having
human-readable data in the archive is so that it can be reconstructed at
a later date, possibly without any reference to any spec, is it not? If
it was guaranteed that a spec was *always* going to be available, having
human-readable data at all wouldn't make much sense as it just
introduces bloat; a pure binary format would be better.
IMHO if you have a file laying around and you have no idea what it is, it's
still nice to be able to
read it in as text and see what's going on, the Unix "file" command
notwithstanding.
I'm not quite sure what having binary data represented as hex for the
original disk data gives you over having the raw binary data itself -
all it seems to do is make the resultant file bigger and add an extra
conversion step into the decode process.
Usually text moves around multiple machines/oses a lot easier than binary
data. If you can dump a text file to a machine somehow, you can transfer a
media archive and a .c program to compile and decode it.
Yep, I'm with you there. CRC's are a nice idea. Question: does it make
sense to make CRC info a compulsory section in the archive file? Does it
make sense to have it always present, given that it's *likely* that
these archive files will only ever be transferred from place to place
using modern hardware? I'm not sure. If you're spitting data across a
buggy serial link, then the CRC info is nice to have - but maybe it
should be an optional inclusion rather than mandatory, so that in a lot
of cases archive size can be kept down? (and the assumption made that
there exists source code / spec for a utility to add CRC info to an
existing archive file if desired)
I think CRC's should be compulsorary, with an external hash optional.
Corruption can happen on new hardware as well as old :)
cheers
Jules
6:57 a.m.
On Wednesday 11 August 2004 09:50, Jules Richardson wrote:
I'm personally prejudiced against XML, but that's just me. : )
You could decode it by hand (ick) or write a Q&D program to do it for
you. I'd hope that *programming* won't be a lost art in 50+ years.
Again, producing paper copies of stuff with non-printable characters
becomes "problematic".
Then use uuencode or similar that does a bit less wasteful conversion.
Anyways, the only computer media type where KB/in^2 (or KB/in^3) isn't
increasing rapidly is paper.
Yes. A CRC is *always* a good idea. Or, you could do an ECC even ;)
I don't really understand why you're quite so concerned about archive
size bloat, at least over things like CRC's (which if applied liberally
might add a 4% bloat in size) or plain-text encoding (which would add
between 33% to about 100% to the size). I'd rather give up some
efficiency in this case for ensuring that the data is stored correctly,
and can be properly read (and easily decoded) in 50+ years.
Pat
--
Purdue University ITAP/RCS --- http://www.itap.purdue.edu/rcs/
The Computer Refuge --- http://computer-refuge.org
On Wed, 2004-08-11 at 13:13, Steve Thatcher wrote:
UNIX's compress format has been around for decades now... has its
patents expired yet? If not, there's always gzip...
I would encode binary data as hex to keep
everything ascii. Data
size would expand, but the data would also be compressable so
things could be kept in ZIP files of whatever choice a person would
want to for their archiving purposes.
"could be kept" in zip files, yes - but then that's no use in 50
years time if someone stumbles across a compressed file and has no
idea how to decompress it in order to read it and see what it is :-) Hence keeping the archive small would seem sensible so
that it can be
left as-is without any compression. My wild guesstimate on archive
size would be to aim for 110 - 120% of the raw data size if
possible...
You could uuencode the data... For long term archiving it might be
suggestable to have plain-text paper copies of the data (hoping in 50+
years they'll have decent OCR technology :). So, you *need* something
that only uses some sane subset of characters, like what uuencode or
BASE64 encoding gives you. Uuencode is a bit older so I'd tend to
lean towards that over BASE64.
XML has become
a storage format choice for a lot of different
commercial packages. My knowledge is more based on the windows
world, but I would doubt that other computer software houses are
avoiding XML. Sun/Java certainly embraces it.
My background (in XML terms) is with Java - but I've not come across
software that mixes a human-readable format in with a large amount of
binary data though (whether encoded or not). Typically the metadata's
kept seperate from the binary data itself, either in parallel files
(not suitable in our case) or as seperate sections within the same
file. I don't
quite understand why representing binary as hex would
affect the ability to have command line utilitities.
See my posting the other week when I was trying to convert
ASCII-based hex data back into binary on a Unix platform :-) There's
no *standard* utility to do it (which means there certainly isn't on
Windows). If the data within the file is raw binary, then it's just a
case of using dd to extract it even if there's no high-level utility
available to do it. Certainly more
cpu cycles are needed for conversion and image file
size is larger, but we need a readable format
I'm not quite sure what having binary data represented as hex for the
original disk data gives you over having the raw binary data itself -
all it seems to do is make the resultant file bigger and add an extra
conversion step into the decode process. and I would
think that cpu cycles is not as much of a concern or
file size.
In terms of CPU cycles, for me, no - I can't see me ever using the
archive format except on modern hardware. I can't speak for others on
the list though.
As for file size, if encoding as hex that at least doubles the size
of your archive file compared to the original media (whatever it may
be). That's assuming no padding between hex characters. Seems like a
big waste to me :-( The only
difference I see in the sections that were described is
that the first one encompasses the format info and the data. My
description had the first one as being a big block that contained
the two other sections as well as length and CRC info to verify
data consistency. Adding author, etc to the big block would make
perfect sense.
Yep, I'm with you there. CRC's are a nice idea. Question: does it
make sense to make CRC info a compulsory section in the archive file?
Does it make sense to have it always present, given that it's
*likely* that these archive files will only ever be transferred from
place to place using modern hardware? I'm not sure. If you're
spitting data across a buggy serial link, then the CRC info is nice
to have - but maybe it should be an optional inclusion rather than
mandatory, so that in a lot of cases archive size can be kept down?
(and the assumption made that there exists source code / spec for a
utility to add CRC info to an existing archive file if desired)
7:48 a.m.
On Wed, 2004-08-11 at 15:57, Patrick Finnegan wrote:
You could decode it by hand (ick) or write a Q&D program to do it for
you. I'd hope that *programming* won't be a lost art in 50+ years.
Well I ended up doing the latter - I was actually quite surprised there
was no standard util to take a stream of hex characters, stripping out
any junk, and writing out the resulting binary (sort of a reverse
hexdump).
On Wednesday 11 August 2004 09:50, Jules Richardson
wrote:
Unix goes pop in what, 2038 though? :-)
Maybe zip's not the ideal example. My point really is that if the
archives are enormous, people are going to be tempted to compress them.
If they compress them, what guarantee is there that a) the compression
method is going to be around when someone totally unrelated wants to
handle these files in x years, and b) is it going to be obvious to
someone in x years what compression method was even used to compress the
file?
Again, it's back to longevity of the archives themselves. If something's
needed for the short term (next ten years, say), it's not a problem. But
it'd be nice if a future generation, upon discovering one of these
archives, could know exactly what it was (and stand a good chance of
decoding it) just by looking at it (hence the human-readable part)
"could be kept" in zip files, yes - but
then that's no use in 50
years time if someone stumbles across a compressed file and has no
idea how to decompress it in order to read it and see what it is :-)
UNIX's compress format has been around for decades now... has its
patents expired yet? If not, there's always gzip... You could uuencode the data... For long term
archiving it might be
suggestable to have plain-text paper copies of the data (hoping in 50+
years they'll have decent OCR technology :). So, you *need* something
that only uses some sane subset of characters, like what uuencode or
BASE64 encoding gives you. Uuencode is a bit older so I'd tend to
lean towards that over BASE64.
Again, I don't like the idea of anything happening to the archive files
after creation though. I suppose the data from the raw device (floppy,
hard disk, whatever) within the archive could be encoded somehow
(leaving the config section as plain-text) - providing it's in a common
enough format that we think someone will be able to find the spec for
the encoding method in x years and so be able to get at the data. That's
somewhat hard to say for sure though!
I'm personally prejudiced against XML, but
that's just me. : )
It has its uses; as a hierarchical storage mechanism for plain text it's
good IMHO. But I wouldn't use it for small or human-editable config
files, or in any situation where the data you're encapsulating is
totally swamped by the surrounding markup.
I don't quite understand why representing binary
as hex would
affect the ability to have command line utilitities.
See my posting the other week when I was trying to convert
ASCII-based hex data back into binary on a Unix platform :-) There's
no *standard* utility to do it (which means there certainly isn't on
Windows). If the data within the file is raw binary, then it's just a
case of using dd to extract it even if there's no high-level utility
available to do it. Again, producing paper copies of stuff with
non-printable characters
becomes "problematic".
That's actually an extremely good point, and perhaps the best argument
(IMHO) for not using binary data so far :-) Hmmm...
Yes. A CRC is *always* a good idea. Or, you could do
an ECC even ;)
shush :-)
I don't really understand why you're quite so
concerned about archive
size bloat,
mainly because I have hard drives swamped with scans of paper
documentation :-) Each page is 'only' 2-3MB but it doesn't half add up
quickly.
at least over things like CRC's (which if applied
liberally
might add a 4% bloat in size)
well it's the usual case of analysing and weighing up features against
how often they'll actually be needed, and acting accordingly - otherwise
you end up with something like MS Word's document format and I'm sure
nobody wants to end up there! :)
Seriously, if there's a good argument for having CRC's in more than x
(50?) percent of cases because corrupted data expected to be a real
possibility, then make them mandatory. If not, then make them an
optional extra. I certainly can't see a good reason why they'll *never*
be needed, that's for sure.
or plain-text encoding (which would add
between 33% to about 100% to the size). I'd rather give up some
efficiency in this case for ensuring that the data is stored correctly,
and can be properly read (and easily decoded) in 50+ years.
With you on the longevity side of things. Hmm, off the wall suggestion,
but it's only the storage format for the raw data that's an issue,
right? So does it make sense to define both binary and ASCII
representation as valid storage formats, and the format in use within a
particular archive is recorded as a parameter within the human-readable
config section?
That's no different to most common image formats, say, where data might
be uncompressed, or compressed in several different ways.
Only a fraction of any program capable of producing such an archive
would be given over to the raw data encode stage, so it doesn't add any
complexity really. Utilities to convert between formats should be pretty
trivial; no more complex than decoding the archive in the first place
anyway. Nothing to stop someone at a future date - say in 30 years -
wanting to convert an archive to paper format, even if it's in a format
which doesn't lend itself to this - the spec defines the pure-ASCII
method and so conversion is possible at that point.
In this way those wanting compact archives to save space, run against
various existing utilities etc. can have them containing binary data;
those who think they need ASCII representation of the data due to tool
or transmission medium limitations can use that format - all whilst
maintining compatibility with the spec. (potentially the 'encoding
method' parameter could include other defined types - uuencode, base64
etc. but let's not get ahead of ourselves...)
(funny how someone mentioned IFF files earlier; I keep on thinking of
TIFF images where the data's structured and the format both versioned
and maintained under strict control)
cheers
Jules
8:03 a.m.
Unix goes pop in what, 2038 though? :-)
Only if you insist on still using signed 32-bit values for time.
I fully expect that by 2038 int, never mind long, will be at least 64
bits wide, which will push the "go pop" date well past the expected
lifetime of our solar system.
[...] - I was actually quite surprised there was no
standard util to
take a stream of hex characters, stripping out any junk, and writing
out the resulting binary (sort of a reverse hexdump).
cvtbase x d | code-to-char
Of course, cvtbase and code-to-char are both my own work, but anyone
who wants either is welcome to a copy. (ftp.rodents.montreal.qc.ca, in
/mouse/hacks/cvtbase.c and /mouse/local/src/charcode/.)
/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML mouse(a)rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
8:38 a.m.
Jules Richardson wrote:
On Wed, 2004-08-11 at 15:57, Patrick Finnegan wrote:
*32-bit* Unix goes pop in 2038. Since every currently-maintained
Unix (or Unix-alike for you OpenGroup fanatics) runs on 64-bit hardware
and in 64-bit mode, I don't see it as a problem.
The fact that most Unix admins are even aware of the 2038 issue
infers that it won't be the problem that Y2K was.
Doc
On Wednesday 11 August 2004 09:50, Jules
Richardson wrote:
Unix goes pop in what, 2038 though? :-) "could be kept" in zip files, yes - but
then that's no use in 50
years time if someone stumbles across a compressed file and has no
idea how to decompress it in order to read it and see what it is :-)
UNIX's compress format has been around for decades now... has its
patents expired yet? If not, there's always gzip...
8:45 a.m.
Doc Shipley wrote:
Jules Richardson wrote:
Oh, crap, what a fubar! Unforgiveable, too, since my PDP-11 is
running 2.11BSD downstairs....
s/every currently-maintained/almost every currently-maintained/
Doc
On Wed, 2004-08-11 at 15:57, Patrick Finnegan
wrote:
*32-bit* Unix goes pop in 2038. Since every currently-maintained Unix
(or Unix-alike for you OpenGroup fanatics) runs on 64-bit hardware and
in 64-bit mode, I don't see it as a problem. On Wednesday 11 August 2004 09:50, Jules
Richardson wrote:
Unix goes pop in what, 2038 though? :-) "could be kept" in zip files, yes - but
then that's no use in 50
years time if someone stumbles across a compressed file and has no
idea how to decompress it in order to read it and see what it is :-)
UNIX's compress format has been around for decades now... has its
patents expired yet? If not, there's always gzip... 
9:41 a.m.
On Wed, 11 Aug 2004, Jules Richardson wrote:
Agreed.
Maybe zip's not the ideal example. My point really
is that if the
archives are enormous, people are going to be tempted to compress them.
If they compress them, what guarantee is there that a) the compression
method is going to be around when someone totally unrelated wants to
handle these files in x years, and b) is it going to be obvious to
someone in x years what compression method was even used to compress the
file?
Yep. Good points. Primarily, the human psychology aspect: large file
sizes will compel people to want to compress the images, quite possibly
ruining the effort of making the images to begin with. The spec should
be designed such that it allows for the smallest filesize possible.
Again, it's back to longevity of the archives
themselves. If something's
needed for the short term (next ten years, say), it's not a problem. But
it'd be nice if a future generation, upon discovering one of these
archives, could know exactly what it was (and stand a good chance of
decoding it) just by looking at it (hence the human-readable part)
Right. This is the whole reason for designing a spec like this.
Again, I don't like the idea of anything happening
to the archive files
after creation though. I suppose the data from the raw device (floppy,
hard disk, whatever) within the archive could be encoded somehow
(leaving the config section as plain-text) - providing it's in a common
enough format that we think someone will be able to find the spec for
the encoding method in x years and so be able to get at the data. That's
somewhat hard to say for sure though!
Or at least be able to figure it out. Encoding data in a wider base, such
as in hex or Base64, still allows a smart human to figure it out. If we
add meta compression, this will also need to be readily decodable by a
smart human.
Again,
producing paper copies of stuff with non-printable characters
becomes "problematic".
That's actually an extremely good point, and perhaps the best argument
(IMHO) for not using binary data so far :-) Hmmm... Seriously, if there's a good argument for having
CRC's in more than x
(50?) percent of cases because corrupted data expected to be a real
possibility, then make them mandatory. If not, then make them an
optional extra. I certainly can't see a good reason why they'll *never*
be needed, that's for sure.
I'd make them an optional extra, with the default assuming no CRCs. In
fact, the spec should be designed in such a way that as litte as possible
is assumed. Any encoding features should have to be explicitly invoked in
the image header.
With you on the longevity side of things. Hmm, off the
wall suggestion,
but it's only the storage format for the raw data that's an issue,
right? So does it make sense to define both binary and ASCII
representation as valid storage formats, and the format in use within a
particular archive is recorded as a parameter within the human-readable
config section?
I still don't like it. As Roger M. pointed out, what will the binary data
look like after it's been paraded through several different platforms?
In this way those wanting compact archives to save
space, run against
various existing utilities etc. can have them containing binary data;
those who think they need ASCII representation of the data due to tool
or transmission medium limitations can use that format - all whilst
maintining compatibility with the spec. (potentially the 'encoding
method' parameter could include other defined types - uuencode, base64
etc. but let's not get ahead of ourselves...)
I'd rather give the option of being able to specify which text-based
encoding scheme was used (i.e. base16, base64, etc.)
(funny how someone mentioned IFF files earlier; I keep
on thinking of
TIFF images where the data's structured and the format both versioned
and maintained under strict control)
So it should be with this specification.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
11:43 a.m.
On Wed, 2004-08-11 at 18:41, Vintage Computer Festival wrote:
On Wed, 11 Aug 2004, Jules Richardson wrote:
Well going back to the image example, it's like trying to ftp a TIFF
image to a remote server in ASCII mode. It's going to mess up, but
anyone with any smarts either knows not to transfer in that way because
it's going to mess up, or they inspect the data and realise that it
can't be transferred as ASCII. If people didn't learn by being told
something, common sense, or from their mistakes, then we probably all
wouldn't still be here :-)
Same goes for any line termination issues; archivists would not be daft
enough to transfer data from one medium to the next and then destroy the
original (or if they did destroy the original, they'd be damned sure
that the copy matched the original *exactly* first). Whilst the average
person does make silly mistakes and then typically learns from them, I'd
expect anyone preserving or dealing with historical data to have a few
more smarts about them in the first place!
And if someone *does* want to transfer the data between systems and
there's *really* no way of turning off features that will mangle binary
data - well, that's exactly why we design the spec with lots of
flexibility in mind, and they can always convert the data to a different
format should they wish.
All a question of providing lots of choices whilst keeping within a
strict specification so that people can choose what suits them
personally I suppose, whilst knowing that the data should still be
usable in the future...
cheers,
Jules
With you on the longevity side of things. Hmm,
off the wall suggestion,
but it's only the storage format for the raw data that's an issue,
right? So does it make sense to define both binary and ASCII
representation as valid storage formats, and the format in use within a
particular archive is recorded as a parameter within the human-readable
config section?
I still don't like it. As Roger M. pointed out, what will the binary data
look like after it's been paraded through several different platforms?
12:24 p.m.
On Wed, 11 Aug 2004, Jules Richardson wrote:
Hmmm. Ok, stupid me. Several years back, I transferred all my stored
e-mails (both received and sent) from the ISP I used to use to my own
server. I gzipped the files so they would be smaller and then used FTP to
transfer them over. I then deleted the original files and went on my
merry business.
Sometime later, when I wanted to access those files, they would not
unzip. It was then that I realized I ftp'd them in ASCII mode. DOH!
DOH! DOH!
So I'm not terribly smart, but I'm no George Bush either. I forgot that I
had to put FTP into binary mode before transferring those files.
This sort of error can propagate quite easily. I'd rather not rely on
people to remember to switch to binary mode when they transfer large
archives of imaged media. And I'd hate for an archive that was
transferred this way to become a defacto archive of a specific set of
important imaged data, only to find out years or decades down the road
that they are completely useless because the binary data is missing its
eight bit.
I still
don't like it. As Roger M. pointed out, what will the binary data
look like after it's been paraded through several different platforms?
Well going back to the image example, it's like trying to ftp a TIFF
image to a remote server in ASCII mode. It's going to mess up, but
anyone with any smarts either knows not to transfer in that way because
it's going to mess up, or they inspect the data and realise that it
can't be transferred as ASCII. If people didn't learn by being told
something, common sense, or from their mistakes, then we probably all
wouldn't still be here :-) Same goes for any line termination issues; archivists
would not be daft
enough to transfer data from one medium to the next and then destroy the
original (or if they did destroy the original, they'd be damned sure
that the copy matched the original *exactly* first). Whilst the average
Haha. You're assuming everyone is as competent as you are :)
person does make silly mistakes and then typically
learns from them, I'd
expect anyone preserving or dealing with historical data to have a few
more smarts about them in the first place!
Or not. And at any rate, this specification should be as foolproof as is
possible. You can't assume that everyone who's going to be making these
archives or maintaining them is going to be smart. You just can't. The
specification should be as robust as possible, and then more so.
All a question of providing lots of choices whilst
keeping within a
strict specification so that people can choose what suits them
personally I suppose, whilst knowing that the data should still be
usable in the future...
As I said, I would implement a binary data tag, but would strongly
discourage its use.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
12 Aug
12 Aug
2:35 a.m.
Am 11 Aug 2004 14:24 meinte Vintage Computer Festival:
On Wed, 11 Aug 2004, Jules Richardson wrote:
> > I still don't like it. As Roger M. pointed out, what will the binary data
> > look like after it's been paraded through several different platforms?
> Well going back to the image example, it's
like trying to ftp a TIFF
> image to a remote server in ASCII mode. It's going to mess up, but
> anyone with any smarts either knows not to transfer in that way because
> it's going to mess up, or they inspect the data and realise that it
> can't be transferred as ASCII. If people didn't learn by being told
> something, common sense, or from their mistakes, then we probably all
> wouldn't still be here :-)
Hmmm. Ok, stupid me. Several years back, I
transferred all my stored
e-mails (both received and sent) from the ISP I used to use to my own
server. I gzipped the files so they would be smaller and then used FTP to
transfer them over. I then deleted the original files and went on my
merry business.
Sometime later, when I wanted to access those files,
they would not
unzip. It was then that I realized I ftp'd them in ASCII mode. DOH!
DOH! DOH!
So I'm not terribly smart, but I'm no George
Bush either. I forgot that I
had to put FTP into binary mode before transferring those files.
This sort of error can propagate quite easily.
I'd rather not rely on
people to remember to switch to binary mode when they transfer large
archives of imaged media. And I'd hate for an archive that was
transferred this way to become a defacto archive of a specific set of
important imaged data, only to find out years or decades down the road
that they are completely useless because the binary data is missing its
eight bit.
I vey much like that reasoining. Ok, your argument might be considered
weak by a lot of people, since Zips, Exes and such are quite common
offered over the web nowadays, but still, it shows exactly my concern
with binary data on the way between machines ...
Gruss
H.
--
VCF Europa 6.0 am 30.April und 01.Mai 2005 in Muenchen
http://www.vcfe.org/
11 Aug
11 Aug
12:43 p.m.
Rumor has it that Jules Richardson may have mentioned these words:
[snip]
Same goes for any line termination issues; archivists
would not be daft
enough to transfer data from one medium to the next and then destroy the
original (or if they did destroy the original, they'd be damned sure
that the copy matched the original *exactly* first). Whilst the average
person does make silly mistakes and then typically learns from them, I'd
expect anyone preserving or dealing with historical data to have a few
more smarts about them in the first place!
Rule #1: *Never* underestimate the power of human stupidity.
Rule #2: If you ever forget this, look at the people who run your country.
Rule #3: If you wonder about the sanity of Rule #2, remember, for most
countries on this earth, *you put 'em there.*
;-)
Laterz,
Roger "Merch" Merchberger
P.S. What was that one museum quoted recently that didn't give a ratz rump
about keeping their computers running (CHM was it?) but gutted 'em, put
lightstrings around them and said "lookey-here." Remember, *they* are our
archivists... [[Hopefully, we can change that, but it's a downhill slide,
my friend... ;-) ]]
--
Roger "Merch" Merchberger --- sysadmin, Iceberg Computers
zmerch(a)30below.com
Hi! I am a .signature virus. Copy me into your .signature to join in!
9:28 a.m.
On Wed, 11 Aug 2004, Patrick Finnegan wrote:
Another point worth highlighting. The image files should be readily
printable, so that a complete image could be recovered by "manual OCR"
(i.e. typing it in) from a printed form of the image file. Keep in mind:
a printed form of the image file, on the right physical medium, will
probably outlast any magnetic or optical media.
I like the idea of Base64, because it's something that can still be
readily decoded manually by a human (converting between number bases is
easy once you understand the concept). I'm not knowledgeable as to how
the uuencode format is encoded.
The type of encoding used could be specified as a meta tag, perhaps even
per each data segment. Say it would be more efficient to encode the data
in hex for one part and then Base64 for another. Each part would have a
meta tag added describing which base was used for the encoding. Just an
idea...
I'm not
quite sure what having binary data represented as hex for the
original disk data gives you over having the raw binary data itself -
all it seems to do is make the resultant file bigger and add an extra
conversion step into the decode process.
Again, producing paper copies of stuff with non-printable characters
becomes "problematic". As for file
size, if encoding as hex that at least doubles the size
of your archive file compared to the original media (whatever it may
be). That's assuming no padding between hex characters. Seems like a
big waste to me :-(
Then use uuencode or similar that does a bit less wasteful conversion.
Anyways, the only computer media type where KB/in^2 (or KB/in^3) isn't
increasing rapidly is paper. I don't really understand why you're quite so
concerned about archive
size bloat, at least over things like CRC's (which if applied liberally
might add a 4% bloat in size) or plain-text encoding (which would add
between 33% to about 100% to the size). I'd rather give up some
efficiency in this case for ensuring that the data is stored correctly,
and can be properly read (and easily decoded) in 50+ years.
Exactly. Given the current trends of hard drives, image filesize is not
an issue. Human readability is the prime concern.
Of course, if a file is too large to be, say, printed (practically) onto
paper, that could also be problematic. Again, this is getting back to the
issue of a perpetual media, which is a separate project.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
1:09 p.m.
I like the idea of Base64, because it's something
that can still be
readily decoded manually by a human (converting between number bases
is easy once you understand the concept).
I'd hardly say "readily". Hex I can read by eye; base64 I would use a
program for, even if it's a five-minute one-off hack. (I can write a
rudimentary base64 unpacker in fifteen minutes; I know because I once
did. Fifteen minutes would be unlikely to be enough time for me to
decode more than a few tens of bytes, without computer assist.)
I'm not knowledgeable as to how the uuencode
format is encoded.
It's conceptually almost identical to base64: it's a base-64
representation, representing three octets of binary data as four
encoded characters. The major differences are probably that uuencode
has a length field on each encoded line and uses a different set of 64
characters.
"man 5 uuencode" on your friendly neighbourhood NetBSD box should
explain it - I'll be happy to send the manpage (nroff input or
formatted output, as desired) to anyone who wants a copy.
If it's a choice between base64 and uuencode, I'd say go with base64 -
it's more resistant to mangling by non-ASCII systems. If you're
willing to commit to ASCII, btoa might be better - it uses base 85,
expanding by 4:5 rather than the 3:4 expansion produced by base64 or
uuencode.
However, the only reason I can see to use base64 instead of hex is a
desire to expand by only 3:4 instead of 1:2, and in my opinion that
verges on penny-wise and pound-foolish given the fundamental goals of
this archive format. As Sellam writes,
Given the current trends of hard drives, image
filesize is not an
issue. Human readability is the prime concern.
/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML mouse(a)rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
7:03 a.m.
Rumor has it that Jules Richardson may have mentioned these words:
I'm not quite sure what having binary data
represented as hex for the
original disk data gives you over having the raw binary data itself -
all it seems to do is make the resultant file bigger and add an extra
conversion step into the decode process.
Different systems interpret binary data differently, especially if there is
human-readable code within the file. MS-DOS very likely could think the
file is binary, and the first hex 0x06 would terminate the file. However,
if it is considered binary many viewers wouldn't open it if it's got
non-printable ASCII characters within.
Or -- how about transferring this file from MS-DOS <=> Linux <=> MacOS
<=>
BeOS ... will it (and if so, how will it) convert line ending chars, tabs &
other binary chars? A hex representation will reduce conversion problems
measurably - remember, this is supposed to be an "ultra-portable" format.
As for file size, if encoding as hex that at least
doubles the size of
your archive file compared to the original media (whatever it may be).
That's assuming no padding between hex characters. Seems like a big
waste to me :-(
Not necessarily - IIRC, Moto S-records only have records for where there's
actually data, right? So if the disk is only 1/2 full of data, the S-record
would reflect that, right?
If we're already going to take this "beyond" the spec, couldn't we
institute some RLE encoding as well?
Just thoughts,
Roger "Merch" Merchberger
--
Roger "Merch" Merchberger --- sysadmin, Iceberg Computers
Recycling is good, right??? Randomization is better!!!
If at first you don't succeed, nuclear warhead
disarmament should *not* be your first career choice.
9:32 a.m.
On Wed, 11 Aug 2004, Roger Merchberger wrote:
Different systems interpret binary data differently,
especially if there is
human-readable code within the file. MS-DOS very likely could think the
file is binary, and the first hex 0x06 would terminate the file. However,
if it is considered binary many viewers wouldn't open it if it's got
non-printable ASCII characters within.
Or -- how about transferring this file from MS-DOS <=> Linux <=> MacOS
<=>
BeOS ... will it (and if so, how will it) convert line ending chars, tabs &
other binary chars? A hex representation will reduce conversion problems
measurably - remember, this is supposed to be an "ultra-portable" format.
All very good points, which is why binary data should not be included.
If we're already going to take this
"beyond" the spec, couldn't we
institute some RLE encoding as well?
The spec will definitely benefit from some sort of compression feature
that can be implemented using tags, as was described previously.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
11:07 a.m.
On Wed, 11 Aug 2004, Roger Merchberger wrote:
Different systems interpret binary data differently,
especially if there is
human-readable code within the file. MS-DOS very likely could think the
file is binary, and the first hex 0x06 would terminate the file. However,
I don't think so. 06h is a PUSH ES command, which is valid in
programs/binary files. My .EXE and .COM files that use that
instruction do NOT terminate prematurely.
But your point is still valid, in that a 1Ah (^Z) byte
WILL cause many text processing utilities to terminate
processing of a text file in MS-DOS or CP/M.
Or -- how about transferring this file from MS-DOS
<=> Linux <=> MacOS <=>
BeOS ... will it (and if so, how will it) convert line ending chars, tabs &
other binary chars? A hex representation will reduce conversion problems
measurably - remember, this is supposed to be an "ultra-portable" format.
That's why any reading of a binary file, or an alien text file,
must fopen in "RAW" mode!
If we're already going to take this
"beyond" the spec, couldn't we
institute some RLE encoding as well?
Many folks seem to think that a .ZIP file is unreadable,
and impossible to write replacements readers, if the
now existing utilities aren't available.
(as opposed to files on a ZIP disk, where none of the
drives can be expected to remain working.)
7:09 a.m.
Jules Richardson wrote:
Binary data is usually encoded in an XML file into a some type of text:
base64, hex/octal, etc. You don't want actual raw binary data embeded
in an XML document.
But how do you know that DD will exist in 50 years?
It isn't hard to write a simple hex-bin and bin-hex program.
Couple of lines of perl, and you are done. The main problem is
there isn't a real hex dump standard. There are several hex
conversions in 'recode' on linux.
Binary data can get badly munged during transmission across the internet,
with ascii-ebcdic conversions, 7-bit/8-bit paths, etc. Not as much a problem
as it used to be. A text version doesn't suffer these problems.
If you then ZIP the file, it will likely become smaller than the original.
With the availability of programs like zcat/zmore, reading it wouldn't be
much of a problem either.
The necessity of CRC's depend on what you plan on doing with the data.
If it is just going to be sitting in a nice, safe box, then it doesn't
matter.
It it is going to be tossed all over the place, through strange networks,
around the world, bounced off mars, etc. then it becomes much more
important.
Depending on how large a chunk you CRC, the size differencial can be
very minimal. A CRC for every byte, expensive, for one track, minimal.
On Wed, 2004-08-11 at 13:13, Steve Thatcher wrote:
I'd believe that ZIP archives will be well known in the future, due to
the large number of software products using it right now, and the
length of time is has existed so far. (Famous last words?)
The compression methods are fairly well known, and there exist several
open source libraries to handle it, aswell as being embeded in many new
projects.
OpenOffice/StarOffice, as an example, uses zipped xml files as its
native save format. This fact alone should help keep the ZIP format
alive with the large number of groups/companies/governments/countries
switching to OpenOffice.
I would encode binary data as hex to keep
everything ascii. Data size would expand,
but the data would also be compressable so things could be kept in ZIP files of
whatever choice a person would want to for their archiving purposes.
"could be kept" in zip files, yes - but then that's no use in 50 years
time if someone stumbles across a compressed file and has no idea how to
decompress it in order to read it and see what it is :-)
Hence keeping the archive small would seem sensible so that it can be
left as-is without any compression. My wild guesstimate on archive size
would be to aim for 110 - 120% of the raw data size if possible...
XML has become
a storage format choice for a lot of different commercial
packages. My knowledge is more based on the windows world, but I would doubt
that other computer software houses are avoiding XML. Sun/Java certainly
embraces it.
My background (in XML terms) is with Java - but I've not come across
software that mixes a human-readable format in with a large amount of
binary data though (whether encoded or not). Typically the metadata's
kept seperate from the binary data itself, either in parallel files (not
suitable in our case) or as seperate sections within the same file.
I don't
quite understand why representing binary as hex would affect the
ability to have command line utilitities.
See my posting the other week when I was trying to convert ASCII-based
hex data back into binary on a Unix platform :-) There's no *standard*
utility to do it (which means there certainly isn't on Windows). If the
data within the file is raw binary, then it's just a case of using dd to
extract it even if there's no high-level utility available to do it.
Certainly more
cpu cycles are needed for conversion and image file size is
larger, but we need a readable format
But the data describing all aspects of disk image would be readable by a
human; it's only the raw data itself that wouldn't be - for both
efficiency and for ease of use. The driving force for having
human-readable data in the archive is so that it can be reconstructed at
a later date, possibly without any reference to any spec, is it not? If
it was guaranteed that a spec was *always* going to be available, having
human-readable data at all wouldn't make much sense as it just
introduces bloat; a pure binary format would be better.
I'm not quite sure what having binary data represented as hex for the
original disk data gives you over having the raw binary data itself -
all it seems to do is make the resultant file bigger and add an extra
conversion step into the decode process.
and I would
think that cpu cycles is not as much of a concern or file size.
In terms of CPU cycles, for me, no - I can't see me ever using the
archive format except on modern hardware. I can't speak for others on
the list though.
As for file size, if encoding as hex that at least doubles the size of
your archive file compared to the original media (whatever it may be).
That's assuming no padding between hex characters. Seems like a big
waste to me :-(
The only
difference I see in the sections that were described is that
the first one encompasses the format info and the data. My description
had the first one as being a big block that contained the two other sections
as well as length and CRC info to verify data consistency.
Adding author, etc to the big block would make perfect sense.
Yep, I'm with you there. CRC's are a nice idea. Question: does it make
sense to make CRC info a compulsory section in the archive file? Does it
make sense to have it always present, given that it's *likely* that
these archive files will only ever be transferred from place to place
using modern hardware? I'm not sure. If you're spitting data across a
buggy serial link, then the CRC info is nice to have - but maybe it
should be an optional inclusion rather than mandatory, so that in a lot
of cases archive size can be kept down? (and the assumption made that
there exists source code / spec for a utility to add CRC info to an
existing archive file if desired)
9:20 a.m.
On Wed, 11 Aug 2004, Kevin Handy wrote:
I'd believe that ZIP archives will be well known
in the future, due to
the large number of software products using it right now, and the
length of time is has existed so far. (Famous last words?)
The compression methods are fairly well known, and there exist several
open source libraries to handle it, aswell as being embeded in many new
projects.
People are welcome to create archives of zipped images, but that should be
something not even addressed in the spec. The spec defines how the image
will be represented. How people choose to store the images is outside the
scope of the spec.
Binary data is usually encoded in an XML file into a
some type of text:
base64, hex/octal, etc. You don't want actual raw binary data embeded
in an XML document.
Agreed. Using Base64 would be nice to further reduce the data size, and
should be considered. I like the idea of using hex because it's
well-known and therefore very easy to decode.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
2:19 p.m.
The necessity of CRC's depend on what you plan on
doing with the data.
If it is just going to be sitting in a nice, safe box, then
it doesn't
matter.
True. Unless you've gone to the trouble of archiving the
stuff because you would like it to be reliably retrievable
at some point in the future.
If reliable retrieval in the future is of no concern,
then I propose a format conisting of an empty file.
At least you'll be certain of what you get :-)
Seriously, after all the discussion of bit-rot on
various media, how can you consider some sort of
checksum to be anything other than essential?
Some sort of error-correcting code might be an
even better idea.
Antonio
--
---------------
Antonio Carlini arcarlini(a)iee.org
12 Aug
12 Aug
10:23 a.m.
On Thu, 12 Aug 2004, Antonio Carlini wrote:
If reliable retrieval in the future is of no concern,
then I propose a format conisting of an empty file.
At least you'll be certain of what you get :-)
Seriously, after all the discussion of bit-rot on
various media, how can you consider some sort of
checksum to be anything other than essential?
Some sort of error-correcting code might be an
even better idea.
I've considered that the spec should include the ability to encode in
several different DEFINED formats, using different DEFINED checksum
schemes.
An error-correcting CRC scheme would definitely be one.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
11 Aug
11 Aug
8:20 a.m.
Jules Richardson wrote:
On Wed, 2004-08-11 at 13:13, Steve Thatcher wrote:
There's going to have to be _some_ assumption of continuity or this
project is hopeless. It's my opinion that short-term or long-term,
anyone who wants or needs access to the data** will have some historical
understanding of the computing environment contemporary to that data.
Provisions for changing technology and loss of continuity are good,
but we still need to draw the line at some point, especially concerning
"external" archival of the archived data, i.e. zipped arcives and
storage media. To go to an extreme example, even printed ASCII on paper
or mylar isn't reliable. Paper may be just as archaic as hieroglyphics
when the data is wanted.
We have to depend on ongoing maintenance of these archives. If they
are not periodically migrated to current media, and if the
attached/imbedded documentation is not augmented to account for social
and technical "loss of memory", future retrieval will be difficult, if
not impossible, no matter what we do now.
Doc
** Technically speaking. We can't provide for, say a non-technical
attorney who wants recorded files as evidence. That example will end up
in the hands of someone like us.
I would encode binary data as hex to keep
everything ascii. Data size would expand,
but the data would also be compressable so things could be kept in ZIP files of
whatever choice a person would want to for their archiving purposes.
"could be kept" in zip files, yes - but then that's no use in 50 years
time if someone stumbles across a compressed file and has no idea how to
decompress it in order to read it and see what it is :-)
9:46 a.m.
On Wed, 11 Aug 2004, Doc Shipley wrote:
Provisions for changing technology and loss of
continuity are good,
but we still need to draw the line at some point, especially concerning
"external" archival of the archived data, i.e. zipped arcives and
storage media. To go to an extreme example, even printed ASCII on paper
or mylar isn't reliable. Paper may be just as archaic as hieroglyphics
when the data is wanted.
We have to depend on ongoing maintenance of these archives. If they
are not periodically migrated to current media, and if the
attached/imbedded documentation is not augmented to account for social
and technical "loss of memory", future retrieval will be difficult, if
not impossible, no matter what we do now.
That's where the Longnow Foundation may be able to play a rather
significant part in the development of this specification.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
10:41 a.m.
----- Original Message -----
From: "Vintage Computer Festival" <vcf(a)siconic.com>
To: "General Discussion: On-Topic and Off-Topic Posts"
<cctalk(a)classiccmp.org>
Sent: Wednesday, August 11, 2004 2:46 PM
Subject: Re: Let's develop an open-source media archive standard
That's where the Longnow Foundation may be able to
play a rather
significant part in the development of this specification.
If it were up to them the spec would be encoded in integer-binary form on a
string of beryllium-cobalt-copper links, fed into a giant water-powered
mechanical difference engine tabulation computer to reproduce the works on
stainless steel discs in any number of base-n numbering systems :)
This article from them pretty much sums it up.
http://www.longnow.org/10klibrary/library.htm
Cool bunch of people. I saw them on Big Thinkers back when TechTV was good.
Now it's time to write up my idea for the Long Now clock... Is it me or do
many of these look like the time controller machine in Tomb Raider (The
Movie)?
http://www.longnow.org/10kclock/clockother.htm
11:29 a.m.
On Wed, 11 Aug 2004, Jason McBrien wrote:
Well, thank goodness we won't be having them actually develop the spec
itself :) What I had in mind with them is to help to put into place a
perpetual system of regeneration so that the image library can be updated
as long as there are humans around to do so.
That's
where the Longnow Foundation may be able to play a rather
significant part in the development of this specification.
If it were up to them the spec would be encoded in integer-binary form on a
string of beryllium-cobalt-copper links, fed into a giant water-powered
mechanical difference engine tabulation computer to reproduce the works on
stainless steel discs in any number of base-n numbering systems :) Cool bunch of people. I saw them on Big Thinkers back
when TechTV was good.
Now it's time to write up my idea for the Long Now clock... Is it me or do
many of these look like the time controller machine in Tomb Raider (The
Movie)?
http://www.longnow.org/10kclock/clockother.htm
This is a really cool project that I follow (from time to time) with
great interest. I wish I could be more involved but I've already got a
million projects going on.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
1:55 p.m.
Vintage Computer Festival wrote:
Well, thank goodness we won't be having them
actually develop the spec
itself :) What I had in mind with them is to help to put into place a
perpetual system of regeneration so that the image library can be updated
as long as there are humans around to do so.
Nope, after the collapse of civlization and some He-man sword swinger and
female Sex-clone discover the records using forgotten 'magic'.
OK I read too much.
Ben.
9:15 a.m.
On Wed, 11 Aug 2004, Jules Richardson wrote:
I agree, but we'd definitely have to include compression features if we
are to meet this goal. Using a floppy disk as an example, a worst case
scenario is that the image would be maybe 205% the size of the original
media (200% is the fact that you are now using two bytes to store one, and
5% is all the markup tags).
Keeping the archive small should be a major goal, since that would
encourage people to keep the images stored uncompressed. Hard drives are
getting larger and all that, and my guess is at some point this issue will
be moot, but we can't know that for certain, so we should always assume a
worst case scenario (i.e. pessimism will be useful when designing this
specification :)
This is a point that needs to be highlighted. These images are meant to
be human readable, first and foremost. Machine readable is a secondary
concern. We know there will definitely be humans in the future (and if
not then who cares about this anyway). There will probably be machines.
Said machines may not be useful to the task of decoding these images, so
it must be designed with human readability in mind.
I would
encode binary data as hex to keep everything ascii. Data size would expand,
but the data would also be compressable so things could be kept in ZIP files of
whatever choice a person would want to for their archiving purposes.
"could be kept" in zip files, yes - but then that's no use in 50 years
time if someone stumbles across a compressed file and has no idea how to
decompress it in order to read it and see what it is :-)
Hence keeping the archive small would seem sensible so that it can be
left as-is without any compression. My wild guesstimate on archive size
would be to aim for 110 - 120% of the raw data size if possible... Certainly more
cpu cycles are needed for conversion and image file size is
larger, but we need a readable format
But the data describing all aspects of disk image would be readable by a
human; it's only the raw data itself that wouldn't be - for both
efficiency and for ease of use. The driving force for having human-readable data in the archive is so that it can
be reconstructed at
a later date, possibly without any reference to any spec, is it not? If
Indeed.
it was guaranteed that a spec was *always* going to be
available, having
human-readable data at all wouldn't make much sense as it just
introduces bloat; a pure binary format would be better.
Correct. So even if the spec was lost, people (who could read English at
least) would be able to figure out how to reconstruct the image from the
archive.
I'm not quite sure what having binary data
represented as hex for the
original disk data gives you over having the raw binary data itself -
all it seems to do is make the resultant file bigger and add an extra
conversion step into the decode process.
But it also makes it human readable, and readable in any standard text
editor. Mixing binary data in with human readable data in a format that's
meant, first and foremost, to be human readable is antithetical to the
idea.
As for file size, if encoding as hex that at least
doubles the size of
your archive file compared to the original media (whatever it may be).
That's assuming no padding between hex characters. Seems like a big
waste to me :-(
Nope. Not a waste. Essential.
Yep, I'm with you there. CRC's are a nice
idea. Question: does it make
sense to make CRC info a compulsory section in the archive file? Does it
Yes. It's only one or two added bytes at the end of each data segment.
make sense to have it always present, given that
it's *likely* that
these archive files will only ever be transferred from place to place
using modern hardware? I'm not sure. If you're spitting data across a
buggy serial link, then the CRC info is nice to have - but maybe it
should be an optional inclusion rather than mandatory, so that in a lot
of cases archive size can be kept down? (and the assumption made that
there exists source code / spec for a utility to add CRC info to an
existing archive file if desired)
It doesn't hurt. It only adds negligible overhead. Certainly something
to discuss more.
I would make the specification unassuming about anything like this. For
example, say there is an optional CRC feature. I would make the default
for the image be that there was no CRC added to the data segments, unless
a meta tag was included in the header explicitly specifying that CRCs are
added. This makes it ever so slightly easier to decode the image data by
someone who knows nothing of the spec. No assumptions are made regarding
what people in the future will know about these images.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
11:26 a.m.
On Wed, 2004-08-11 at 18:15, Vintage Computer Festival wrote:
Ahh, I was talking about having the binary data as a seperate section in
the archive to the human-readable part though for precisely that reason.
So, what difference does it make to a human analyst whether the data is
stored as hex pairs or binary data? Both need decoding by some process
to make them usable. A human eye is no better off viewing a stream of
hex digits than they are a stream of arbitary ASCII data.
Actually, binary data could possibly be more useful to the human eye in
a browsing scenario, as at least the eye can quickly pick out meaningful
strings - such as filenames on the original media - from a sea of binary
data, without needing to do any decoding. At least viewing a file
containing binary data could give a clue as to what it contained if the
archive metadata (eg. description) wasn't up to much.
As raised earlier though, I do wonder if it's an idea to define several
possible encoding methods as part of the spec. Maximum flexibility
always seems the key to long-lived data formats, so it perhaps makes a
lot of sense to do so anyway. Who's to tell what use such archives might
be put to in the future - but if the spec covers a reasonable base for
now (with extensibility in mind such that others can be added if needs e
in future versions) then everyone's happy, and future generations can
always convert between formats as they see fit.
Oh, next random thought (which I expect someone's already raised) - the
addressing format for the data on the original media needs to be
flexible enough to cope with different classes of data. Or rather, I'd
expect different addressing classes. For hard disk and floppy archives,
head/sector/track seem a logical addressing scheme. But for say a ROM
image, there's no concept of head/sector/track; maybe just an index to
the data and a length. Maybe someone will want to add scans of
documentation pages to an archive, in which case chapter / page
addressing is logical.
Personally I wouldn't worry too much about what types of information
people might want to store in such an archive at this stage - just be
aware that the flexibility needs to be there just in case :-)
e.g. something like (using vague xml just to get the idea across):
<config>
<metadata>
<version>1.00</version>
<description>blah blah blah</description>
...
</metadata>
<definition>
<dataformat>xyz</dataformat>
<compression>none</compression>
...
<index type="headtracksector">
<block id="0" crc="xyz" head="0"
track="0" sector="0" offset="0">
<block id="1" crc="xyz" head="0"
track="0" sector="1"
offset="256">
...
<block id="12345" crc="xyz" head="15"
track="31" sector="17"
offset="7654321">
</index>
</definition>
</config>
...............
{bunch of data in format xxxx}
...............
Here the index element's 'type' field gives the type of addressing used,
in this case using data blocks split by head / track / sector. An
appropriate handler for that type would then take over and actually
process the data within the index element accordingly.
I used the idea of the media data following the config section here for
clarity, but mixing media data with the semantic data in the index
section would work too (although I'm starting to realise that doing that
is going to make things about as human-readable readable as a typical
large HTML file :-)
I'd say that important field values like the compression type should
always be human-readable rather than a numeric id, just rigidly defined
by the spec (e.g. 'none', 'base64' 'uuencode' etc.). That makes
life a
lot easier for someone potentially looking at this in the future if they
don't happen to have a copy of the spec handy!
Hmm, I miss the old days of everyone chucking ideas around like this :-)
cheers
Jules
On Wed, 11 Aug 2004, Jules Richardson wrote:
I'm not
quite sure what having binary data represented as hex for the
original disk data gives you over having the raw binary data itself -
all it seems to do is make the resultant file bigger and add an extra
conversion step into the decode process.
But it also makes it human readable, and readable in any standard text
editor. Mixing binary data in with human readable data in a format that's
meant, first and foremost, to be human readable is antithetical to the
idea.
12:15 p.m.
On Wed, 11 Aug 2004, Jules Richardson wrote:
So, what difference does it make to a human analyst
whether the data is
stored as hex pairs or binary data? Both need decoding by some process
to make them usable. A human eye is no better off viewing a stream of
hex digits than they are a stream of arbitary ASCII data.
I disagree. I can see "1A". I might not be able to see a CTRL-Z.
Actually, binary data could possibly be more useful to
the human eye in
a browsing scenario, as at least the eye can quickly pick out meaningful
strings - such as filenames on the original media - from a sea of binary
data, without needing to do any decoding. At least viewing a file
containing binary data could give a clue as to what it contained if the
archive metadata (eg. description) wasn't up to much.
In that case, I would suggest someone develop a browser program to
interpret the archive data on the fly if they want to run a "strings" on
it.
As raised earlier though, I do wonder if it's an
idea to define several
possible encoding methods as part of the spec. Maximum flexibility
always seems the key to long-lived data formats, so it perhaps makes a
lot of sense to do so anyway. Who's to tell what use such archives might
be put to in the future - but if the spec covers a reasonable base for
now (with extensibility in mind such that others can be added if needs e
in future versions) then everyone's happy, and future generations can
always convert between formats as they see fit.
I would be perfectly fine with enabling the implantation of binary data
into the archive by having a tag to specify such. But I would strongly
discourage its use.
Oh, next random thought (which I expect someone's
already raised) - the
addressing format for the data on the original media needs to be
flexible enough to cope with different classes of data. Or rather, I'd
expect different addressing classes. For hard disk and floppy archives,
head/sector/track seem a logical addressing scheme. But for say a ROM
image, there's no concept of head/sector/track; maybe just an index to
the data and a length. Maybe someone will want to add scans of
documentation pages to an archive, in which case chapter / page
addressing is logical.
Right, and there will be appropriate tags for each type of medium.
I'd say that important field values like the
compression type should
always be human-readable rather than a numeric id, just rigidly defined
by the spec (e.g. 'none', 'base64' 'uuencode' etc.). That makes
life a
lot easier for someone potentially looking at this in the future if they
don't happen to have a copy of the spec handy!
Totally agreed. There should not be anything cryptic in the tags, and to
the extent possible, they should make sense to a smart human.
Hmm, I miss the old days of everyone chucking ideas
around like this :-)
Well, they're back! ;)
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
2:28 p.m.
I would make the specification unassuming about
anything like
this. For
example, say there is an optional CRC feature. I would make
the default
for the image be that there was no CRC added to the data
segments, unless
a meta tag was included in the header explicitly specifying
that CRCs are
added. This makes it ever so slightly easier to decode the
image data by
someone who knows nothing of the spec. No assumptions are
made regarding
what people in the future will know about these images.
Omitting the CRC when creating the archive is easier than
genrating it and then including it. But when reading the
archive, having the CRC there does not make anything
appreciably harder. You just have to know to ignore
the CRC.
I think - quite strongly - that the protection mechanism
(CRC every so often or whatever), should be mandatory
on creation. It's not hard to generate (or check) these
days and the maths behind it is widely understood. If
our descendants forget how to check them, they can skip
them.
Antonio
--
---------------
Antonio Carlini arcarlini(a)iee.org
10:41 a.m.
On Wed, 2004-08-11 at 13:13, Steve Thatcher wrote:
> I would encode binary data as hex to keep everything ascii. Data size would
expand,
> but the data would also be compressable so things could be kept in ZIP files of
> whatever choice a person would want to for their archiving purposes.
On Wed, 11 Aug 2004, Jules Richardson wrote:
"could be kept" in zip files, yes - but then
that's no use in 50 years
time if someone stumbles across a compressed file and has no idea how to
decompress it in order to read it and see what it is :-)
Keeping documentation with them of how LZW compression works
would not be hard, and writing the decompression routines
is not very hard.
But if you want to keep it VERY simple, then just include a
run-length-encoding, instead of LZW.
Hence keeping the archive small would seem sensible so
that it can be
left as-is without any compression. My wild guesstimate on archive size
would be to aim for 110 - 120% of the raw data size if possible...
But the data describing all aspects of disk image would be readable by a
human; it's only the raw data itself that wouldn't be - for both
efficiency and for ease of use. The driving force for having
human-readable data in the archive is so that it can be reconstructed at
a later date, possibly without any reference to any spec, is it not? If
it was guaranteed that a spec was *always* going to be available, having
human-readable data at all wouldn't make much sense as it just
introduces bloat; a pure binary format would be better.
A human readable form is not going to hit your goal of
110 - 120% unless there is SOME form of compression.
8:45 a.m.
On Wed, 11 Aug 2004, Steve Thatcher wrote:
I don't quite understand why representing binary
as hex would affect the
ability to have command line utilitities. Certainly more cpu cycles are
needed for conversion and image file size is larger, but we need a
readable format and I would think that cpu cycles is not as much of a
concern or file size. A utility to create a disk would only have to run
through the conversion once to buffer a representation of the floppy
disk (unless we are talking about a hard drive image of course). The
file size to re-create a floppy disk is only going to be 2 to 3meg at
the most (if thinking about a 1.2meg floppy with fortmatting info).
I'm leaning away from including a raw binary image of the data in the
archive itself. Sure, it would be handy to have a dual use file, that's
both readily human and machine readable. But as has been pointed out,
there's two issues: mixing characters with binary might be messy, and the
archive size then bloats considerably, which might be a problem for
certain platforms.
The image can always be parsed in realtime by the native platform, if it's
fast enough and has enough resources (RAM, HD space, etc.) A driver of
sorts could translate the archive file in realtime to make it appear as a
floppy or tape or whatever. Creative solutions like connecting to a host
processor over a serial port, having the host processor do the
translation, and then making the serial port look like a disk drive or
whatever on the target system is another option. And then of course, the
image can be simply converted back to it's original format and sent to the
target system so that it can be written out back to its native media.
As for GCR, that would have been covered under etc...
I am not familiar
with GCR, but I would guess that it has to deal at least with physical
tracks and heads. In this case, a track would consist of whatever the
format needed plus the data blocks required for the track.
GCR is no different than MFM in terms of data organization. It's just the
way the bits are read/written from/to the media that's different. GCR has
tracks and sectors just like MFM.
I am taking notes on everything being written, BTW. I'm currently
planning a special session at the VCF to officially launch this as an RFC.
If all the big names I plan to invite actually come (and I think they
might be compelled to do so) it will turn into quite a neat session.
--
Sellam Ismail Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers ]
[ and academia at www.VintageTech.com || at http://marketplace.vintage.org ]
12:01 p.m.
On Wed, 11 Aug 2004, Steve Thatcher wrote:
As for GCR, that would have been covered under etc...
I am not familiar
with GCR, but I would guess that it has to deal at least with physical
tracks and heads. In this case, a track would consist of whatever the
format needed plus the data blocks required for the track.
GCR would be represented in the same field that specifies FM, MFM, RLL,
etc.
But, GCR and RLL have multiple variants. Even within Apple,
there are several mutually incompatible GCRs.
7266
days inactive
7267
days old
33 comments
13 participants
participants (13)
-
a.carlini＠ntlworld.com -
bfranchuk＠jetnet.ab.ca -
cisin＠xenosoft.com -
doc＠mdrconsult.com -
Hans.Franke＠siemens.com -
jbmcb＠hotmail.com -
julesrichardsonuk＠yahoo.co.uk -
kth＠srv.net -
melamy＠earthlink.net -
mouse＠Rodents.Montreal.QC.CA -
pat＠computer-refuge.org -
vcf＠siconic.com -
zmerch＠30below.com