17 May
2005
17 May
'05
7:03 p.m.
If it *is* portable, it might seem a better choice for archives over
tar, simply because more systems these days can handle zip files than
can handle tar files...
--
Writing a program to unpack a tar archive is not difficult.
The problem I see with zip is the single table of contents at the end.
Did you try corrupting THAT with a hex editor?
The file headers in a tar archive appear in front of each file. It is
generally possible to resync after errors in the decoding stream.
17 May
17 May
7:44 p.m.
On Tue, 2005-05-17 at 16:03 -0700, Al Kossow wrote:
If it *is* portable, it might seem a better choice for
archives over
tar, simply because more systems these days can handle zip files than
can handle tar files...
--
Writing a program to unpack a tar archive is not difficult.
Sure - but more current systems typically have zip tools on them than
tar IMHO... my thought was more one of how likely someone else would be
of being able to handle the archive with whatever they had to hand.
The problem I see with zip is the single table of
contents at the end.
Did you try corrupting THAT with a hex editor?
Ahh, no not at the time. I've just tried it now though and it seems
remarkably good at recovering from corruption in the TOC area. Actually,
looking at the zip file it appears to have something resembling a file
header before each file in the archive as well as the TOC at the end.
Whether that's always the case or some sort of optional extra, I don't
know.
cheers
Jules
18 May
18 May
2:30 p.m.
Jules Richardson wrote:
As long as we're talking about fault-tolerant archives, neither TAR nor ZIP are
acceptable. For years I've used RAR (WinRAR for windows, RAR and RAR32 for
DOS) which has "recovery record" support (parity info). A "recovery
record"
usually burns about 1% additional space (configurable up to 10%) but can
completely recover mangled compressed data if the errors are small enough (ie
no more than 512 bytes at a time, at a certain minimum distance from the next
error, etc.). For larger archives, RAR supports parity *files*, so if you
split a large archive into, say, 10 parts and 3 recovery files, you can lose up
to any 3 of all 13 files and still be able to recover everything. I do this
when archiving data to DVD-R and it has saved my butt once (BOTH DVD sets got
bitrot because of a flood).
ZIP was never built to be fault tolerant, and trying to recover a mangled TAR
file goes completely pants if the TAR has other TARs inside it.
There *are* tools that generate parity information for archive files that don't
have it themselves... The generated files look like *.PAR. Unfortunately I
don't recall the names of them, but a quick google should find them.
--
Jim Leonard (trixter at oldskool.org) http://www.oldskool.org/
Want to help an ambitious games project? http://www.mobygames.com/
Or check out some trippy MindCandy at http://www.mindcandydvd.com/
The problem I
see with zip is the single table of contents at the end.
Did you try corrupting THAT with a hex editor?
Ahh, no not at the time. I've just tried it now though and it seems
remarkably good at recovering from corruption in the TOC area. Actually,
looking at the zip file it appears to have something resembling a file
header before each file in the archive as well as the TOC at the end. 
2:45 p.m.
>>>> "Jim" == Jim Leonard
<trixter at oldskool.org> writes:
Jim> Jules Richardson wrote:
>> The problem I see with zip is the single table
of contents at the
>> end. Did you try corrupting THAT with a hex editor?
> Ahh, no not at the time. I've just tried it now though and it
> seems remarkably good at recovering from corruption in the TOC
> area. Actually, looking at the zip file it appears to have
> something resembling a file header before each file in the archive
> as well as the TOC at the end.
Jim> As long as we're talking about fault-tolerant archives, neither
Jim> TAR nor ZIP are acceptable. For years I've used RAR (WinRAR for
Jim> windows, RAR and RAR32 for DOS) which has "recovery record"
Jim> support (parity info). ...
If you want fault tolerance, it may be a good idea to learn the topic
of "erasure codes" -- a general concept for way to split data into N+K
pieces such that you can reconstruct the data from any N pieces (for N
and K chosen to be whatever you wish).
VMS also implemented the XOR thing you mentioned in the BACKUP
utility (as did RSTS, of course -- since it supports the same
format).
paul
4:10 p.m.
Paul Koning wrote:
If you want fault tolerance, it may be a good idea to
learn the topic
of "erasure codes" -- a general concept for way to split data into N+K
pieces such that you can reconstruct the data from any N pieces (for N
and K chosen to be whatever you wish).
Yes, that's what I already wrote about in my previous post :-)
--
Jim Leonard (trixter at oldskool.org) http://www.oldskool.org/
Want to help an ambitious games project? http://www.mobygames.com/
Or check out some trippy MindCandy at http://www.mindcandydvd.com/
19 May
19 May
1:11 p.m.
At 2:45 PM -0400 5/18/05, Paul Koning wrote:
Error Correcting and Detecting Codes are a huge topic in and of
themselves. Fire Codes and others are designed to correct for
particular expected error characteristics (e.g., contiguous bit
errors of some maximum length, block errors, random bit errors,
etc.). I'm not sure about completely arbitrary N and K; last I
looked there were tradeoffs and optimizations based on the expected
characteristics.
For example, the error characteristics (and thus detection/correction
capability) of a space-earth link is different from that of a 9-track
magtape, which is different from that of any sort of disk.
>>>> "Jim" == Jim Leonard <trixter at oldskool.org> writes:
Jim> Jules Richardson wrote:
>> The problem I see with zip is the single
table of contents at the
>> end. Did you try corrupting THAT with a hex editor?
> Ahh, no not at the time. I've just tried it now though and it
> seems remarkably good at recovering from corruption in the TOC
> area. Actually, looking at the zip file it appears to have
> something resembling a file header before each file in the archive
> as well as the TOC at the end.
Jim> As long as we're talking about fault-tolerant archives, neither
Jim> TAR nor ZIP are acceptable. For years I've used RAR (WinRAR for
Jim> windows, RAR and RAR32 for DOS) which has "recovery record"
Jim> support (parity info). ...
If you want fault tolerance, it may be a good idea to learn the topic
of "erasure codes" -- a general concept for way to split data into N+K
pieces such that you can reconstruct the data from any N pieces (for N
and K chosen to be whatever you wish). VMS also implemented the XOR thing you mentioned in the
BACKUP
utility (as did RSTS, of course -- since it supports the same
format).
Yes, Andy Goldstein (or whoever, but I think Andy wrote the first
version) really did a nice job in VMS BACKUP. I once wrote a program
to read such tapes on a Unix system and needed to learn the
techniques used. Very nice. Indeed an entire /BLOCK_SIZE block of
data within one /GROUP_SIZE could be completely corrupted on tape,
yet BACKUP could correctly restore it.
John
18 May
18 May
4:34 p.m.
Is there a reason to settle on a single format rather than two? Store
both a zip file and a gzipped tar file. (Or, for that matter, why only
keep a single copy of an archive)? A cheap ($100) hard drive holds
about 312,000 uncompressed DS/SD 5.25" floppy images. Next year's
drive will hold 500K. I'd guess that's enough for every CP/M software
package ever published (plus source code) with room left over for
every paper tape currently in existence and a few VMS distributions.
I usually store a zip file, a .tar.bz2 file and a bz2 compressed
Floppy/CD image where applicable. Copy protected CDs are a problem
because I don't know of an open format, so I'm currently using
VirtualDrive to make backup images of them. Compressed ISOs work fine
for unprotected disks.
My current (home) archiving technique is to have two networked RAID-1
setups. One is old technology (in the case a pair of 30GB drives).
The other is newer tech (a pair of 100GB drives). I've just ordered
a pair of 400G drives to replace the 30G drives. When they arrive,
I'll archive the 30G drives to tape, I'll copy everything from the
100G on to the new drives. The the 100G drives become read only, and
the 400G drives become the active archive. The 30G drives go into
sealed CO2 storage. The active archive always contains a copy of the
older archives and gets backed up regularly. When 1.5TB drives hit
the market, the process repeats itself (probably with new hardware,
since there might not be 1.5TB ATA drives).
If an archive format conversion is required (as was for .ar[ck] to
.zip or .tar.gz to .tar.bz2), it's easy enough for a script to handle
the job. Since I also keep source code to gtar, gzip, and an unzip on
the drives and executables in for every machine and OS I've ever used,
things should be recoverable, if kept up to date.
Eric
On 5/17/05, Al Kossow <aek at bitsavers.org> wrote:
If it *is* portable, it might seem a better choice for
archives over
tar, simply because more systems these days can handle zip files than
can handle tar files...
--
Writing a program to unpack a tar archive is not difficult.
The problem I see with zip is the single table of contents at the end.
Did you try corrupting THAT with a hex editor?
The file headers in a tar archive appear in front of each file. It is
generally possible to resync after errors in the decoding stream.
7006
days inactive
7007
days old
6 comments
6 participants
participants (6)
-
aek＠bitsavers.org -
dundas＠caltech.edu -
julesrichardsonuk＠yahoo.co.uk -
korpela＠gmail.com -
pkoning＠equallogic.com -
trixter＠oldskool.org