5 Feb
2007
5 Feb
'07
6:37 p.m.
Is everything
you're 'backing up' checksummed?
Yes. I regularly ( == when I remember!) backup my email
by throwing it all into a zip archive and then that gets
written (along with other stuff) to CD, along with an
MD5SUM file. 
5 Feb
5 Feb
6:46 p.m.
In article <45C77952.8050609 at bitsavers.org>,
Al Kossow <aek at bitsavers.org> writes:
I have to implement a digital archiving mechanism for
the
CHM software collection, so this is all VERY fresh in my
mind right now.
What do government and corporate data archives do?
Surely a long-lived company like IBM has already had to deal with this
problem?
--
"The Direct3D Graphics Pipeline" -- DirectX 9 draft available for download
<http://www.xmission.com/~legalize/book/download/index.html>
Legalize Adulthood! <http://blogs.xmission.com/legalize/>
8:35 p.m.
Al Kossow wrote:
Good for you!
Yes, but the md5sum will only detect corruption, not correct it. I use
winrar with parity (what it calls "recovery info") set to 5%, and if I
have gigabytes to archive over several mediums, I use quickpar
http://www.quickpar.org.uk/ to generate the parity info.
--
Jim Leonard (trixter at oldskool.org) http://www.oldskool.org/
Help our electronic games project: http://www.mobygames.com/
Or check out some trippy MindCandy at http://www.mindcandydvd.com/
A child borne of the home computer wars: http://trixter.wordpress.com/
Is everything you're 'backing up'
checksummed?
Yes. I regularly ( == when I remember!) backup my email
by throwing it all into a zip archive and then that gets
written (along with other stuff) to CD, along with an
MD5SUM file. 
10:07 p.m.
Jim Leonard wrote:
Al Kossow wrote:
file.
Good for you!
Yes, but the md5sum will only detect corruption, not correct it. I
use winrar with parity (what it calls "recovery info") set to 5%, and
if I have gigabytes to archive over several mediums, I use quickpar
http://www.quickpar.org.uk/ to generate the parity info.
As I said, most of the CDs and DVDs that have failed have been
entirely unreadable (except usually for the TOC). That suggests
to me that no reasonable recovery mechanism exists (given just
one exemplar of the data in question). I might be wrong (my
sample size of dead media is statistically insignificant as
I have a handful of dead CDs and one dead DVDs out of a
total in the hundreds), but I think I'm better off sticking
with a simple checksum and writing multiple copies of the
media (or, as with my email, having so many copies around
that losing one costs me nothing ... given that I also
keep all that email on disk _and_ regularly copy to an
"online backup" on another disk).
So for stuff that I backup and care about I rely on the
checksums to tell me that my local copy is OK and I rely
on having another copy (or copies) handy in case the
media goes bad.
My only aim is to try to ensure that I can get to the stuff for as
long as I care to get to the stuff. If I wanted to make sure the
stuff was still available for future generations I would have to
do a lot more. For a start I would have to make sure that someone
else has a clue what is backed up and where it might be and why
it might matter. That last one is my major problem :-)
Antonio
Is everything you're 'backing up'
checksummed?
Yes. I regularly ( == when I remember!) backup my email
by throwing it all into a zip archive and then that gets
written (along with other stuff) to CD, along with an > MD5SUM
6 Feb
6 Feb
2:25 a.m.
arcarlini at iee.org wrote:
As I said, most of the CDs and DVDs that have failed
have been
entirely unreadable (except usually for the TOC).
That have failed *you*. Mine have been gradual.
How are you storing yours? Mine go into black folders that are closed
completely (ie. no light gets in) and stored in a cool dry place. The
first CD I ever burnt was in 1996 and it still reads...
--
Jim Leonard (trixter at oldskool.org) http://www.oldskool.org/
Help our electronic games project: http://www.mobygames.com/
Or check out some trippy MindCandy at http://www.mindcandydvd.com/
A child borne of the home computer wars: http://trixter.wordpress.com/
6:40 p.m.
Jim Leonard wrote:
arcarlini at iee.org wrote:
Yes. In my Universe, those are the ones that matter :-)
As I said, most of the CDs and DVDs that have
failed have been
entirely unreadable (except usually for the TOC).
That have failed *you*. Mine have been gradual.
I _think_ many of mine (not yet into double digits) were badly written
rather than victims of decay. I only know for certain of one CD and one
DVD that tested OK at write-time and failed at some later stage. Both
of those were total failures. The very first failure I noticed (the one
that taught me that a checksum of some sort was useful!) was a partial
failure - only a few files were unrecoverable and those were available
elsewhere anyway. I've had many more failed-verification-during-write
cases, and when I play around with those I usually find that only a
few blocks of data are dead.
How are you storing yours? Mine go into black folders that are closed
completely (ie. no light gets in) and stored in a cool dry place. The
first CD I ever burnt was in 1996 and it still reads...
Apart from the non-archival, daily-use ones, mine end up in either
black folders or DJ-style cases. CDs and mushrooms both do much better
in the dark!
Antonio
8:30 p.m.
arcarlini at iee.org wrote:
At what speed do you write them? I think 1x speed would work the best,
but that about a hour for your data.
Mine have been gradual.
I _think_ many of mine (not yet into double digits) were badly written
rather than victims of decay. I only know for certain of one CD and one
DVD that tested OK at write-time and failed at some later stage. Both
of those were total failures. The very first failure I noticed (the one
that taught me that a checksum of some sort was useful!) was a partial
failure - only a few files were unrecoverable and those were available
elsewhere anyway. I've had many more failed-verification-during-write
cases, and when I play around with those I usually find that only a
few blocks of data are dead.
9:33 p.m.
woodelf wrote:
At what speed do you write them? I think 1x speed
would work the
best, but that about a hour for your data.
I tend to write DVD-R at 4x, although the batch I have at the moment
will go down to 2x (but not 1x ... I think that the acceptable speeds
are encoded onto the media). I think that the CD-Rs I have will go down
to x or 8x but I tend to write them at 24x or so. In fact the CD-Rs
are rated at 52x writing (or something equally ridiculous) but the
DVD rewriter drive won't do them at more than 36x or so.
I don't know whether write speed makes much of a difference. I just
think that if it can just about cope with 52x then giving it a little
longer (by writing at say 24x) should mean it doesn't have to turn
the laser up so high to heat up the medium (or maybe it can just
write the medium a little more thoroughly). But I have no idea whether
either of those might be true and I certainly do not have the
equipment to examine the disks physically and check.
Of course, writing at a lower speed should mean I'm less likely to
hit a buffer underrun - although drives mask such problems these
days it does eat away at the available space a little.
Antonio
10:22 p.m.
The initial comments are relevant to this conversation! :-)
Pete Townshend in his studio - part 1 of 2
<http://www.youtube.com/watch?v=39Qi7ggp6_k&mode=related&search=>
--
"The Direct3D Graphics Pipeline" -- DirectX 9 draft available for download
<http://www.xmission.com/~legalize/book/download/index.html>
Legalize Adulthood! <http://blogs.xmission.com/legalize/>
1:13 a.m.
Jim Leonard wrote:
Al Kossow wrote:
Good for you!
Yes, but the md5sum will only detect corruption, not correct it.
I've always used DLT for backups in recent years; far as I'm aware they have a
reasonable level of error detection / correction information built in at the
lowest level, I don't see them dying out for a long time, and I prefer the
fact that the media's self-contained (unlike CD / DVD which runs the risk of
attracting dirt / fingerprints due to handling).
I tend to keep two copies of any data on separate tapes (uncompressed tar
format) plus a third copy on a hard disk (which only gets plugged into a
system for the duration of a backup run).
Test restores are always done, using the cksum util on each restored file for
comparison [1] against the original data just to verify tape integrity.
[1] Yes, tar has some verification options built in, but at least for the GNU
version the documentation seems to be a bit lacking; I'd trust cksum's output
more for detecting any problem files as it's not entirely clear what exactly
tar does for the verify step.
cheers
Jules
Is everything you're 'backing up'
checksummed?
Yes. I regularly ( == when I remember!) backup my email
by throwing it all into a zip archive and then that gets
written (along with other stuff) to CD, along with an
MD5SUM file.
2:28 a.m.
Jules Richardson wrote:
I've always used DLT for backups in recent years;
far as I'm aware they
have a reasonable level of error detection / correction information
built in at the lowest level, I don't see them dying out for a long
time, and I prefer the fact that the media's self-contained (unlike CD /
DVD which runs the risk of attracting dirt / fingerprints due to handling).
It is rare that they fail completely (they usually slowly develop errors
over time in heavy use until you have to replace them). But let me tell
you, if you get the one rare one that does fail, it fails SPECTACULARLY.
As in, it takes the drive with it!
[1] Yes, tar has some verification options built in,
but at least for
Its verification is practically a non-feature. Always generate parity
or make redundant copies.
--
Jim Leonard (trixter at oldskool.org) http://www.oldskool.org/
Help our electronic games project: http://www.mobygames.com/
Or check out some trippy MindCandy at http://www.mindcandydvd.com/
A child borne of the home computer wars: http://trixter.wordpress.com/
4:38 p.m.
Jim Leonard wrote:
Yep, that was my thinking too. I really need to find the time to write a Linux
util to suck tar data off DLT and checksum file-by-file to test backups;
currently I restore the whole archive and then use find/cksum/diff to check
against source data, but it's annoying needing that extra disk capacity just
for the sake of backup testing.
cheers
Jules
Jules Richardson wrote:
:-)
I've never had one fail on me, but thankfully it'll always be easy to pick up
either an identical one or one that's compatible with the media used.
I've always used DLT for backups in recent
years; far as I'm aware
they have a reasonable level of error detection / correction
information built in at the lowest level, I don't see them dying out
for a long time, and I prefer the fact that the media's self-contained
(unlike CD / DVD which runs the risk of attracting dirt / fingerprints
due to handling).
It is rare that they fail completely (they usually slowly develop errors
over time in heavy use until you have to replace them). But let me tell
you, if you get the one rare one that does fail, it fails SPECTACULARLY.
As in, it takes the drive with it! [1] Yes, tar
has some verification options built in, but at least for
Its verification is practically a non-feature. Always generate parity
or make redundant copies. 
5:14 p.m.
I really need to find the time to write a Linux util
to suck tar data
off DLT and checksum file-by-file to test backups; currently I
restore the whole archive and then use find/cksum/diff to check
against source data, but it's annoying needing that extra disk
capacity just for the sake of backup testing.
I wrote my own version of tar, which among other things has a mode in
which it reads a tape as if to restore, but, instead of restoring, it
compares what it finds on the tape with what it finds in the
filesystem.
It may well turn out to be not what you want, but it might be worth at
least a look-in. If it seems close but not quite there, I'll be happy
to correspond about it (though in your case it might be a bit, um,
interesting, since it appears you use both Yahoo for your email and are
in the UK).
ftp.rodents.montreal.qc.ca:/mouse/misc/tar.c for them as wants.
There's also tar.man alongside it, though that may be a bit outdated.
/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML mouse at rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
9 Feb
9 Feb
2:12 p.m.
der Mouse wrote:
Well I *think* that's what the diff/compare option is supposed to do (or maybe
W/verify, can't remember which), but unfortunately short of looking in the
source code itself I couldn't find any documentation on exactly how the
comparison is done (and quite possibly it differs between tar implementations
anyway). In theory it *should* be a byte-for-byte comparison, but unless the
docs say so we don't know that for sure...
I really need
to find the time to write a Linux util to suck tar data
off DLT and checksum file-by-file to test backups; currently I
restore the whole archive and then use find/cksum/diff to check
against source data, but it's annoying needing that extra disk
capacity just for the sake of backup testing.
I wrote my own version of tar, which among other things has a mode in
which it reads a tape as if to restore, but, instead of restoring, it
compares what it finds on the tape with what it finds in the
filesystem. It may well turn out to be not what you want, but it
might be worth at
least a look-in. If it seems close but not quite there, I'll be happy
to correspond about it
Do you happen to know if GNU tar adds extensions that aren't in your version
(or "original tar", whatever that may mean)? I suppose I'd like to stick to
GNU or GNU-compatible tools where possible...
(though in your case it might be a bit, um,
interesting, since it appears you use both Yahoo for your email and are
in the UK).
Well I'm in the US at the moment ;) If anyone has any suggestions for good
ISP-independent email services then let me know; the reason I've used yahoo
for years is so that I don't get tied to a single ISP for receiving email, but
there are doubtless better offerings around...
cheers
Jules
5:47 p.m.
Do you happen to know if GNU tar adds extensions that
aren't in your version
(or "original tar", whatever that may mean)? I suppose I'd like to stick to
GNU or GNU-compatible tools where possible...
Unless a given systems tar *is* GNU tar or based on it, you can pretty much
guarentee GNU tar has features it doesn't. Which isn't to say that GNU tar
is perfect, I know there are some bugs in it that we've run into, but can't
remember what they are at the moment.
Zane
11 Feb
11 Feb
6:42 a.m.
It may well
turn out to be not what you want, but it might be worth
at least a look-in. If it seems close but not quite there, I'll be
happy to correspond about it
Do you happen to know if GNU tar adds extensions that
aren't in your
version (or "original tar", whatever that may mean)?
I suppose I'd like to
stick to GNU or GNU-compatible tools where possible...
My tar tries to understand some of the GNU tar private header formats
(specifically, the long-name and sparse-file ones) but, since I was
unable to find any documentation on them, I probably don't have them
all and quite possibly don't have the ones I do handle quite right.
But if you have a thing for the GNU name, or GPL licensing, my tar
probably is not for you.
If anyone has any suggestions for good ISP-independent
email services
then let me know; the reason I've used yahoo for years is so that I
don't get tied to a single ISP for receiving email, but there are
doubtless better offerings around...
Probably. My approach to that is to run my own mailserver (static IPs
and no administrative prohibitions on mailservers are non-optional to
me when selecting a connectivity provider). Not to say that that
approach is suitable for everyone, of course - I've seen it said often
enough that there are good mail-handling services out there, but as
I've never wanted one, I've never looked into any in detail.
/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML mouse at rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
4:57 p.m.
der Mouse wrote:
Yes. Neither my tar nor GNU tar is a subset of the other. (For
example, my tar supports files larger than 8GB, which I was somewhat
surprised to find GNU tar didn't
Recent versions at least do; I wrote some which were around 14GB a few weeks
ago and they read back and verified against original data OK (no errors,
missing files etc.)
It's one of the things I dislike about tar in general though; I'm never sure
when I'm going to hit either file size or file path length limits. Other than
that though it's generally better than the alternatives in that the format's
documented well and it's simple enough to write an extraction util in the
unlikely event that one didn't exist for platform foobar.
It may well turn out to be not what you want, but it
might be worth
at least a look-in. If it seems close but not quite there, I'll be
happy to correspond about it
Do you happen to know if GNU tar adds extensions that
aren't in your
version - and GNU tar has some kind of attempt
at incremental backups which I don't understand; mine makes no attempt
at that.
Yeah, I've never touched any of that either...
But if you have a thing for the GNU name, or GPL
licensing, my tar
probably is not for you.
Oh, not at all... but with traditional UNIX vendors dying out, GNU tar seems
likely to be the one that will go the distance and still be around in decades
to come - so any tar offering from other sources really needs to be "GNU
compatible" (at least in terms of archive writing; reading is probably
slightly less of an issue)
[ email ]
Probably. My approach to that is to run my own mailserver (static IPs
and no administrative prohibitions on mailservers are non-optional to
me when selecting a connectivity provider). Not to say that that
approach is suitable for everyone, of course - I've seen it said often
enough that there are good mail-handling services out there, but as
I've never wanted one, I've never looked into any in detail.
Fair enough. I'm currently not a fan of running my own server because I'm
spending half my time thousands of miles away from where it would be - and
despite remote access being very good for various OSes, there are times when
it's useful being able to physically sit in front of the hardware.
One day that won't be the case and it probably will make sense to run my own,
but in the meantime I might consider moving to something else that's
location-independent (hotmail and gmail won't be any better than yahoo, though)
cheers
Jules
/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML mouse at rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
--
there's a carp in the tub
there's a carp in the tub
so nobody's taken a bath
12 Feb
12 Feb
4:34 a.m.
(For example,
my tar supports files larger than 8GB, which I was
somewhat surprised to find GNU tar didn't [...])
Recent versions at least do;
I wrote some which were around 14GB a
few weeks ago and they read back and verified against original data
OK (no errors, missing files etc.) It's one of the things I dislike about tar in
general though; I'm
never sure when I'm going to hit either file size or file path length
limits.
That's why my tar uses private header formats to transparently handle
path length limits (on which it imposes no restrictions except
available memory, I think) and file size limits (on which it does
impose a 2^88-byte limit; I could lift even that limit with a little
work if I were convinced it were worth it, though I'd be unable to test
it as I have no OSes which support file sizes over 2^64). I think GNU
tar does something similar for pathname length limits.
Other than that though it's generally better than
the alternatives in
that the format's documented well
It is? Where? I've been looking for a long time for documentation on
tar format, and so far all the pointers I've found have ended up at
pay-for-play "standards". I'd love to get hold of a real spec for the
tar header block format.
/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML mouse at rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
11 Feb
11 Feb
7:19 p.m.
If anyone has
any suggestions for good ISP-independent email services
then let me know; the reason I've used yahoo for years is so that I
don't get tied to a single ISP for receiving email, but there are
doubtless better offerings around...
Probably. My approach to that is to run my own mailserver (static IPs
and no administrative prohibitions on mailservers are non-optional to
me when selecting a connectivity provider). Not to say that that
approach is suitable for everyone, of course - I've seen it said often
enough that there are good mail-handling services out there, but as
I've never wanted one, I've never looked into any in detail.
6 Feb
6 Feb
5:42 p.m.
On 6 Feb 2007 at 10:38, Jules Richardson wrote:
Back when we were in the data forensics racket, the business of "Does
a separate verify pass do any good with DLT/DAT/8mm media,
considering that the recording process is read-write-verify-erase-if-
error-and-retry (this is what you pay the bucks for on these
drives)"?
We rewound and checked the VTOC at the beginning of the tape because
doing so would often turn up SCSI data transfer problems, but found
it unnecessary to provide an automatic separate verify pass. We did
provide a manual verification option, however.
Cheers,
Chuck
Its
verification is practically a non-feature. Always generate parity
or make redundant copies.
Yep, that was my thinking too. I really need to find the time to write a Linux
util to suck tar data off DLT and checksum file-by-file to test backups;
currently I restore the whole archive and then use find/cksum/diff to check
against source data, but it's annoying needing that extra disk capacity just
for the sake of backup testing.
6:42 p.m.
Jim Leonard wrote:
Jules Richardson wrote:
I've had one CD-R shatter and the drive was indeed unusable after that.
I tried to repair it, but after an hour of clearing bits out and putting
the thing back together, testing, dismantling, rinse, lather, repeat
I decided it wasn't worth the hassle. I still have it in case I
magically
run out of CD drives in some far distant future ...
Antonio
I've always used DLT for backups in recent
years; far as I'm aware
they have a reasonable level of error detection / correction
information
built in at the lowest level, I don't see them dying out for a long
time, and I prefer the fact that the media's self-contained (unlike
CD / DVD which runs the risk of attracting dirt / fingerprints due
to handling).
It is rare that they fail completely (they usually slowly develop
errors over time in heavy use until you have to replace them). But
let me tell you, if you get the one rare one that does fail, it fails
SPECTACULARLY. As in, it takes the drive with it!
6529
days inactive
6536
days old
20 comments
10 participants
participants (10)
-
aek＠bitsavers.org -
arcarlini＠iee.org -
bfranchuk＠jetnet.ab.ca -
brain＠jbrain.com -
cclist＠sydex.com -
healyzh＠aracnet.com -
julesrichardsonuk＠yahoo.co.uk -
legalize＠xmission.com -
mouse＠Rodents.Montreal.QC.CA -
trixter＠oldskool.org