Data Scrubbing in archives and paranoia

Cut and pasted from Wikipedia, not that it has any Deep history of the subject, but to start the conversation: * Data scrubbing?is an error correction technique which * uses a background task that periodically inspects memory * for errors, and then corrects the error using?ECC memory * or another copy of the data. It reduces the likelihood that * single correctable errors will accumulate; thus, reducing * the risk of uncorrectable errors. Some on-topic hardware relation: I first found out about data scrubbing back when it was a way to deal with flaky DRAM (what Wikipedia calls "Memory Scrubbing".) Mid-70's stuff. Later found out that the Xerox Alto had done it in that era too. Not sure where it first originated, I would be surprised if this hadn't been done back to the first storage media. In the context of large data/disk archives it is reassuring to have this done not just at the media layer, but above the transport (e.g. SATA or SCSI or USB bus) layer. e.g. I store my files as bzip2 format, and every so often not just make sure the drive thinks it can read the sectors, but that also it passes a bunzip2 -v checksum validation. This habit probably dates from flaky SCSI bus/network protocols in the past that had this odd way of dropping or mangling random bytes or sectors without any error indication. When I'm truly paranoid not only do I count on bzip2 checksums but also resort to storing MD5's or SHA-1's of the uncompressed contents. Every so often I get ultra-paranoid and worry that the file System is not mangling files but simply losing them. It's been A long time since I've witnessed this, but storing the MD5's/SHA1's In a text listing gives me some reassurance. Of course the Filesystem might not just lose the file but also hide that it lost The file by removing the file from the listing so I also print out The listing and attach it to the drive/tape/CD etc... this is getting To 2001/HAL eras of paranoia to talk about this using a Computer (never mind locking ourselves into a space pod to discuss Verbally and then finding out the computer can read lips!) It is very reassuring to me to do these validations not Just automatically but manually too. And I even purposefully Mangle some backup files to make sure my tools would kick Out a failure. Probably just paranoia on my part. Wondering if others might even be more extreme :-). Tim.