[cctalk] Re: ZFS, was [... GreaseWeazle ..]

On Thu, Feb 2, 2023 at 5:54 AM emanuel stiebler via cctalk < cctalk(a)classiccmp.org&gt; wrote: I wanted to note a story about AWS: I used to be heavy into astronomy imaging (I still am, just less so at the moment since building a new house and other projects is taking much of my attention for now). There is a site Astrobin.com specifically for astronomers to share their images, with a sort of database to store equipment metrics used for each image. It's an interesting service, since with decades of image data, you can now compare your image processing results with similar equipment and gauge how efficiently you're using a tracking mount, focuser, or imaging camera (or judge if quality issues are equipment related or post-processing related). Another interesting aspect of this database is users provide "versions" of their images -- that's been an annoying thing about YouTube for me: you can't revise a video, like you can't have editions. If you've learned new information or want to correct or clarify things, you can try to do cc comments, but basically have to start over with a new video link. That's a pain for content creators who have to "start over" back to 0 views on a video with a new edition, and also annoying for us users who now have to sort through dozens of variants of how-to's or guides, etc. Anyway... What I'm getting to is, the Astrobin database crashed a few years back. Was wiped out and (I think) they were using AWS. The site administration completely confessed it was his mistake -- I forget the details, his letter may be still up somewhere at the site. But basically he clicked something wrong on AWS, and ended up deleting files by accident. Being Amazon, you'd think it was just an easy recovery - no big deal. But nope, the nature of what the admin had done ended up being not easy to undo. Astrobin is a pay service and many people were bitterly angry for losing years of archives. Yes, most users should have their own backup -- but it was the database also of equipment, and decades of revision info (to see the progress of terrestial imaging capability over the years). A good 80+% of my own were wiped. I wasn't bitter - I appreciated that the admin immediately took blame, didn't try to hide it, and "to err is human." But it was a mess for that community. Well - to Amazon's credit, they didn't give up either. Months and months later, gradually some files did end up getting recovered. Again, I don't know the specifics of what Amazon did -- I just know about a year later, I logged in and noticed a great many of my image records had been recovered, and the admin had a note to expect more to get recovered since they were still actively working on the problem. I'm just relaying the story as a real-world example of a recent "data disaster recovery" - sorry I'm not more specific on the details, but astrobin.com still exists and the admin is approachable and may be willing to share lessons learned for those interested.