Digital archiving tools - test-drb@ccmp.vtda.org

12 Mar 2011

...
  Speaking of DVD and archives too large to fit on one
media, does anyone 
 have a good solution for pointing at a large file system (such as 
 bitsavers top level directory) and specifying a size of the target media 
 and let it sort out the archiving problem?  I would like not to use an 
 archiver, but rather have the individual files copied to the target such 
 that they are available individually as files, as well as an index 
 somewhere specifying which item of the backup the file landed on. 
For the past decade my solution has to been write a perl script :-).

Previous splits to CD often worked hard on filling every nook and cranny
of every CD but I don't try so hard anymore. (This was the original use
of the "foundbymb" sort you see remnants of below.) To be super duper
efficient required knowing things like rockridge/joliet extension names and
the extra space they took up etc.

This one splits up bitsavers into slightly-smaller-than-DVD-sized chunks (using
soft links), and makes (to standard output) a index that can be copied to each DVD:

#!/usr/bin/perl
use strict;
use File::Path qw(make_path);
use File::Find;
my $total;
my $prefix = '/home/shoppa/www.bitsavers.org';
find(\&wanted, $prefix);
my @foundbymb;
my @foundbyalpha;
my %files;
sub wanted { 
  my $name = $File::Find::name;
  my $size = -s $name;
  next if -d $name;
  $files{$name} = { "name" => $name, "size" => $size,
"dir"=> $File::Find::dir }
;
  $total += $size;
}
my $cdn =1; my $maxbytes = 4400000000; my $thisbytes = 0;
my %madedir;
for (sort keys %files) {
  my $size = $files{$_}{"size"};
  if ($thisbytes + $size > $maxbytes) {
    print "DVD $cdn finished with $thisbytes bytes\n";
    $thisbytes = 0;
    $cdn++;
  }
  my $cdm = sprintf("%02d",$cdn);
  $thisbytes += $size;
  my $indir = $files{$_}{"dir"};
  my $outdir = $indir;
  $outdir =~ s/$prefix/$cdm/;
  if (!exists $madedir{$outdir}) {
    make_path($outdir); # or die "didn't make $outdir\n";
    $madedir{$outdir}++;
  }
  my $inf = $_;
  my $outf = $inf;
  $outf =~ s/$prefix/$cdm/;
  link($inf,$outf) or die "didn't ln $inf $outf";
  print "$outf\n";
}
print "DVD $cdn finished with $thisbytes bytes\n";
print "Total size is $total\n";