On 3/6/2013 11:01 AM, Dave Wade wrote: The program below works on solaris with the mtio interface. It reads each record into an individual file. the name on the record is FFFF-RRRR where FFFF is the file number and RRRR is the record number. A tape or file mark is a 0 length file. You have to tweak it manually to allow for three tape mark EOT format, but other than that it preserves the record format. I have a program which also goes from this format, which is pretty much universal to the tap format which Al K uses for his archiving. There are tape utilities to go from tap format to AWS and back written by Jay Maynard. I am not sure it Jay's utilities are the same as the mentioned SCSI to aws or not. both are viable archival methods. I find this methods allows very simple tools to get into tape data and examine it, but it is bulky from the unix system point of view, eating up a lot of directory and inode real estate. tapread.c *************************** #include <stdio.h> #include <sys/types.h> #include <sys/uio.h> #include <unistd.h> #include <fcntl.h> #include <sys/ioctl.h> #include <sys/mtio.h> /* * copyright jim stephens 2006 please check with me before passing this on tapread at jwsss.com tread reads files from tape device onto fixed storage. Each tape record is written as a file on the disk, in the current directory. creation of separate directories for each tape is left as an exercise for the user to manage. Each file is named for the file number (tape file, determined by counting EOF marks from tape) and record number. Each field is padded with leading zeros, and becomes the file name by separateing them with a "-" character. The counters start at 1 (file number) and record number 1. the first record on the tape is therfore 0001-0001 a 0 length record is a file mark, so each file will be on the tape will record at least one disk file. Tapes are assumed to end when two consecutive file marks are read, with no intervening data record. Most tape drive / formatter combinations require a read record length of 18 bytes, or the data is ignored. This program has to be modified to handle DEC format tapes which end with three file marks, with file separation being by 2 consecutive file marks, and internal file marks (within tape files) being allowable. Current tape library relies on Sun libraries to run. The system calls to set the maximum tape record size, and retrieve the size of each record read. There are equivalent ones in the MT libs for linux. * */ struct mtop mt_command; struct mtget mt_status; #define BUFLEN 32768 int in_fd; int out_fd; int fmcntr; int rccntr; void mtrew() { mt_command.mt_op = MTREW; mt_command.mt_count = 0; ioctl(in_fd, MTIOCTOP, &mt_command); ioctl(in_fd, MTIOCGET, (char *)&mt_status); /*sts = ioctl(in_fd, */ } void mttsize() { mt_command.mt_op = MTGRSZ; ioctl(in_fd, MTIOCTOP, &mt_command); ioctl(in_fd, MTIOCGET, (char *)&mt_status); printf("record size is %x\n", mt_command.mt_count); } void mtstatus() { printf("mt_type %x\n",mt_status.mt_type); printf("mt_dsreg %x\n",mt_status.mt_dsreg); printf("mt_erreg %x\n",mt_status.mt_erreg); printf("mt_resid %x\n",mt_status.mt_resid); printf("mt_fileno %x\n",mt_status.mt_resid); printf("mt_blkno %x\n",mt_status.mt_blkno); printf("mt_flags %x\n",mt_status.mt_flags); printf("mt_bf %x\n",mt_status.mt_bf); } void main(int argc, char *argv[]) { char fname[500] = "/dev/rmt/0b"; char outname[] = " - "; int i,j; char buf[32768]; int lim; int bsize; int fmflg = 0; int doing_file = 1; int fmcnt = 0; if( (in_fd = open(fname,O_RDONLY)) == -1) { printf("error opening file\n"); exit(1); } fmcntr = 1; rccntr = 1; mtrew(); mttsize(); mtstatus(); i=1; for(;;) { while(doing_file) { bsize = read(in_fd, buf, BUFLEN); sprintf(fname,"%4.4i",fmcntr); strncpy(outname,fname,4); sprintf(fname,"%4.4i", rccntr); strncpy(&outname[5],fname,4); outname[9] = '\0'; outname[4] = '-'; if( (out_fd = open(outname, O_RDWR | O_CREAT)) == -1) { printf("error opening output file\n"); } write(out_fd, buf,bsize); close(out_fd); printf("%4.4x " ,bsize); if(bsize == 0) { doing_file = 0; if(fmflg) { printf("eot %i files\n", fmcnt); exit(1); } } else { fmflg = 0; } if(doing_file == 0) break; printf("%2.2i ", i); lim = bsize; if(bsize > 40) lim=40; for(j=0;j<lim;j++) { printf("%2.2X", buf[j] & 0x000000ff); } printf("\n"); //mtstatus(); i=i+1; rccntr=rccntr+1; } i=1; rccntr = 1; fmcntr = fmcntr+1; fmflg = 1; // to detect eom printf("EOF %i\n", fmcnt); fmcnt = fmcnt + 1; doing_file = 1; } } tape2tap.c ************************************************** ///////////////////////////// // utility to read .tap files and parse into records // // usage: tapread <filename> // output: file number, and record number + record sizes to stdout // // limitations: 100000 byte record size maximum (can be changed) ///////////////////////////// #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> #include <time.h> #include <sys/types.h> #include <sys/stat.h> #include <sys/ioctl.h> #include <sys/mman.h> #include <sys/wait.h> ///////////////////////////// #define TAPERECSIZE 100000 ///////////////////////////// FILE *scriptptr; FILE *tapfile; int recsize; char taperec[TAPERECSIZE]; char tapepath[2000]; int filenum; int recnum; char recfilename[2000]; ///////////////////////////// void usage(char *pgmname) { printf("usage: %s input tape file\n", pgmname); } ///////////////////////////// // // read tape record // return 0 if record // return 1 if tape mark // // tape format is: <record size><data><record size>.... <end of file> // assumed to start at state BOT, beginning of a record ///////////////////////////// int gettaperec(int *parmsize, char *record) { int tapemark = 0; int fileeof; int taprecl; sprintf( recfilename, "%s/%4.4i-%4.4i", tapepath, filenum, recnum); printf("read %s\n", recfilename ); if( (scriptptr = fopen( recfilename, "rb")) == NULL) { printf(" error at record %s\n", recfilename ); // if tape output file open, close it if( tapfile != NULL) { fclose(tapfile); } exit(1); } taprecl = fread( taperec , 1, TAPERECSIZE, scriptptr); fclose( scriptptr ); if( taprecl == 0) { tapemark=1; printf("file end reached\n"); } *parmsize = taprecl; recsize = taprecl; return(tapemark); } ///////////////////////////// // write tape mark, which is 0x00000000 ///////////////////////////// void putfilemark(void) { int tapemark = 0; // write tape mark, which is 0x00000000 fwrite(&tapemark, 4, 1, tapfile); return; } void putrec(void) { // put out record size fwrite(&recsize, 4, 1, tapfile); // output record fwrite(&taperec, recsize, 1, tapfile); // output following record size fwrite(&recsize, 4, 1, tapfile); } ///////////////////////////// int main(int argc, char *argv[]) { int exitflag = 1; int lastrecwastapemark = 0; int lastrecmark; int filecount = 1; int reccount = 0; if(argc < 2) { usage( argv[0] ); exit(1); } printf("reading %s tape file\n", argv[1]); strcpy( tapepath, argv[1]); filenum = 1; recnum = 1; if( (tapfile = fopen("tapfile", "wb" )) == NULL) { printf("problem opening out file tapfile\n"); exit(1); } while( exitflag ) { //exitflag = 0; lastrecmark = gettaperec( &recsize, taperec ); if( lastrecwastapemark && lastrecmark ) { putfilemark(); printf("double filemark\n"); exitflag = 0; } if(lastrecmark) { lastrecwastapemark = 1; filecount+= 1; filenum+=1; reccount = 0; recnum=1; putfilemark(); printf("file mark\n"); } else { lastrecwastapemark = 0; recnum+=1; reccount+=1; putrec(); } if( !lastrecmark) printf("record %4.4i:%4.4i size is %i\n", filecount, reccount, recsize ); } }

On Wed, Mar 6, 2013 at 3:54 PM, Torfinn Ingolfsen <tingox at gmail.com> wrote: That would not work because the record information would be lost, and the requirement was to make a replica of the tape (in addition 'dd' will only read until the first EOF mark and then stop.) I have been using a piece of software I wrote years ago which does essentially what the previously mentioned 'tcopy' does, but in addition it can write to files. It's a very simple tool I'm sure many others have also written, what it does boils down to the following: while (not eof) { record_size = read (tape deviice. record) write (file, record_size, 4) // writes a 4-byte record with the record size if (record_size > 0) { write (file, record, record_size) // write record } } The disk file will then consist of 'records' prepended with a 4-byte value with the record size. End-of-file marks on the tape (between files) will be indicated with a record size of zero. At the end of the tape, write 2 ore more zero-size records (ANSI tapes often used 2 or 3 EOF marks to indicate e-o-t or e-o-v). To generate a new tape from the file just write a tool to understand this simple format and write back record-by-record with size as indicated in the file. -Tor

On Mar 6, 2013, at 10:49 AM, Glen Slick <glen.slick at gmail.com> wrote: dd doesn't change blocking, it's just a byte-by-byte dump, unless you specify otherwise. Tar, on the other hand, looks for files.

On 03/06/2013 08:01 AM, Jonathan Katz wrote: Neither is really germane to the problem. Tar is wonderful if you're dealing with files. This is an old problem. Early tapes did not have named files--just sections of blocks separated by filemarks. Records wither corresponded to physical block sizes or some other convention, such as "a record is a group of 512-word blocks until a block shorter than 512 words is encountered". Hierarchical tape structures were possible, with each record having a record-level indicator (COBOL-style). You could similarly, have a tape where a single block could take up a sizeable part of a reel. Having spent a couple of years maintaining tape software, I can say that there were a bewildering number of tape conventions and schemes. Software to deal with differences between systems and vendors with regards to tape was an important part of the mainframe culture. A tape imaging scheme that doesn't preserve the differences is rather short-sighted. dd does not preserve blocking. --Chuck

TCOPY has been mentioned. I usually suggest Eric Smith's tapeutils: http://www.brouhaha.com/~eric/software/tapeutils/ which avoids needing two tape drives. Many types support blockings other than a single fixed size: Exabyte 8mm, DLT and LTO, to pick a few common ones. Maybe. Some implementations of QIC formats used a non-standard block size, e.g. some of the Prime unix systems. As another odd case, the Prime 50-series wrote the standard QIC fixed blocks, but the controller managed variable sized blocks, writing a block length and flags into a header in each tape block, then did reassembly on read. I believe I've heard AS400 people kvetching about non-standard block sizes on some tape format or another. Short version, DO NOT USE dd TO IMAGE TAPES. Just don't. It's not hard to use something that preserves the block structure, and then you don't have to guess. Also, the archivists won't shoot you. :) De

What about COPYTAPE? http://manned.org/copytape/8e792e25 I've used it on FreeBSD to dup DG AViiON DG/UX tapes. This was a bootable tape image; the AViiON couldn't tell the difference between the original and the copy.... -- Jeff ____________________________________________________________ FREE 3D MARINE AQUARIUM SCREENSAVER - Watch dolphins, sharks & orcas on your desktop! Check it out at http://www.inbox.com/marineaquarium

On Wed, Mar 06, 2013 at 09:25:35PM -0500, Dennis Boone wrote: I think it'd be very difficult to find a modernish (SCSI) 9-track drive that will do reverse parity. So it'd take more than friendly drivers... BTW my earlier msg seems to have gotten stuck somewhere: Does anyone have a VT102 they'd be interested in selling/trading? Day-trip distance from Monson, MA, USA would be ideal. Thanks! John Wilson D Bit

On 03/06/2013 06:25 PM, Dennis Boone wrote: In the old days, a lot of very strange stuff was written using tape. Nowadays with tape drives made in the 80s, I'm not sure that you can get away with a lot of stuff. Will an auto-load drive, for example, handle a tape with multiple BOT markers on it? In the old days, if you wanted to build a tape containing important files and utilities that you could just take along with you to a customer's site, you could write a file or two, move the tape manually down a bit, stick on a BOT, write some more, and so on. What's the maximum block length that drives and OSes can handle? Just kilobytes or megabytes? I can't really comment just from a man page. --Chuck

Actually, dd uses 512-byte blocks by default,and always has. At least since V6, anyway...

On Mar 8, 2013, at 1:29 AM, Chuck Guzis <cclist at sydex.com> wrote: True, but going in the other direction, blocking factor is preserved in variable sized writes to tape. For example, if I write a series of 1k blocks to a tape, and try to read them back as 2k blocks, each read will return only 1024 bytes, at least on streaming-type tapes. I'm not sure what would happen on real magtape. So,writing a tape with dd will, by default, create 512-byte blocks.

On 03/08/2013 05:40 AM, Roe Peterson wrote: I think we have a problem with nomenclature. To me, "blocking factor" is the number of records placed in a block. So, if you're writing 2048 byte physical blocks, with a record length of 512 bytes, the blocking factor is 4. What I'm talking about is preserving physical block length. IIRC, there is a minimum block length (2 bytes or thereabouts?) that most drives treat as "noise" and discard. AFAIK, there is no maximum defined physical block length--the limitation there seems to be mostly in operating systems or controller/formatter use. But it is possible to write blocks on a tape that are many times the size of physical memory on some systems. I think telemetry systems used to do just that. On CDC SCOPE, there were three basic tape drivers--standard (512 words per physical block, short last block; no particular blocking factor), "stranger" (any odd block size up to 512 words; treated as one logical record per physical block) and "long" tapes--no limit on physical record size. The "L" tape, in theory, could treat the entire reel of tape as a single record. So long as the user program could empty the buffer on read and fill it on write, the tape would continue to spin. Can one do the same with Unix? I've never tried. The ultra-short physical blocks could be used for great sound effects as the drives used a vacuum capstan system which was regulated by a large voice-coil valve. --Chuck

On 3/6/13, Torfinn Ingolfsen <tingox at gmail.com> wrote: AFAIK, dd won't preserve the block structure or marks from tapes, and thus is unsuitable for making faithful copies.