Reading disks with Catweasel

I noticed that someone forwarded one of my early messages about the Catweasel software I was working on to the list. I've made a lot more progress since then, so in case anyone is interested, below are a few more messages giving the latest news. Basically, it's working well, even on 8" disks. My program (cw2dmk) reads disks into a file format that represents decoded FM or MFM data, with the location of ID address marks noted, but no other indication of missing clocks or other odd stuff. This might be useful beyond the two TRS-80 emulators that currently support it, but it isn't the universal format that folks on this list were trying to design a month or two ago. Designing a format like that is an interesting problem. After my experience reading disks with the Catweasel, I'm not sure that simply recording the raw sample data is a good archival format. After my program decodes the FM or MFM data and checks for things like CRC errors and missing/unexpected data address marks, there are a fair number of errors that go away if I do a few retries at resampling the track. (Most disks have none, but they happen often enough to be noticeable when I'm reading a lot of disks.) So an archiver that just blindly sampled each track once without trying to decode it might end up recording data that could not be decoded later. Also, writing a good decoder is a bit of a pain, and some (what appears to me) black magic to reconstruct the data after bit-shifting has occurred. It would be nice to do the decoding once, up front, and archive decoded data. On the other hand, really strangely copy-protected disks from systems with disk controllers that are partly implemented in software might not be decodable using a general-purpose algorithm. Maybe it would be best to *try* to decode the data, do some retries if it can't be decoded, and archive both the result of decoding and the raw samples just for maximum safety.... --Tim ========Message 1======== From: mann(a)pa.dec.com (Tim Mann) Date: Thu, 22 Jun 2000 19:47:28 -0700 (PDT) To: trs80(a)cs.ubc.ca Subject: Announcing cw2dmk: read any TRS-80 floppy with a Catweasel ----- Background: The Catweasel ISA floppy disk controller is an add-in card for the PC ISA bus. It uses specialized hardware, not a standard floppy disk controller chip, and it can be programmed to read and write just about any disk format. Catweasel cards were in short supply for a while, but as of July 2000 they should be available for purchase again. See http://www.jschoenfeld.com/ and http://apd2.tripod.com/catweasl.htm. I've written a free program for the Catweasel called cw2dmk that can read any disk written by a TRS-80, including single-density and copy-protected disks. The disk is saved as a DMK image file, suitable for use in either the xtrs TRS-80 emulator for Unix, or David Keil's TRS-80 emulator for MS-DOS. cw2dmk runs on Linux, MS-DOS, or Windows 95/98. You can download it from http://www.tim-mann.org/trs80resources.html. Source code is included. My program uses some code from Michael Krause's Linux drivers for the Catweasel (which work with Amiga and MS-DOS diskettes). Besides TRS-80 disks, cw2dmk should be able to read any disk written by a WD177x/179x or by a standard PC (NEC765-compatible) disk controller. Currently it's not too useful to have non-TRS-80 disks in DMK format, but perhaps additional tools that work with this format will become available in the future. For example, I'd like to write a program that uses a Catweasel to write a DMK image back to a real floppy disk. This would provide a nice way of archiving disks from many old machines and making physical copies when needed. Another likely future direction for cw2dmk is to be able to write JV1 and JV3 disk images, when the disk being read can be represented in one of those formats. Those are the formats that Jeff Vavasour's and Matthew Reed's TRS-80 emulators need. By the way, I don't get a cut of Catweasel sales. :-) ========Message 2======== From: mann(a)pa.dec.com (Tim Mann) Date: Thu, 22 Jun 2000 21:33:59 -0700 (PDT) Newsgroups: comp.sys.tandy, comp.os.cpm Subject: Re: Announcing cw2dmk: read any TRS-80 floppy with a Catweasel ----- Oops. In that announcement, I forgot to say "TRS-80 Model I, III, 4, or 4P". These are the models that are currently supported by the emulators that use DMK format. Disks written by other computers that used the TRS-80 brand name will probably be readable too, but the images aren't currently too useful since there is not yet any software that uses DMK format other than these emulators. I do plan to read some Model II/12 disks for archival purposes when I get my 8" drive plugged in, and I'm also going to take a shot at reading some 3.5" Model 100 disks that someone is mailing me, but I don't have any results to report on that yet. ========Message 3======== From: Gary Shanafelt <...> Date: Thu, 22 Jun 2000 22:02:21 -0500 To: trs80(a)cs.ubc.ca Subject: Re: Announcing cw2dmk: read any TRS-80 floppy with a Catweasel ----- Tim, In your last message about the Catweasel board, you mentioned David Keil's emulator using DMK image files while Jeff Vavasour's and Matt Reed's used JV1 and JV3 format files. What's the difference? All the emulators call their files .DSK files; I recently downloaded David's program, set it up with some of the .DSK files from the Vavasour/Reed emulators, and it worked with them without further ado. I thought all these .DSK files were a single standard TRS-80 emulator format, but I guess not? ========Message 4======== From: mann(a)pa.dec.com (Tim Mann) Date: Thu, 22 Jun 2000 21:27:01 -0700 (PDT) To: Gary Shanafelt <...> Cc: trs80(a)cs.ubc.ca Subject: Re: Announcing cw2dmk: read any TRS-80 floppy with a Catweasel ----- The emulators all give the .DSK extension to their files, but they are using several different formats, which I've given the names DMK, JV1, and JV3. Full details are on http://www.tim-mann.org/trs80/dskspec.html, but here is a summary. JV1 can only handle single density disks with directory on track 17. Jeff Vavasour's Model I emulator works only with JV1 images. Jeff's Model III/4 emulator doesn't support JV1 images at all, but all the other major emulators will work with a JV1 image if you give one to them. JV3 can handle normal single and double density disks, but most forms of copy-protected disk are not representable. Almost all of the images on Ira Goldklang's site (www.trs-80.com) are in JV3. All the major emulators except Jeff's Model I emulator will work with JV3 images. Jeff originated this format in his Model III/4 emulator. DMK can handle anything that a TRS-80 disk controller can write to the disk, including all the copy-protected formats. Even protected game disks where the program does a Read Track and looks for specific data in the gaps between sectors can be recorded in DMK format and will run in emulators that support it. Currently only David Keil's emulator and xtrs support this format. David originated the format. Ira now has a small collection of images in DMK format that can't be represented in JV3 format. ========Message 5======== From: mann(a)pa.dec.com (Tim Mann) Date: Wed, 28 Jun 2000 16:17:10 -0700 (PDT) To: ... Subject: Early 8" results with Catweasel on LS-DOS disks ----- Thought you might be interested to hear that I tried reading a few 8" disks with a Catweasel last night, using my custom program (cw2dmk). They were mostly TRS-80 Model II LS-DOS disks with an FM track 0 and MFM on the rest of the disk. The FM was easy to read; the MFM required tweaking the thresholds in my program because the bit-shifting effect (that write precompensation is meant to adjust for) varies so much from the outer to inner tracks, and makes a sudden jump backward at track 44 where write precomp was turned on when the disks were written. I found a setting that worked, but it's touchy; a few retries were needed on tracks near 43 and 76. I might try to devise an algorithm that sets the thresholds automatically for each track by doing a histogram and looking for the exact location of the peaks and valleys, or I might try doing something that emulates a phase locked loop in software instead of using fixed thresholds.... The main disks I tried reading were copies of Model II/12 LS-DOS that Lamar Owen and Fred Dolan sent. The disk Fred sent had CRC errors on about half the tracks (usually one error per track), no matter what I did. I'm going to poke a bit more to see if it's a problem with my software or if the disk is really bad. The disks Lamar sent read OK once I got the thresholds right. ========Message 6======== From: mann(a)pa.dec.com (Tim Mann) Date: Mon, 3 Jul 2000 20:30:21 -0700 (PDT) To: ... Subject: Re: Early 8" results with Catweasel on LS-DOS disks ----- An update for those who may be interested: I found a heuristic to make up for insufficient write-precompensation (or for flux transitions that were close together moving farther apart over time). With this added to cw2dmk, I was able to read most of the 8" disks that I have without getting errors, including the one from Fred Dolan that previously showed CRC errors on about half the tracks. Here's the idea. When decoding MFM, the interval between each two successive flux transitions should be either short (2 cycles), medium (3), or long (4). Because of a physical effect, short transitions that come next to medium or long ones tend to spread out into their space. So after I classify a transition, I compute how long it ideally should have been and save the difference between the real and ideal lengths. I then add k times the difference to the next transition before classifying it, where k is a tuning parameter. With k=0, we have the old algorithm. A value of k around 0.5 to 0.8 made the difficult disks readable. Usually the exact value wasn't too critical, but on the really bad disks I had to experiment a bit to find the best setting. I imagine this technique (or something better) is well known in the disk drive industry, but it was a kick to me when I thought of it and it worked so well. I might go back to some 5.25" disks I have that showed CRC errors when read on a normal PC controller, and see if this trick will let me read some of them. Tim Mann tim.mann(a)compaq.com http://www.tim-mann.org Compaq Computer Corporation, Systems Research Center, Palo Alto, CA