dfitoimd: decoding Intel M2FM floppy flux images (was Re: Intel 432 floppy flux images for decoding)
6 Aug
2016
6 Aug
'16
4:12 p.m.
Because the cwtool support for Intel M2FM floppy disk flux images
isn't publicly available, I've written a crude Python 3 program for
converting such images (in DFI format) to ImageDisk images, and put it
on github:
https://github.com/brouhaha/dfitoimd
My ADPLL data separator has parameters to control the proportions of
frequency adjustment and instantaneous phase adjustment that are done
at each flux transition. I've experimented with the parameters quite a
bit, but it's still not good enough to reliably recover all sectors
from the disk images that started this thread. The
default parameters
are currently 0.5% frequency adjustment and 10% instantaneous
phase
adjustment. I was actually fairly surprised that I didn't get better
results with more frequency adjustment and less instantaneous phase
adjustment. It's entirely possible that my algorithm isn't very good.
The code can handle normal IBM 3740 single-density FM format as well,
though there are other published programs that already do that.
There's a start at code for IBM System/34 double-density MFM format
also, but it hasn't been tested at all as I don't happen to have any
suitable DFI images.
The dfitoimd program is quite slow; on a typical Intel M2FM image with
two revolutions per track and 25 MHz sampling, it takes 64 seconds on
a 4.0 GHz AMD CPU. I'm pretty sure that rewriting it in C/C++/C# or
the like would make it much faster, but I specifically chose Python as
I find it easier to experiment with the algorithms.
7 Aug
7 Aug
7:53 p.m.
Folks, I do have a version of software that supports READING Intel m2fm on a cat weasel,
and will post a link in the coming days - just got back from a road trip.
Sent from my iPad
On Aug 6, 2016, at 18:12, Eric Smith <spacewar at
gmail.com> wrote:
Because the cwtool support for Intel M2FM floppy disk flux images
isn't publicly available, I've written a crude Python 3 program for
converting such images (in DFI format) to ImageDisk images, and put it
on github:
https://github.com/brouhaha/dfitoimd
My ADPLL data separator has parameters to control the proportions of
frequency adjustment and instantaneous phase adjustment that are done
at each flux transition. I've experimented with the parameters quite a
bit, but it's still not good enough to reliably recover all sectors
from the disk images that started this thread. The default parameters
are currently 0.5% frequency adjustment and 10% instantaneous phase
adjustment. I was actually fairly surprised that I didn't get better
results with more frequency adjustment and less instantaneous phase
adjustment. It's entirely possible that my algorithm isn't very good.
The code can handle normal IBM 3740 single-density FM format as well,
though there are other published programs that already do that.
There's a start at code for IBM System/34 double-density MFM format
also, but it hasn't been tested at all as I don't happen to have any
suitable DFI images.
The dfitoimd program is quite slow; on a typical Intel M2FM image with
two revolutions per track and 25 MHz sampling, it takes 64 seconds on
a 4.0 GHz AMD CPU. I'm pretty sure that rewriting it in C/C++/C# or
the like would make it much faster, but I specifically chose Python as
I find it easier to experiment with the algorithms.
8:39 p.m.
On Sun, Aug 7, 2016 at 8:53 PM, Jay Jaeger <cube1 at charter.net> wrote:
Folks, I do have a version of software that supports
READING Intel m2fm on a cat weasel, and will post a link in the coming days - just got
back from a road trip.
That's great! I look forward to seeing it, even though I don't have a
Catweasel. :-)
10 Aug
10 Aug
1:44 p.m.
Well, it turns out what I have is a version of Karsten Schiebler's "cw"
program (cwtool) that is NOT on the web site (http://unusedino.de/cw/),
that I got directly from him. I used that code to image my Intel M2FM
floppies.
What I have is labeled as version 0.14, but it is actually dated from
October 2 2009, so I would presume that the code I have is subsumed into
the later versions, so I suggest just picking up his latest version from
his web site.
JRJ
On 8/7/2016 10:39 PM, Eric Smith wrote:
On Sun, Aug 7, 2016 at 8:53 PM, Jay Jaeger <cube1
at charter.net> wrote:
Folks, I do have a version of software that
supports READING Intel m2fm on a cat weasel, and will post a link in the coming days -
just got back from a road trip.
That's great! I look forward to seeing it, even though I don't have a
Catweasel. :-)
7 Aug
7 Aug
9:52 p.m.
On Sat, Aug 6, 2016 at 5:12 PM, Eric Smith <spacewar at gmail.com> wrote:
My ADPLL data separator has parameters to control the
proportions of
frequency adjustment and instantaneous phase adjustment that are done
at each flux transition. I've experimented with the parameters quite a
bit, but it's still not good enough to reliably recover all sectors
from the disk images that started this thread. The default parameters
are currently 0.5% frequency adjustment and 10% instantaneous phase
adjustment. I was actually fairly surprised that I didn't get better
results with more frequency adjustment and less instantaneous phase
adjustment. It's entirely possible that my algorithm isn't very good.
Of the four Intel M2FM disk images I've tried, which are single-sided
77 track 52 sector 128 byte, for a total of 4004 sectors, I get 3990,
3981, 4004, and 3976 sectors read apparently correctly (without CRC
error). That's more than 99.3%, which isn't good enough.
Can anyone share a sample DiscFerret image (.dfi) of a normal
double-density (MFM) disk? I don't think my data separator
performance should be much different for MFM vs. Intel M2FM, but it
would be nice to give it a try.
I've built my own floppy imager hardware, but I haven't yet written
firmware for it:
https://www.flickr.com/photos/22368471 at N04/26782411256/
It's a daughterboard for an Artix 7 FPGA board that Emanuel Stiebler
made. The only thing wrong with his FPGA board is that it isn't
readily available. The daughterboard just has SN7438 drivers and
SN74LS14 Schmitt triggers, terminators, and series resistors for 5V
tolerance.
There's also a SMC FDC92C39B data separator on the daughterboard, but
it's not actually in the floppy data path. All of its I/O signals go
to the FPGA. It's there to serve as a reference for comparison.
8 Aug
8 Aug
1:33 p.m.
On 8/7/16 9:52 PM, Eric Smith wrote:
Of the four Intel M2FM disk images I've tried,
which are single-sided
77 track 52 sector 128 byte, for a total of 4004 sectors, I get 3990,
3981, 4004, and 3976 sectors read apparently correctly (without CRC
error). That's more than 99.3%, which isn't good enough.
is there any correlation between creation dates of the files and the failure rate?
i'm wondering if the heads started to clog farther into the imaging session.
he didn't say how often he cleaned the heads.
1:36 p.m.
On 8/8/16 1:33 PM, Al Kossow wrote:
On 8/7/16 9:52 PM, Eric Smith wrote:
that is a big problem with acquisition not concurrent with analysis.
Of the four Intel M2FM disk images I've
tried, which are single-sided
77 track 52 sector 128 byte, for a total of 4004 sectors, I get 3990,
3981, 4004, and 3976 sectors read apparently correctly (without CRC
error). That's more than 99.3%, which isn't good enough.
is there any correlation between creation dates of the files and the failure rate?
i'm wondering if the heads started to clog farther into the imaging session.
he didn't say how often he cleaned the heads.
7:31 p.m.
On Mon, Aug 8, 2016 at 2:33 PM, Al Kossow <aek at bitsavers.org> wrote:
is there any correlation between creation dates of the
files and the failure rate?
i'm wondering if the heads started to clog farther into the imaging session.
I'm seeing more errors on low numbered tracks, so I really think I
just need a better data separator algorithm.
References on PLL design state that the error signal has to be low
pass filtered for loop stability, and I've seen plenty of designs for
analog PLLs, but I can't find any example ADPLL design that has low
pass filtering. The references describe using an IIR filter. I've
never had good luck with IIR filters, so I tried a 2nd order (3 tap)
FIR filter, and a range of cutoff frequencies, and didn't see any
improvement. Maybe 2nd order doesn't have sharp enough rolloff to
help. I really need to learn more about control theory and the
Z-transform.
10:06 p.m.
On 08/08/2016 07:31 PM, Eric Smith wrote:
I'm seeing more errors on low numbered tracks, so
I really think I
just need a better data separator algorithm.
That's surprising--low-numbered tracks should be the easiest to read.
At the expense of sounding like a complete idiot, have you checked the
alignment of the drive? It's not uncommon to get a batch of disks
written with alignment slightly off (blame whomever you want). I keep
an "unaligned" drive around to tweak when I run into those cases.
--Chuck
11:48 p.m.
On Mon, Aug 8, 2016 at 11:06 PM, Chuck Guzis <cclist at sydex.com> wrote:
On 08/08/2016 07:31 PM, Eric Smith wrote:
I agree. That's why I suspect data separator algorithm problems over
hardware problems or bad images.
I'm seeing more errors on low numbered
tracks, so I really think I
just need a better data separator algorithm.
That's surprising--low-numbered
tracks should be the easiest to read. At the expense of sounding like a complete idiot, have
you checked the
alignment of the drive?
I can't. I only have the images.
9 Aug
9 Aug
12:33 a.m.
On 08/08/2016 11:48 PM, Eric Smith wrote:
That's unfortunate. I run into mis-aligned disks not infrequently. I
don't know how to correct that one with software. When making cw-type
images, you can usually just inspect the histogram for a track and tell
by the groupings whether you're on track or not.
--Chuck
On Mon, Aug 8, 2016 at 11:06 PM, Chuck Guzis
<cclist at sydex.com>
wrote:
At the expense
of sounding like a complete idiot, have you checked
the alignment of the drive?
I can't. I only have the images. 
8:02 a.m.
I was having problems recovering inner tracks on some 5.25 disk. I added some extra screws
that allowed me to move the head both radially and up and down.
I was able to recover more but still didn't get it all.
The up and down allowed a little rotation as well as tracking.
On the inner tracks a small up or down looks like the head is
rotated slightly.
I just looked at the analog amplitude. I was just using a controller
card instead of sampling.
Dwight
________________________________
From: cctalk <cctalk-bounces at classiccmp.org> on behalf of Chuck Guzis <cclist
at sydex.com>
Sent: Tuesday, August 9, 2016 12:33:24 AM
To: General Discussion: On-Topic and Off-Topic Posts
Subject: Re: dfitoimd: decoding Intel M2FM floppy flux images (was Re: Intel 432 floppy
flux images for decoding)
On 08/08/2016 11:48 PM, Eric Smith wrote:
That's unfortunate. I run into mis-aligned disks not infrequently. I
don't know how to correct that one with software. When making cw-type
images, you can usually just inspect the histogram for a track and tell
by the groupings whether you're on track or not.
--Chuck
On Mon, Aug 8, 2016 at 11:06 PM, Chuck Guzis
<cclist at sydex.com>
wrote:
At the expense
of sounding like a complete idiot, have you checked
the alignment of the drive?
I can't. I only have the images.
8 Aug
8 Aug
11:17 a.m.
On 08/06/2016 04:12 PM, Eric Smith wrote:
Because the cwtool support for Intel M2FM floppy disk
flux images
isn't publicly available, I've written a crude Python 3 program for
converting such images (in DFI format) to ImageDisk images, and put
it on github:
https://github.com/brouhaha/dfitoimd
Interesting, Eric!
I did my cw-MMFM converter in 32-bit C years ago, but then I use my own
format for the cw files (i.e. I don't use cwtools), so it's probably not
a lot of use to folks--but it's fast. Metadata is important to me and I
pack my cw image files with as much as possible.
I record two revolutions worth of data per track to get *everything* and
also to give an implied second chance at data that the first rev may
have missed.
To date, on Intel and AES MMFM (they're different) disks, I'm batting 100%.
While I can appreciate wanting to do decoding real-time (i.e. reading
and decoding), the cw approach does give you the option of going back
over the data without the need to stress the disk as many times as one
would care. One can, for example, insert missing transition data where
dropouts may have occurred on the disk.
And accumulating sample data is pretty much child's play nowadays. I
have a rig with an micro SD card, an STM32 "maple mini" board and a
couple of TTL driver ICs that works a treat. It probably set me back
less than $10--and it has USB connectivity, though it can't be powered
by the USB port--a 150 ohm terminated floppy drive just pulls too much
current to be safe.
--Chuck
P.S. On my bucket list, I'm curious if anyone's ever worked out the
internals of the WD9914 data separator chip.
11:56 a.m.
On Mon, Aug 8, 2016 at 12:17 PM, Chuck Guzis <cclist at sydex.com> wrote:
I did my cw-MMFM converter in 32-bit C years ago, but
then I use my own
format for the cw files (i.e. I don't use cwtools), so it's probably not
a lot of use to folks--but it's fast. Metadata is important to me and I
pack my cw image files with as much as possible.
If you're willing to share the code, I'd be interested in taking a look.
I record two revolutions worth of data per track to
get *everything* and
also to give an implied second chance at data that the first rev may
have missed.
The dfi images I'm working with have two revolutions. One limitation
(bug?) in my current data separator code is that the "VCO" of my ADPLL
always starts at the nominal frequency when the code starts processing
a track, rather than the frequency it settled to on the previous
track. I should change that on the assumption that the rotation rate
is generally going to be the same for all tracks. Anyhow, right now
the code processes both rotations (or as many as are represented in
the dfi image), but with only a single pass, so sometimes it doesn't
successfully decode a sector in the first revolution, but gets it in
the second. I should change the code so that if there's only one
revolution in the file, it cycles through it twice (or a configurable
number), to give the ADPLL a chance to settle.
To date, on Intel and AES MMFM (they're different)
disks, I'm batting 100%.
Is there any documentation about AES MMFM format online?
So far I haven't encountered any two M2FM/MMFM formats that are the same.
P.S. On my bucket list, I'm curious if
anyone's ever worked out the
internals of the WD9914 data separator chip.
Typo in the part number? The WD9914 is WD's second source (or clone?)
of the TMS9914 GPIB chip.
I was able to find some info on the internals of the SMC FD9216 data
separator. WD second-sourced that as the WD9216.
12:19 p.m.
On 08/08/2016 11:56 AM, Eric Smith wrote:
--
--Chuck
-------------------------------------------------------------
"The first thing we do, let's kill all the spammers."
If you're willing to share the code, I'd be
interested in taking a
look.
I'll see what I can do to come up with something. It's going to be
Win32-based.
Is there any documentation about AES MMFM format
online?
Not that I know of--heck, I can't find any documentation on the original
machine as it is. I had to "brute force" that one; complicated a bit by
the bit order being backward (i.e. USART order, not floppy order). To
date, I still don't know what the machine model is--just have the floppy
written by some journalist back in the 70s.
Typo in the part number? The WD9914 is WD's
second source (or
clone?) of the TMS9914 GPIB chip.
I was able to find some info on the internals of the SMC FD9216 data
separator. WD second-sourced that as the WD9216.
Yeah, that one--brain fart on my part--I was looking at the WD data comm
products book, which looks just the same as the storage management
products book and picked out what I thought was the data separator.
Regardless, I'd like to see what the 9216 details are. I suspect it's
nothing more than a counter-based data separator, but it's remarkable
for how well it works. Shouldn't be hard to incorporate into an FPGA or
even a CPLD.
--Chuck
12:51 p.m.
On Mon, Aug 8, 2016 at 1:19 PM, Chuck Guzis <cclist at sydex.com> wrote:
On 08/08/2016 11:56 AM, Eric Smith wrote:
The FDC9216 is actually a fair bit more sophisticated than the typical
counter-based data separator, which is probably why it works so well.
For details, see US patent 4,472,818:
https://www.google.com/patents/US4472818
I was able to find some info on the internals of
the SMC FD9216 data
separator. WD second-sourced that as the WD9216.
Yeah, that one [...] I'd like to see what the 9216 details are. I suspect it's
nothing more than a counter-based data separator, but it's remarkable
for how well it works. Shouldn't be hard to incorporate into an FPGA or
even a CPLD. From a photomicrograph of the FDC9216 I've
discerned that the PLA
contents in the production part don't exactly match the
patent.
The same (or extremely similar) separator is used in the FDC9229,
which adds write precompensation and head load timing circuitry. A
photomicrograph shows that the PLA contents are very slightly tweaked
from the FDC9216.
SMC later introduced the FDC92C16 and FDC92C36, an improved CMOS datas
eparator, pin-compatible with the FD92C16 but operating at twice the
internal frequency. The difference between the two is the polarity of
the SEPD output.
The FDC92C39 used the improved data separator and was pin-compatible
with the FDC9229.
I don't (yet?) have photomicrographs of the improved CMOS parts.
1:15 p.m.
On 08/08/2016 12:51 PM, Eric Smith wrote:
From a photomicrograph of the FDC9216 I've
discerned that the PLA
contents in the production part don't exactly match the patent.
"... includes a logic function array which implements a predetermined
algorithm".
Ah, yes, the magic in the mixture. Thanks for the link to the patent.
--Chuck
3007
days inactive
3011
days old
17 comments
5 participants
participants (5)
-
aek＠bitsavers.org -
cclist＠sydex.com -
cube1＠charter.net -
dkelvey＠hotmail.com -
spacewar＠gmail.com