5 Dec
2008
5 Dec
'08
5:12 a.m.
I've just been thinking about the problem of imaging/copying MFM hard discs...
I've run the numbers, and this seems right to me, but I'd appreciate a sanity
check.
Let's say you have an ST-412 drive you want to read. You've also got a
Catweasel or something like it, that has somehow had an extra bunch of I/O
lines added to it (to control the head select lines, etc. on the HDD).
An ST-412 rotates at 3600 RPM. That's (3600/60)=60 revolutions per second, or
1/60 = 16.66(recurring) milliseconds per revolution.
The drive's data rate is 5 megabits per second, but could be lower (or indeed
higher). But the spec says 5Mbps, and for the sake of argument I'm going to
stick with that...
(5Mbps/1000) = 5Kbits per millisecond.
5Kbits * 16.667 = 83.35 kilobits per track, absolute maximum.
A Catweasel records the data from a disc by measuring the time between flux
transitions.
If we assume every data bit on the disc is a "1" (MFM coding "01")
then you'd
need to store at most 90,000 timing samples per disc track.
So hypothetically, if your "Catweasel or something like it" had 128Ksamples
worth of buffer RAM, and enough I/O lines to drive the HDD, you could read an
MFM hard drive track-by-track and copy it onto another drive of the same type?
Do my calculations look right?
With 304 cylinders and 4 heads, that works out to 304 * 4 * 83.35 = 101353.6
Kbits, which does seem awfully low to me...
Thanks,
--
Phil.
classiccmp at philpem.me.uk
http://www.philpem.me.uk/
5 Dec
5 Dec
9:20 a.m.
Philip Pemberton wrote:
I've just been thinking about the problem of
imaging/copying MFM hard
discs... I've run the numbers, and this seems right to me, but I'd
appreciate a sanity check.
I'm on the wrong side of coffee for this, but...
Let's say you have an ST-412 drive you want to
read. You've also got a
Catweasel or something like it, that has somehow had an extra bunch of
I/O lines added to it (to control the head select lines, etc. on the HDD).
Well, extra lines isn't a big deal I suppose - heck, you could even do that
bit 'manually' and read platters one at a time, flipping a switch between each
pass.
An ST-412 rotates at 3600 RPM. That's (3600/60)=60
revolutions per
second, or 1/60 = 16.66(recurring) milliseconds per revolution.
OK (I think I ran some numbers on this here on the list a couple of years ago,
but it's probably quicker to just reply now than dig the message out :-)
The drive's data rate is 5 megabits per second,
but could be lower (or
indeed higher). But the spec says 5Mbps, and for the sake of argument
I'm going to stick with that...
(5Mbps/1000) = 5Kbits per millisecond.
5Kbits * 16.667 = 83.35 kilobits per track, absolute maximum.
Well, run some figures as a sanity-check... say 256*8 bits per sector, 32
sectors/track on a formatted ST506 - 32 x 256 x 8 = 64Kb of actual data per track.
83.35 seems sensible as some kind of theoretical maximum, given storage of
sector headers and the like.
A Catweasel records the data from a disc by measuring
the time between
flux transitions.
I'm not sure what sort of granularity is needed, though - presumably the CW
uses counters geared toward expected floppy rates, so most likely can't
represent timing gaps accurately enough for hard disk speeds?
So hypothetically, if your "Catweasel or
something like it" had
128Ksamples worth of buffer RAM, and enough I/O lines to drive the HDD,
you could read an MFM hard drive track-by-track and copy it onto another
drive of the same type?
I'm not sure about copying - but I think you could at least extract the data
via software, which is still a vital step forward.
Do my calculations look right?
With 304 cylinders and 4 heads, that works out to 304 * 4 * 83.35 =
101353.6 Kbits, which does seem awfully low to me...
101353.6 Kb is approx 98Mb, or approx 12MB - so that seems not unreasonable.
However, remember that's *sample count*, not total storage.
So, what's an upper boundary on bits per sample? I suppose you could have an
entire track with just one sample on it right at the end, so your upper
boundary is the 16.67ms figure.
More typically though samples are going to occur *far* more frequently - so to
cope with all possible situations you (in theory) need a very large sample
length (which in 99.9999% of cases is going to contain a lot of 0s in the
upper bits!)
Some possible approaches I can see:
1) Just have an absolutely colossal buffer with a large number of bits per sample
2) Use the first bit (or bits) of each sample as a flag to indicate the
resolution of the following sample data (essentially toss away lower bits for
lengthy samples)
3) Use the first bit (or bits) of each sample to indicate the bit-length of
the following sample data,
4) Have a short-length sample resolution geared toward 'ordinary' data and
flag any tracks causing sample timer overflows as being damaged.
Personally ISTR looking at the second and third options - the third is
probably the better of the two, but needs a bit more intelligence (and at the
time I was wondering if I could do this in pure logic without a CPU). I'm not
sure what approach the CW takes (but at floppy rates the amount of buffer
needed is a lot smaller, so maybe it just gets away with a fixed sample size
and large enough buffer).
The fourth option is just yuck - I'd really want an exact (as possible) copy
of what's on the drive, even if it's utter garbage.
For some reason when I looked at this I think I believed I could get away with
9 bits per sample including a flag bit dictating two different sample
resolutions (i.e. somewhere around 1Mb of buffer) - unfortunately all my notes
on this are stuck in storage right now though (plus ideally I'd go for a
microcontroller approach with a variable sample length I think)
Recreating the data onto another drive is a different matter, of course - but
I've always been more interested in salvaging existing data onto more modern
media for analysis.
cheers
Jules
6:12 p.m.
Jules Richardson wrote:
True. An FDI file containing the sample data isn't likely to be /that/ much
bigger though.
But it'll take a bloody age transferring that much data over USB2 Full Speed
(1.2Mbps peak).
Well, extra lines isn't a big deal I suppose -
heck, you could even do
that bit 'manually' and read platters one at a time, flipping a switch
between each pass.
True, though the hardware I'm actually thinking of using is my USB
hack-and-bodge floppy reader. FPGA, PICmicro, and a few buffers/level translators.
The discrete FDD I/O stage is gone though -- mainly because in an overvoltage
condition, it's highly likely the 74LS14 input buffer would get deep-fried...
Thus rendering the "slightly more rugged" output drivers utterly pointless.
I've kept to the original spec for the connector though -- a 40-pin IDC with
the first 34 pins forming a Shugart-compatible interface, and the last 6
defined as user I/O (each pin configurable as either input or output, all
open-collector).
Well, run some figures as a sanity-check... say 256*8
bits per sector,
32 sectors/track on a formatted ST506 - 32 x 256 x 8 = 64Kb of actual
data per track.
83.35 seems sensible as some kind of theoretical maximum, given storage
of sector headers and the like.
And MFM is basically one flux transition per (data) bit guaranteed, meaning
83.35kbps isn't actually that strange a figure.
I'm not sure what sort of granularity is needed,
though - presumably the
CW uses counters geared toward expected floppy rates, so most likely
can't represent timing gaps accurately enough for hard disk speeds?
The read clock on a CW is about 8MHz or so, if memory serves. Effectively
that's ~16 clocks per bit cell time (at 500kbps MFM that is). The hardware I'm
building up is going to sample at 20 or 40MHz (software selectable), which is
40 (or 80) clocks per bitcell on 500k MFM, or 4 (or 8) on 5Mbps MFM...
Adding another sampling rate to the hardware probably wouldn't be that hard --
add another input to the clock-mux and either add a PLL multiplier to the
logic (there are four on an Altera Cyclone II 2C20) or another TTL oscillator
to the PCB.
Do my
calculations look right?
With 304 cylinders and 4 heads, that works out to 304 * 4 * 83.35 =
101353.6 Kbits, which does seem awfully low to me...
101353.6 Kb is approx 98Mb, or approx 12MB - so that seems not
unreasonable.
However, remember that's *sample count*, not total storage. So, what's an upper boundary on bits per sample? I
suppose you could
have an entire track with just one sample on it right at the end, so
your upper boundary is the 16.67ms figure.
MFM is a (1,3)RLL code. That means you can have a minimum of 1 and a maximum
of 3 empty bit cells between flux transitions.
Best-case is a stream of 1s, which is NT - NT - NT...
(1 no-transition period between each transition)
Worst-case is a repeating 1-0 bit stream which is NT-NN - NT-NN..
(3 no-transition periods between each transition)
You can also have 2 no-transition periods separating a pair of transitions --
that's a 1-0-0 sequence: NT-NN-TN - NT-NN-TN...
More typically though samples are going to occur *far*
more frequently -
so to cope with all possible situations you (in theory) need a very
large sample length (which in 99.9999% of cases is going to contain a
lot of 0s in the upper bits!)
The RAM I'm using is 4Mbit -- i.e. 512Kbyte, but mapped into 16-bit words. So
you have 262144 16-bit words. One status bit (for the index pulse) leaves 15
bits for the counter. The counter is rigged to lock at 0x7FFF instead of
rolling back to 0 (which saves a bit).
15 bit counter = span of 0 to 32767 counts.
40MHz = 25ns per count
25ns * 32767 = 819175 ns = 819.175 us
Hmm, for floppy discs you'd have to slow it down a bit (possibly to 20MHz,
which would be 1638.35 us, or about 1.6 milliseconds), but for MFM (and RLL?)
hard drives this should be more than enough.
Some possible approaches I can see:
1) Just have an absolutely colossal buffer with a large number of bits
per sample
SRAM is relatively expensive though. SDRAM isn't, but then you have to deal
with refreshing and other "fun" things like that.
2) Use the first bit (or bits) of each sample as a
flag to indicate the
resolution of the following sample data (essentially toss away lower
bits for lengthy samples)
Hmm, use the top 2 bits to specify the timing resolution, and the rest as a
timing value. Drop between 0 and 3 bits depending on how far over 14 bits your
count has gone...
That's going on my "try this" list.
Personally ISTR looking at the second and third
options - the third is
probably the better of the two, but needs a bit more intelligence (and
at the time I was wondering if I could do this in pure logic without a
CPU). I'm not sure what approach the CW takes (but at floppy rates the
amount of buffer needed is a lot smaller, so maybe it just gets away
with a fixed sample size and large enough buffer).
It's a 7-bit counter bolted onto a 128Ksample RAM buffer. The top bit stores
the state of the index pulse at the time the transition was stored, and the
lower 7 bits store the timing value.
For some reason when I looked at this I think I
believed I could get
away with 9 bits per sample including a flag bit dictating two different
sample resolutions (i.e. somewhere around 1Mb of buffer) - unfortunately
all my notes on this are stuck in storage right now though (plus ideally
I'd go for a microcontroller approach with a variable sample length I
think)
Ultimately it depends on what you're sampling. A 3.5" DSDD floppy, sampled at
7.08MHz ends up with few samples above ~64 counts (IIRC, my density graphs are
on the laptop, which is switched off). Based on that, 7 bits would probably be
fine up to 14MHz, 8 bits up to 28MHz.
Assuming, of course, that the disc is "perfectly" formatted. As in, the entire
track is MFM.
Recreating the data onto another drive is a different
matter, of course
- but I've always been more interested in salvaging existing data onto
more modern media for analysis.
Half the problem is that the timing is going to get progressively worse the
further down the line you go. Kinda like what happens when you dub an audio
tape. By the time you've gotten to the point of having a 3rd- or
4th-generation copy, the audio is almost completely masked by the background hiss.
Thanks,
--
Phil.
classiccmp at philpem.me.uk
http://www.philpem.me.uk/
7:03 p.m.
Philip Pemberton wrote:
...
The read clock on a CW is about 8MHz or so, if memory serves.
Effectively that's ~16 clocks per bit cell time (at 500kbps MFM that
is). The hardware I'm building up is going to sample at 20 or 40MHz
(software selectable), which is 40 (or 80) clocks per bitcell on 500k
MFM, or 4 (or 8) on 5Mbps MFM...
Quoting the catweasel manual:
To set the clockspeed with the CatOption register, reset the memory counter
to 0 by writing 0 to the CatAbort register. Writing 0 to CatOption will set
the clock rate to 14Mhz. Writing 128 (0x80) to CatOption will set the clock
rate to 28Mhz, and writing 192 (0xc0) to CatOption will set the clock speed
to 56Mhz (to be precise, 56.644Mhz).
But it'll take a bloody age transferring that much
data over USB2 Full
Speed (1.2Mbps peak).
what?
USB1 was 12 Mbps peak. USB2 is 480 Mbps peak.
(b=bit)
7:43 p.m.
Philip Pemberton wrote:
Yeah. I think your calculations were right, anyhow...
No; the raw data file size is manageable I think - it's just the buffer size
that needs to be reasonably big.
Yeah, but that's for complete data, is it not? I'm not sure what happens if
given a damaged drive - given that ST506/412 is dumb, presumably it just
blindly spits out what it finds on the disk surface? It'd be nice to be able
to capture whatever the drive throws at this device, in the hope of making
some sense out of (or 'beyond') damaged sections in software later. If the
track data has 'holes' then the device needs to be able to record the length
of those holes in order to portray what's on the disk accurately.
I'll look at those later - it's too noisy to concentrate round here right now :-)
Yeah, I came to that conclusion way back when, too.
Uh huh. There's not necessarily a guarantee on that for a damaged drive, I
fear (and few STxxx drives seem to exist without at least a few defects)
I can't remember if my 9 bits per sample (including flag bit) was just for
MFM, or if I figured it'd work for RLL, too (or ten bits if recording with
each sample whether there's an index pulse or not - I think I'd dreamed up a
way of doing that separately so as to not encroach on buffer size (but it's
more complex to do so).
Yeah. I think what needs to happen is that you sample a drive, essentially
turn it back into data bytes in software on a host machine, bung some sector
header voodoo around it, and *then* spit it back to the destination drive
using this device...
For now, I'm more worried about ailing STxxx drives and getting data off them
(and making sense of it) - writing back is more of a secondary issue (but is
probably an extra 10% logic on this device to support writes, I suspect).
Aside: I'm not sure if it's beneficial having a number of buffers and
reading/writing multiple head data in parallel? I worry about running STxxx
drives for longer than necessary :-) Probably not viable just from a RAM cost
point of view, though...
cheers
Jules
Jules Richardson wrote:
OK, so you can probably throw something in there to switch heads via software,
presumably?
Well, extra lines isn't a big deal I suppose
- heck, you could even do
that bit 'manually' and read platters one at a time, flipping a switch
between each pass.
True, though the hardware I'm actually thinking of using is my USB
hack-and-bodge floppy reader. FPGA, PICmicro, and a few buffers/level
translators. Well, run some
figures as a sanity-check... say 256*8 bits per sector,
32 sectors/track on a formatted ST506 - 32 x 256 x 8 = 64Kb of actual
data per track.
83.35 seems sensible as some kind of theoretical maximum, given
storage of sector headers and the like.
And MFM is basically one flux transition per (data) bit guaranteed,
meaning 83.35kbps isn't actually that strange a figure. 101353.6 Kb is
approx 98Mb, or approx 12MB - so that seems not
unreasonable.
However, remember that's *sample count*, not total storage.
True. An FDI file containing the sample data isn't likely to be /that/
much bigger though. But it'll take a bloody age transferring that much
data over USB2 Full
Speed (1.2Mbps peak).
I think I did once put in a request for a SCSI version ;)
So, what's
an upper boundary on bits per sample? I suppose you could
have an entire track with just one sample on it right at the end, so
your upper boundary is the 16.67ms figure.
MFM is a (1,3)RLL code. That means you can have a minimum of 1 and a
maximum of 3 empty bit cells between flux transitions. More typically
though samples are going to occur *far* more frequently
- so to cope with all possible situations you (in theory) need a very
large sample length (which in 99.9999% of cases is going to contain a
lot of 0s in the upper bits!)
[lots of numbers] 1) Just have
an absolutely colossal buffer with a large number of bits
per sample
SRAM is relatively expensive though. SDRAM isn't, but then you have to
deal with refreshing and other "fun" things like that. Hmm, use the top 2 bits to specify the timing
resolution, and the rest
as a timing value. Drop between 0 and 3 bits depending on how far over
14 bits your count has gone...
That's going on my "try this" list.
It should work, I think, subject to not throwing too much away.
For some
reason when I looked at this I think I believed I could get
away with 9 bits per sample including a flag bit dictating two
different sample resolutions (i.e. somewhere around 1Mb of buffer) -
unfortunately all my notes on this are stuck in storage right now
though (plus ideally I'd go for a microcontroller approach with a
variable sample length I think)
Ultimately it depends on what you're sampling. A 3.5" DSDD floppy,
sampled at 7.08MHz ends up with few samples above ~64 counts (IIRC, my
density graphs are on the laptop, which is switched off). Based on that,
7 bits would probably be fine up to 14MHz, 8 bits up to 28MHz.
Assuming, of course, that the disc is "perfectly" formatted. As in, the
entire track is MFM. Recreating the
data onto another drive is a different matter, of
course - but I've always been more interested in salvaging existing
data onto more modern media for analysis.
Half the problem is that the timing is going to get progressively worse
the further down the line you go. Kinda like what happens when you dub
an audio tape. By the time you've gotten to the point of having a 3rd-
or 4th-generation copy, the audio is almost completely masked by the
background hiss.
7 Dec
7 Dec
6:32 p.m.
Jules Richardson wrote:
That's what the User I/O port is for :)
Six pins on the "extended Shugart" port, and (maybe) another eight (plus 5V
and ground) on a separate 10-pin connector.
Well given that the average hard drive size these days seems to be about
500GB, and even a CD-R will store 700MB, 100MB per image isn't that
unreasonable. It might take a while to process the data, though.
Urgh, I got the figure wrong. Pretend that decimal point isn't there.. USB2
Low Speed is 1.5Mbps, Full Speed is 12Mbps, High Speed is 480Mbps. Of course
if I'd said MBps (i.e. megabytes per second), it would have been almost right
as well... :)
In any case, you're probably looking at a megabyte or so per second, which
isn't too bad.
If we assume we're reading a floppy disc, the most you'll get on a track is
(IIRC) about 120,000 flux transitions. If we say the drive is spinning at
360RPM, that's about 0.17 seconds to actually read the track, followed by
roughly the same again to transfer the data to the PC.
Being pessimistic, we're talking about half a second per track. That's 40
seconds per side, or 80 seconds for all 80 tracks, both sides.
If we think ST506 at 3600RPM and 5Mbps, it'll probably be a little quicker,
but probably not much better than a quarter-second per track.
I'd expect so -- an ST506 type drive is basically a motor, speed controller,
and head amplifier on a PCB, attached to a HDA. Basically the same type of
hardware you'd find on the average floppy drive PCB.
I suppose the alternative would be to use an off-the-shelf SDRAM controller,
and then buffer the writes inside the FPGA. If you're writing sequentially,
then you leave the SDRAM in "burst" mode and just keep writing. Do a partial
refresh every so often, and
True, though
the hardware I'm actually thinking of using is my USB
hack-and-bodge floppy reader. FPGA, PICmicro, and a few buffers/level
translators.
OK, so you can probably throw something in there to switch heads via
software, presumably? True. An FDI
file containing the sample data isn't likely to be /that/
much bigger though.
No; the raw data file size is manageable I think - it's just the buffer
size that needs to be reasonably big. But it'll
take a bloody age transferring that much data over USB2 Full
Speed (1.2Mbps peak).
I think I did once put in a request for a SCSI version ;) MFM is a
(1,3)RLL code. That means you can have a minimum of 1 and a
maximum of 3 empty bit cells between flux transitions.
Yeah, but that's for complete data, is it not? I'm not sure what happens
if given a damaged drive - given that ST506/412 is dumb, presumably it
just blindly spits out what it finds on the disk surface? It'd be nice
to be able to capture whatever the drive throws at this device, in the
hope of making some sense out of (or 'beyond') damaged sections in
software later. If the track data has 'holes' then the device needs to
be able to record the length of those holes in order to portray what's
on the disk accurately.
Good point.. every ST506 drive is going to have one or two (minimum, usually
about 10-20 IME) permanent media errors. The reader hardware is going to need
some way of dealing with these errors, even if that's just stepping the
resolution down and storing a less-accurate measurement of the dead-time.
SRAM is
relatively expensive though. SDRAM isn't, but then you have to
deal with refreshing and other "fun" things like that.
Yeah, I came to that conclusion way back when, too. Yeah. I think what needs to happen is that you sample
a drive,
essentially turn it back into data bytes in software on a host machine,
bung some sector header voodoo around it, and *then* spit it back to the
destination drive using this device...
Well, if you can decode the data then it's probably fair to assume you can
re-encode it as well...
That leaves you with three images -- the raw disc image (say, a .FDI file),
the extracted sector data (an .IMA file if we're talking PC floppies, .ADF for
Amiga, etc.) and the re-encoded data (another .FDI file).
If you split the software up, you can go from FDI -> ADF and play with the ADF
on an emulator, change something on the disc image, then go from ADF -> FDI
again and write it back. For bonus points, you could save the read speed from
stage 1, use it to adjust the data rate for the binary -> MFM conversion, and
you can modify copy-protected discs while leaving speed-based protection
schemes alone (RNC Copylock, Speedlok, etc.)
For now, I'm more worried about ailing STxxx
drives and getting data off
them (and making sense of it) - writing back is more of a secondary
issue (but is probably an extra 10% logic on this device to support
writes, I suspect).
Already done that, it's a finite state machine with index-pulse detection
hardware. Makes dealing with hard-sectored floppies a little easier.
Aside: I'm not sure if it's beneficial having
a number of buffers and
reading/writing multiple head data in parallel? I worry about running
STxxx drives for longer than necessary :-) Probably not viable just from
a RAM cost point of view, though...
Well, the idea was that you'd do two or three reads of each track at once,
meaning that you could use some form of "majority-voting" scheme to figure out
what the data was before it got scrambled.
This would be pretty good for discs where the adhesive that binds the
ferrite/whatever to the mylar disc had weakened -- you may only get one shot
at reading it, so make it count.
I'd still consider the ultimate to be some form of non-contact disc drive, but
that would involve custom-made FDD heads and other such "fun".
Thanks,
--
Phil.
classiccmp at philpem.me.uk
http://www.philpem.me.uk/
8 Dec
8 Dec
11:22 a.m.
Philip Pemberton wrote:
Well given that the average hard drive size these days seems to be about
500GB, and even a CD-R will store 700MB, 100MB per image isn't that
unreasonable.
Well, I've got various hard disk byte-level 'raw' backups kicking around in
the 40-1024MB region; the increased size of a flux-transition image wouldn't
really make a significant amount of difference to the cost of storage.
I'd expect so -- an ST506 type drive is basically a motor, speed
controller,
and head amplifier on a PCB, attached to a HDA. Basically the same type of
hardware you'd find on the average floppy drive PCB.
Uh huh. So I do think that theoretically you could have a transition time the
length of the track. No likely, but possible. In the middle of that extreme
and 'normal data' there may well be transition times that are quite high, so
they do need to be recorded (so that you can hopefully recover sectors of a
damaged track beyond the bad spot)
For sure - or worse. I've still recovered useful data from drives that have
been 80% bad - sometimes it's worth it.
I suppose the alternative would be to use an off-the-shelf SDRAM
controller, and then buffer the writes inside the FPGA.
Part of me still thinks that the 'elegant' solution to this is perhaps a PCI
card which bridges a 'raw' STxxx interface to an off-the-shelf compact PC
motherboard. Use the PC's memory for buffer (and refresh, of course), boot a
cut-down OS from a CF card (and use the CF card for temporary image storage),
and potentially allow the user to make use of whatever interface to the
'outside world' that they wished (personally I'd likely use Ethernet, run a
FTP server & telnet control interface on the machine etc.)
However, a) I'm not sure if PCI is actually fast enough for what's needed
(i.e. the system could sample into memory from a drive quickly enough), and b)
that we have anyone 'on-tap' who knows how to build a PCI card anyway... :-)
That was a complete brain-fart on my part, and I was momentarily thinking that
STxxx gives you data from all heads at once in parallel, which would maybe
make reading into multiple buffers a good idea. It doesn't, of course, and you
select a specific head and get that head only's data as the output stream...
Jules Richardson wrote:
That's what the User I/O port is for :)
:-)
True,
though the hardware I'm actually thinking of using is my USB
hack-and-bodge floppy reader. FPGA, PICmicro, and a few buffers/level
translators.
OK, so you can probably throw something in there to switch heads via
software, presumably? True. An FDI file containing the sample data isn't
likely to be
/that/ much bigger though.
No; the raw data file size is manageable I think - it's just the
buffer size that needs to be reasonably big. It might take a while to process the data, though.
I really don't know - maybe some of the resident Catweasel experts can comment
there. I suppose to 'understand' an image we wouldn't be doing anything in
software that the drive controller doesn't normally do in hardware.
Urgh, I got the figure wrong. Pretend that decimal
point isn't there.. USB2
Low Speed is 1.5Mbps, Full Speed is 12Mbps, High Speed is 480Mbps. Of
course if I'd said MBps (i.e. megabytes per second), it would have been
almost right as well... :)
In any case, you're probably looking at a megabyte or so per second,
which isn't too bad.
To be honest, the quicker the better - but I wouldn't be upset if it took an
hour or two to grab a drive image. Some of the dumps I have were transferred
at 9600 baud serial...
If we assume we're reading a floppy disc, the most
you'll get on a track
is (IIRC) about 120,000 flux transitions. If we say the drive is
spinning at 360RPM, that's about 0.17 seconds to actually read the
track, followed by roughly the same again to transfer the data to the PC.
Being pessimistic, we're talking about half a second per track. That's
40 seconds per side, or 80 seconds for all 80 tracks, both sides.
If we think ST506 at 3600RPM and 5Mbps, it'll probably be a little
quicker, but probably not much better than a quarter-second per track.
Can you read from the buffer and transfer to the PC at the same time as you're
populating the buffer by reading from the drive? Or is what you have
essentially a single microcontroller? One of the things I was trying to
achieve when I was looking at doing this in pure TTL was a degree of
parallelism - because the device doesn't attempt to understand the drive's
data stream (that's done later on the host PC), it can't benefit from re-seek
or re-read attempts anyway, so there's no requirement to ever pause buffer
fill and try again.
MFM is a (1,3)RLL code. That means you can have a
minimum of 1 and a
maximum of 3 empty bit cells between flux transitions.
Yeah, but that's for complete data, is it not? I'm not sure what
happens if given a damaged drive - given that ST506/412 is dumb,
presumably it just blindly spits out what it finds on the disk surface? It'd be
nice
to be able to capture whatever the drive throws at this device, in the
hope of making some sense out of (or 'beyond') damaged sections in
software later. If the track data has 'holes' then the device needs to
be able to record the length of those holes in order to portray what's
on the disk accurately.
Good point.. every ST506 drive is going to have one or two (minimum,
usually about 10-20 IME) permanent media errors. The reader hardware is going to need
some way of dealing with these errors, even if that's just stepping the
resolution down and storing a less-accurate measurement of the dead-time.
Hmm, so you're proposing that the reader actually tries to understand the
storage mechanism? I was thinking that it was a dumb device, basically copying
the transitions that it sees to the host PC - and it was the software on the
PC which actually does the decoding (and potentially requests re-read attempts
at a later date).
SRAM is relatively expensive though. SDRAM isn't,
but then you have
to deal with refreshing and other "fun" things like that.
Yeah, I came to that conclusion way back when, too. Well, if you can decode the data then it's
probably fair to assume you
can re-encode it as well...
True.
Aside: I'm
not sure if it's beneficial having a number of buffers and
reading/writing multiple head data in parallel? I worry about running
STxxx drives for longer than necessary :-) Probably not viable just
from a RAM cost point of view, though...
Well, the idea was that you'd do two or three reads of each track at
once, meaning that you could use some form of "majority-voting" scheme
to figure out what the data was before it got scrambled. This would be pretty good for discs where the adhesive
that binds the
ferrite/whatever to the mylar disc had weakened -- you may only get one
shot at reading it, so make it count.
Yeah, that could work, subject to enough buffer space. (Incidentally you've
now got me worried about how much precomp affects the timing of all of this,
or that clock bits are supposed to be in the middle of a bit cell with
'normal' data at the start. I suppose it doesn't matter if the sampling
resolution is high enough)
I'd still consider the ultimate to be some form of
non-contact disc
drive, but that would involve custom-made FDD heads and other such "fun".
Well for hard disks they're all flying heads anyway...
My ultimate would be something which recorded the signal at a more analogue
level I think, for potential cleaning up in software later (rather than using
a digital interface which can't perform an intelligent analysis of the data
stream). I think it's do-able for a floppy drive, and probably without a lot
of modification to a donor drive itself.
(I've mused before on here about running the entire drive in a bath of
something which might reduce friction and the chances of damage due to
contamination - it'd be interesting purely from an experimentation point of
view, but I'm not sure if it'd ever be possible to prove that it actually helped)
cheers
Jules
3:55 p.m.
Jules Richardson wrote:
I suppose ultimately it's going to depend on the speed of the machine that's
analysing the data, and the efficiency of the algorithm...
No, I'm not proposing that at all -- my goal is to create a generic
reader/writer and do all the decoding work in software. "It slices, it dices,
it makes coffee, it feeds the cat!* (* Subject to software availability)" :)
Only while the disc platters are spinning, AIUI...
It might take
a while to process the data, though.
I really don't know - maybe some of the resident Catweasel experts can
comment there. I suppose to 'understand' an image we wouldn't be doing
anything in software that the drive controller doesn't normally do in
hardware. To be honest, the quicker the better - but I
wouldn't be upset if it
took an hour or two to grab a drive image. Some of the dumps I have were
transferred at 9600 baud serial...
3600RPM = 60 revolutions per second.
1 revolution = 16.667 milliseconds
306 data tracks * 4 heads = 1224 tracks total (ST412)
1224 tracks * 16.667ms = 20 seconds, excluding transfer time
If we assume T_transfer = 100ms/track (to be pessimistic)...
100ms * 1224 tracks = 122.4 seconds = ~2 minutes
Assuming no read retries, and a head switching time of zero (and an infinitely
fast hard drive to store the image on), that's a theoretical minimum of 142.4
seconds.
I wouldn't expect it to be that fast, but ~15 minutes is probably a reasonable
estimate.
Can you read from the buffer and transfer to the PC at
the same time as
you're populating the buffer by reading from the drive?
No, it's a single-ported RAM. Either the MCU can control it, or the FPGA can.
One option would be to add a second SRAM chip, which could be rigged as a
ping-pong buffer (while track N is being downloaded, track N+1 is being read).
Or is what you
have essentially a single microcontroller?
No, it's a PIC18F4550 "USB-Enabled 8-bit Microcontroller" bolted onto an
Altera Cyclone II 2C20 FPGA.
The FPGA handles everything on the disc interface side -- reading and writing
data, setting/querying control lines and so forth.
The PIC effectively acts as a USB-to-high-speed-parallel bridge for the
Cyclone. The intention is to run it at 48MHz (12MIPS) and run the parallel
interface at 12MHz (Fosc/4, the fastest possible speed).
One of the things I was
trying to achieve when I was looking at doing this in pure TTL was a
degree of parallelism - because the device doesn't attempt to understand
the drive's data stream (that's done later on the host PC), it can't
benefit from re-seek or re-read attempts anyway, so there's no
requirement to ever pause buffer fill and try again.
The FPGA logic I'm using doesn't make any attempt to decode the bitstream,
unless the MFM sync-word detector is enabled. In that case, it won't record
anything until a specified sync-word is detected in the bitstream. If the
bitstream isn't MFM it probably won't ever start (and after a few seconds the
PC will probably time out).
Uh huh. So I do think that theoretically you could
have a transition
time the length of the track. No likely, but possible. In the middle of
that extreme and 'normal data' there may well be transition times that
are quite high, so they do need to be recorded (so that you can
hopefully recover sectors of a damaged track beyond the bad spot)
Thing is, the AGC on the head amplifier will probably start boosting the gain
if it doesn't see any transitions in, say, 1/32 of a revolution. If that
happens, then you'll get a nice stream of garbage (more likely spurious
transitions inserted than transitions missed).
The only way I can think of to get around that would be to remove the head-amp
and bolt on an analogue amplifier, sample the whole track at (say) 20
megasamples per second, then drag the data into a PC and analyse it. I wonder
if you could use some tricks from modern hard discs -- PRML and the like -- to
recover data from blocks where the magnetic transitions are indistinct.
For sure - or worse. I've still recovered useful
data from drives that
have been 80% bad - sometimes it's worth it.
True. I had an 8.4GB Maxtor that was almost completely dead. Chucked it in the
freezer, and nearly all of it was unreadable... except my mailspool. The loss
of a few weeks of work (which was backed up) was pretty minor compared to the
loss of five years of e-mail :)
The reader
hardware is going to need
some way of dealing with these errors, even if that's just stepping the
resolution down and storing a less-accurate measurement of the dead-time.
Hmm, so you're proposing that the reader actually tries to understand
the storage mechanism? I was thinking that it was a dumb device,
basically copying the transitions that it sees to the host PC - and it
was the software on the PC which actually does the decoding (and
potentially requests re-read attempts at a later date). However, a) I'm not sure if PCI is actually fast
enough for what's
needed (i.e. the system could sample into memory from a drive quickly
enough), and b) that we have anyone 'on-tap' who knows how to build a
PCI card anyway... :-)
I was going to say "but a PCI device can't take over the bus", then I
realised
you've got DMA and bus-mastering to play with... So you could allocate a block
of RAM, and have the FDD controller write bytes (or 32-bit DWORDs) straight to
RAM.
That's probably worth looking into, but I know very little about PCI device
design...
I'd still
consider the ultimate to be some form of non-contact disc
drive, but that would involve custom-made FDD heads and other such "fun".
Well for hard disks they're all flying heads anyway... My ultimate would be something which recorded the
signal at a more
analogue level I think, for potential cleaning up in software later
(rather than using a digital interface which can't perform an
intelligent analysis of the data stream). I think it's do-able for a
floppy drive, and probably without a lot of modification to a donor
drive itself.
True. If you can live with read-only, it would be a head amplifier and an A/D
converter. The Commodore 1541 schematics (and the older CBM drive manuals..
1540?) might be a good reference -- IIRC some of those used discrete amplifier
circuitry instead of an all-in-one head amplifier/threshold detector IC.
(I've mused before on here about running the
entire drive in a bath of
something which might reduce friction and the chances of damage due to
contamination - it'd be interesting purely from an experimentation point
of view, but I'm not sure if it'd ever be possible to prove that it
actually helped)
Sounds like a nice idea, but I'd be somewhat worried about the effects of
whatever chemical you're using on the motor, and its effect on the head/disc
distance.
I'd be inclined to think it would do more harm than good...
--
Phil.
classiccmp at philpem.me.uk
http://www.philpem.me.uk/
4:39 p.m.
Philip Pemberton wrote:
Hmm, well you could potentially use several SRAM ICs and couple them to
different buses as/when needed, but at the end of the day perhaps it's just
not worth the complexity.
Hmm, true. I suppose anything's possible though, without knowing the internals
of every STxxx drive ever made :-)
OK.
Yep.
I think I've got some Tandon and Epson drive schematics kicking around and
they used separate stages. But tapping into the head signal on a floppy drive
is a darn sight easier than a hard disk...
Yeah, I got around to thinking about moving the motors elsewhere - but you're
right about the head gap. I suspect it just plain won't work, but
speculation's always interesting :-)
cheers
Jules
Jules Richardson wrote:
I suppose ultimately it's going to depend on the speed of the machine
that's analysing the data, and the efficiency of the algorithm...
Yep. But I'm not sure that it's really rocket science - assuming that you know
what encoding format the data's actually in, which is probably the more
difficult part. Coping with errors might be interesting in some cases.
It might
take a while to process the data, though.
I really don't know - maybe some of the resident Catweasel experts can
comment there. I suppose to 'understand' an image we wouldn't be doing
anything in software that the drive controller doesn't normally do in
hardware. Assuming no read retries, and a head switching time of
zero (and an
infinitely fast hard drive to store the image on), that's a theoretical
minimum of 142.4 seconds.
I wouldn't expect it to be that fast, but ~15 minutes is probably a
reasonable estimate.
Well that'd do, I guess. Heck, that's hardly enough time for the electronics
to get up to working temp :-) But yeah, in the order of 15 mins is probably
realistic - I think it normally used to take about that to 'clone' a drive.
Can you read
from the buffer and transfer to the PC at the same time
as you're populating the buffer by reading from the drive?
No, it's a single-ported RAM. Either the MCU can control it, or the FPGA
can.
One option would be to add a second SRAM chip, which could be rigged as
a ping-pong buffer (while track N is being downloaded, track N+1 is
being read). Uh huh. So I
do think that theoretically you could have a transition
time the length of the track. No likely, but possible. In the middle
of that extreme and 'normal data' there may well be transition times
that are quite high, so they do need to be recorded (so that you can
hopefully recover sectors of a damaged track beyond the bad spot)
Thing is, the AGC on the head amplifier will probably start boosting the
gain if it doesn't see any transitions in, say, 1/32 of a revolution. If
that happens, then you'll get a nice stream of garbage (more likely
spurious transitions inserted than transitions missed). The only way I can think of to get around that would
be to remove the
head-amp and bolt on an analogue amplifier
I'm not sure how easy it is to tap into that kind of thing on a lot of drives,
though. Possibly best left for real emergencies on a case-by-case basis...
Hmm, so
you're proposing that the reader actually tries to understand
the storage mechanism? I was thinking that it was a dumb device,
basically copying the transitions that it sees to the host PC - and it
was the software on the PC which actually does the decoding (and
potentially requests re-read attempts at a later date).
No, I'm not proposing that at all -- my goal is to create a generic
reader/writer and do all the decoding work in software. "It slices, it
dices, it makes coffee, it feeds the cat!* (* Subject to software
availability)" :)
We have four cats here, and they eat a phenomenal amount - I'm not sure that
your device could keep up. :-)
That's probably worth looking into, but I know
very little about PCI
device design...
Ditto. Back in the day I wanted to get away from the internal card approach -
but these days it's possible to get a tiny, cheap motherboard and should be
relatively easy to build a networkable "external device" that's a complete
compact PC in its own right.
Well for hard
disks they're all flying heads anyway...
Only while the disc platters are spinning, AIUI... My ultimate
would be something which recorded the signal at a more
analogue level I think, for potential cleaning up in software later
(rather than using a digital interface which can't perform an
intelligent analysis of the data stream). I think it's do-able for a
floppy drive, and probably without a lot of modification to a donor
drive itself.
True. If you can live with read-only, it would be a head amplifier and
an A/D converter. The Commodore 1541 schematics (and the older CBM drive
manuals.. 1540?) might be a good reference -- IIRC some of those used
discrete amplifier circuitry instead of an all-in-one head
amplifier/threshold detector IC. (I've
mused before on here about running the entire drive in a bath of
something which might reduce friction and the chances of damage due to
contamination - it'd be interesting purely from an experimentation
point of view, but I'm not sure if it'd ever be possible to prove that
it actually helped)
Sounds like a nice idea, but I'd be somewhat worried about the effects
of whatever chemical you're using on the motor, and its effect on the
head/disc distance.
6:03 p.m.
Jules Richardson wrote:
Mmm, there's just a bit of a difference between "what the math says" and
"what
Murphy allows".
I still harbour a hatred for ST506 drives. Although mainly directed at the
Kalok 20MB piles-o'-crap.
I actually got banned from the school computer lab many years ago for
allegedly destroying the hard drive in an Acorn A440. Said drive being a 20MB
Kalok thing that was shaky at the best of times. I was dragged out of class
for half a day to help the "IT expert" (actually Just Another Teacher [tm])
fix it. Most of that time was spent staring at !HForm's soak-test screen and
writing down all the errors, on IT Expert's orders. One every two sectors IIRC.
"Make sure you don't get ANY of the numbers wrong! You have to re-enter them
when it actually formats the drive!"
It's "find a clean-room" territory. The head preamps are usually mounted
near
the heads, often on the actuator arms themselves.
On floppy drives, you get a screwdriver and a soldering iron. Suppose you
could make a replacement logic board if you liked, but that's too much like
hard work IMO :)
Two chinchillas here (soon to be three). It's all fun and games until one of
them tries to get under the sink and ends up getting soaked...
Oh, food. A handful of pellets, a handful of fresh hay and a fresh bottle of
water every 24 hours. Not hard, just expensive :)
The problem with PCI is licensing -- you need a PCI Vendor ID, which you get
by joining PCI-SIG, signing the patent licenses and so forth. The damage? ?4k
per year last time I checked.
Or you move to China, open a factory and pick a number at random and pray it
doesn't clash with someone else's ID. Or pick one that isn't on the Linux PCI
Vendor ID list...
USB is open-licensed -- AIUI, it's a case of "do what you like as long as you
buy a Vendor ID and don't use the USB logo". The VIDs were $2000 last time I
checked, but a lot of IC manufacturers will sub-licence individual PIDs in
their VID block to their customers.
Or you ignore the USB-IF and pick a nice, random number that isn't on the
Linux USB VID list.
Yep. But I'm not sure that it's really rocket
science - assuming that
you know what encoding format the data's actually in, which is probably
the more difficult part. Coping with errors might be interesting in some
cases.
It isn't. The hardest part is converting from timing data back to an MFM
bitstream or binary data. You need to either handle changes in drive speed in
real time (i.e. track the read speed and continually adjust the 1t, 1.5t and
2t timing values) or do something similar to disk2fdi -- preprocess the data
and figure out a set of thresholds that work, assuming the disc is in good shape.
I wouldn't
expect it to be that fast, but ~15 minutes is probably a
reasonable estimate.
Well that'd do, I guess. Heck, that's hardly enough time for the
electronics to get up to working temp :-) But yeah, in the order of 15
mins is probably realistic - I think it normally used to take about that
to 'clone' a drive. Hmm, well you could potentially use several SRAM ICs
and couple them to
different buses as/when needed, but at the end of the day perhaps it's
just not worth the complexity.
The added board space is a bigger concern...
Thing is, the
AGC on the head amplifier will probably start boosting
the gain if it doesn't see any transitions in, say, 1/32 of a
revolution. If that happens, then you'll get a nice stream of garbage
(more likely spurious transitions inserted than transitions missed).
Hmm, true. I suppose anything's possible though, without knowing the
internals of every STxxx drive ever made :-) The only way I
can think of to get around that would be to remove the
head-amp and bolt on an analogue amplifier
I'm not sure how easy it is to tap into that kind of thing on a lot of
drives, though. Possibly best left for real emergencies on a
case-by-case basis... "It
slices, it dices, it makes coffee, it feeds the cat!* (* Subject
to software availability)" :)
We have four cats here, and they eat a phenomenal amount - I'm not sure
that your device could keep up. :-) That's
probably worth looking into, but I know very little about PCI
device design...
Ditto. Back in the day I wanted to get away from the internal card
approach - but these days it's possible to get a tiny, cheap motherboard
and should be relatively easy to build a networkable "external device"
that's a complete compact PC in its own right. I think I've got some Tandon and Epson drive
schematics kicking around
and they used separate stages. But tapping into the head signal on a
floppy drive is a darn sight easier than a hard disk...
Mainly because the manuals for floppy drives tend to include schematics... The
ST412/506 OEM manual doesn't.
Woooooo, black magic inside!
Yeah, I got around to thinking about moving the motors
elsewhere - but
you're right about the head gap. I suspect it just plain won't work, but
speculation's always interesting :-)
Many a good idea has been born of idle speculation and what-ifs.
--
Phil.
classiccmp at philpem.me.uk
http://www.philpem.me.uk/
6:18 p.m.
Philip Pemberton wrote:
The problem with PCI is licensing -- you need a PCI
Vendor ID, which you
get by joining PCI-SIG, signing the patent licenses and so forth. The
damage? ?4k per year last time I checked.
OK, I really should check my facts before I open my big trap.
PCI-SIG won't issue a Vendor ID unless you're a member. That costs $3000 per
year.
USB is open-licensed -- AIUI, it's a case of
"do what you like as long
as you buy a Vendor ID and don't use the USB logo". The VIDs were $2000
last time I checked, but a lot of IC manufacturers will sub-licence
individual PIDs in their VID block to their customers.
Might as well correct this too.. $2000 for either a Vendor ID on its own (the
$2k is an "administration fee"), or a VID and a two-year Trademark License. No
idea if the VID lapses when the trademark license expires, the website isn't
exactly clear on that.
But anyway, this sort of thing is OT for classiccmp...
--
Phil.
classiccmp at philpem.me.uk
http://www.philpem.me.uk/
6:45 p.m.
Philip Pemberton wrote:
Well, in a non-clean-room environment I've pulled the complete head stack out
of a drive and put it back in again with surprising success (said stack was
jammed solid with debris, so removing it to clean it all out was the only
way). Figured I had nothing to lose, and was surprised when the drive sprang
into life (and I got about 80-90% of the data back - and some of what I lost
was probably as a result of damage already sustained by the debris flying
around the innards of the drive)
The problem with PCI is licensing -- you need a PCI Vendor ID, which you
get by joining PCI-SIG, signing the patent licenses and so forth. The
damage? ?4k per year last time I checked.
Is that like Ethernet, though, where in theory you could just make up an ID if
in a closed environment, providing it didn't conflict with anything else - or
is there no 'hobbyist/prototype' ID set aside for this kind of thing?
(I suppose in a real Big Brother scenario, the people issuing the IDs would be
the same people as those who sold the ICs necessary to talk to the bus...)
No, but the service manual (with schematics) is on Bitsavers. I don't think
many of the other vendors ever took that approach, though (I don't recall
seeing any, actually, but I'm sure there must have been one or two).
Jules Richardson wrote:
Well presumably it doesn't vary *that* much, even on a flakey old drive...?
Yep. But I'm not sure that it's really
rocket science - assuming that
you know what encoding format the data's actually in, which is
probably the more difficult part. Coping with errors might be
interesting in some cases.
It isn't. The hardest part is converting from timing data back to an MFM
bitstream or binary data. You need to either handle changes in drive
speed in real time (i.e. track the read speed and continually adjust the
1t, 1.5t and 2t timing values) or do something similar to disk2fdi --
preprocess the data and figure out a set of thresholds that work,
assuming the disc is in good shape. Mmm, there's just a bit of a difference between
"what the math says" and
"what Murphy allows".
Heh, yeah...
The added board space is a bigger concern...
bah, stack one board atop the other with a bus between 'em ;)
I actually got banned from the school computer lab
many years ago for
allegedly destroying the hard drive in an Acorn A440. Said drive being a
20MB Kalok thing that was shaky at the best of times. I was dragged out
of class for half a day to help the "IT expert" (actually Just Another
Teacher [tm]) fix it. Most of that time was spent staring at !HForm's
soak-test screen and writing down all the errors, on IT Expert's orders.
One every two sectors IIRC.
"Make sure you don't get ANY of the numbers wrong! You have to re-enter
them when it actually formats the drive!"
I think in that situation I would have deliberately inverted the numbers
before handing the data over :-)
I'm not sure I ever had a Kalok drive, even in any of my Acorn stuff. My
loathing's reserved for the BASF units that Torch commonly used...
I'm not
sure how easy it is to tap into that kind of thing on a lot of
drives, though. Possibly best left for real emergencies on a
case-by-case basis...
It's "find a clean-room" territory. The head preamps are usually mounted
near the heads, often on the actuator arms themselves. Oh, food. A handful of pellets, a handful of fresh hay
and a fresh
bottle of water every 24 hours. Not hard, just expensive :)
I hear ya. Ours spend half their time outdoors, and at this time of year I
suppose they need to eat a lot. We've got a couple of dogs too - but then they
all make nice little portable space heaters :-)
That's probably worth looking into, but I know
very little about PCI
device design...
Ditto. Back in the day I wanted to get away from the internal card
approach - but these days it's possible to get a tiny, cheap
motherboard and should be relatively easy to build a networkable
"external device" that's a complete compact PC in its own right. Or you move to China, open a factory and pick a number
at random and
pray it doesn't clash with someone else's ID. Or pick one that isn't on
the Linux PCI Vendor ID list...
Grr, read entire message before typing reply, Jules... :-)
I'd say stuff it and make something up, assuming this is always going to be a
hobbyist thing rather than a serious commercial venture. Preferably make it
read from a ROM or other non-hard-coded system...
I think
I've got some Tandon and Epson drive schematics kicking around
and they used separate stages. But tapping into the head signal on a
floppy drive is a darn sight easier than a hard disk...
Mainly because the manuals for floppy drives tend to include
schematics... The ST412/506 OEM manual doesn't. Many a good idea has been born of idle speculation and
what-ifs.
... and of beer.
5741
days inactive
5744
days old
11 comments
3 participants
participants (3)
-
classiccmp＠philpem.me.uk -
frustum＠pacbell.net -
jules.richardson99＠gmail.com