15 Nov
2015
15 Nov
'15
2:15 a.m.
While I'm new to speak up here, I've been watching for a while. I've also
used the name "MightyFrame" on groups, although that email address is not
registered with this list.
A few months back, I published a page decoding QIC-24 format.
http://mightyframe.blogspot.com/2015/08/qic-24-tape-data-block-format-decod…
Dwight posted a very meaningful response to this about the CRC on my site,
and it has taken me this long just to get to it. Thanks again for that,
Dwight!
This project has gone very well for me, with one hangup...the CRC.
I've never calculated one of these before, and I'm struggling with this one.
I have a QIC-24 block of data attached in a .txt file, and I just can't get
the calculations to work.
For my simple understanding so far, I'm using an online calculator to test
this:
http://bit.ly/1YbHUZ7
That URL is pre-programmed with the polynomial and the "message" that
should provide the CRC result of
0xE8CE (decimal 59598)
But, it doesn't.
I created this file, wrote it to tape, then read it again and tested it.
It is 100% correct for the QIC-24 tape system. This is block 3 of 4 of the
file that I have extracted, and am testing here.
The ANSI QIC-24 standard booklet (X3.136-1986) says something that I know
is a clue, but I just don't understand.
"The cyclical redundancy check (CRC) shall consist of two bytes, calculated
over the 512 bytes of interchange data, and the 4-byte block address, *starting
with all ONEs, CRC initial value*, and using the CRC Generating polynomial:
x16 + x12 + x5 + 1
*"starting with all ONEs, CRC initial value"*? What does THAT mean? Do I
need to do some kind of register shift? xorin or xorout?
I've even played with http://reveng.sourceforge.net/, but I'm having
trouble even understanding the meaning of the input values and parameters
with this.
I appreciate any feedback that anyone can give, here.
--
Thanks,
-AJ
http://MicrotechM1.blogspot.com
http://MightyFrame.com
15 Nov
15 Nov
2:49 a.m.
On 15/11/15 11:15, Microtech Dart wrote:
(snip)
"The cyclical redundancy check (CRC) shall consist of two bytes, calculated
over the 512 bytes of interchange data, and the 4-byte block address, *starting
with all ONEs, CRC initial value*, and using the CRC Generating polynomial:
x16 + x12 + x5 + 1
*"starting with all ONEs, CRC initial value"*? What does THAT mean? Do I
need to do some kind of register shift? xorin or xorout?
It's the initial
value of each bit, corresponding to the "Init before
calculation" lines in the code at the bottom of the page. There doesn't
seem to be any provision for entering this value in the online
computation (and all-zeros is *not* a good initialisation vector!)
One way around this would be to come up with a byte sequence that
gave a CRC of all ones (i.e. 84 CF) and pre-pend that to the main
sequence, and then it works. But I assume you're using some sort of
code, so just make sure it's initialised appropriately.
I've even played with http://reveng.sourceforge.net/, but I'm having
trouble even understanding the meaning of the input values and parameters
with this.
I appreciate any feedback that anyone can give, here.
Best of luck! CRCs are fun!
--
Lawrence Wilkinson lawrence at ljw.me.uk
The IBM 360/30 page http://www.ljw.me.uk/ibm360
7:13 a.m.
I'm not sure why the program your using doesn't allow an initial value.
Are you sure it is starting with 0's?
If you give it a data set that is all 0's and the result is 0's, it is starting
with 0's.
By starting with all 1's, usually means the check sum is all 1's, not that
one entered all 1's.
Most CRC's do LSB end in first but some do MSB. Try both ways.
Some use a table lookup for a byte at a time rather than bit at a time.
This it is much faster but can't be used for correction ( without difficulty ).
If you have some coding experience, you should be able to code your own.
Dwight
________________________________________
From: cctalk <cctalk-bounces at classiccmp.org> on behalf of Lawrence Wilkinson
<ljw-cctech at ljw.me.uk>
Sent: Sunday, November 15, 2015 2:49 AM
To: cctalk at classiccmp.org
Subject: Re: QIC Tape help with CRC - Dwight?
On 15/11/15 11:15, Microtech Dart wrote:
(snip)
"The cyclical redundancy check (CRC) shall consist of two bytes, calculated
over the 512 bytes of interchange data, and the 4-byte block address, *starting
with all ONEs, CRC initial value*, and using the CRC Generating polynomial:
x16 + x12 + x5 + 1
*"starting with all ONEs, CRC initial value"*? What does THAT mean? Do I
need to do some kind of register shift? xorin or xorout?
It's the initial
value of each bit, corresponding to the "Init before
calculation" lines in the code at the bottom of the page. There doesn't
seem to be any provision for entering this value in the online
computation (and all-zeros is *not* a good initialisation vector!)
One way around this would be to come up with a byte sequence that
gave a CRC of all ones (i.e. 84 CF) and pre-pend that to the main
sequence, and then it works. But I assume you're using some sort of
code, so just make sure it's initialised appropriately.
I've even played with http://reveng.sourceforge.net/, but I'm having
trouble even understanding the meaning of the input values and parameters
with this.
I appreciate any feedback that anyone can give, here.
Best of luck! CRCs are fun!
--
Lawrence Wilkinson lawrence at ljw.me.uk
The IBM 360/30 page http://www.ljw.me.uk/ibm360
2:52 p.m.
Lawrence, Dwight & Chuck...thank you!
The revelation that I can append 84CF to the 512-byte data block and the
4-byte block address, is golden, and this works!
It would have taken a lot longer for me to figure out on my own that this
was a work-around, and at least one working interpretation of the confusing
text I was quoting from the manuals.
I don't grasp all of the rest of what you shared fully, but now I at least
have an example of something that works to play with, and from there, I
hope to get more comfortable with this procedure, and write some programs
that do this checking to validate each block I read with my Universal QIC
reader system.
I've updated my web page about QIC-24 decoding CRC section to include this
knowledge:
http://mightyframe.blogspot.com/2015/08/qic-24-tape-data-block-format-decod…
Thank you again!
-AJ
On Sun, Nov 15, 2015 at 4:49 AM, Lawrence Wilkinson <ljw-cctech at ljw.me.uk>
wrote:
On 15/11/15 11:15, Microtech Dart wrote:
(snip)
--
Thanks,
-AJ
http://MicrotechM1.blogspot.com
"The cyclical redundancy check (CRC) shall
consist of two bytes,
calculated
over the 512 bytes of interchange data, and the 4-byte block address,
*starting
with all ONEs, CRC initial value*, and using the CRC Generating
polynomial:
x16 + x12 + x5 + 1
*"starting with all ONEs, CRC initial value"*? What does THAT mean? Do I
need to do some kind of register shift? xorin or xorout?
It's the initial value of each bit, corresponding to the "Init before
calculation" lines in the code at the bottom of the page. There doesn't
seem to be any provision for entering this value in the online
computation (and all-zeros is *not* a good initialisation vector!)
One way around this would be to come up with a byte sequence that
gave a CRC of all ones (i.e. 84 CF) and pre-pend that to the main
sequence, and then it works. But I assume you're using some sort of
code, so just make sure it's initialised appropriately.
I've even played with
http://reveng.sourceforge.net/, but I'm having
trouble even understanding the meaning of the input values and parameters
with this.
I appreciate any feedback that anyone can give, here.
Best of luck! CRCs are fun!
--
Lawrence Wilkinson lawrence at ljw.me.uk
The IBM 360/30 page http://www.ljw.me.uk/ibm360
10:24 p.m.
Ah, I thought you were writing a specific program, but if you're
just documenting things then specifying the CRC polynomial and
the init value as all-ones should be sufficient.
The corresponding init parameter of reveng is "-i ffff".
On 15/11/15 23:52, Microtech Dart wrote:
Lawrence, Dwight & Chuck...thank you!
The revelation that I can append 84CF to the 512-byte data block and the
4-byte block address, is golden, and this works!
The other thing to point out is
that if you append the CRC to
your original block, the resulting CRC should be 0000.
e.g. 84 CF FF FF gives 0000
This means there's no need to do a specific 'CRC check', but
after your data block you keep feeding the CRC bytes in and
then check that the result is 0000.
It would have taken a lot longer for me to figure out on my own that this
was a work-around, and at least one working interpretation of the confusing
text I was quoting from the manuals.
I don't grasp all of the rest of what you shared fully, but now I at least
have an example of something that works to play with, and from there, I
hope to get more comfortable with this procedure, and write some programs
that do this checking to validate each block I read with my Universal QIC
reader system.
I've updated my web page about QIC-24 decoding CRC section to include this
knowledge:
http://mightyframe.blogspot.com/2015/08/qic-24-tape-data-block-format-decod…
When specifying a CRC it's always good to provide a short
example that can be used to validate an implementation - it
doesn't need to be a whole block, but just 10 or so bytes
together with the resulting CRC.
Thank you again!
-AJ
--
Lawrence Wilkinson lawrence at ljw.me.uk
The IBM 360/30 page http://www.ljw.me.uk/ibm360
16 Nov
16 Nov
12:01 a.m.
Thanks, Lawrence, again. Actually, I am writing a very specific program,
for my universal tape reader system to do these CRC checks as the program
identifies each block.
Thanks to adding the 84CF, this has now been made much, much easier.
I'm still new to all of this though, so while my program works as of
tonight, there is still much for me to learn about how these work.
Great information, please...keep it coming!
Thanks again,
-AJ
On Mon, Nov 16, 2015 at 12:24 AM, Lawrence Wilkinson <ljw-cctech at ljw.me.uk>
wrote:
Ah, I thought you were writing a specific program, but
if you're
just documenting things then specifying the CRC polynomial and
the init value as all-ones should be sufficient.
The corresponding init parameter of reveng is "-i ffff".
On 15/11/15 23:52, Microtech Dart wrote:
--
Thanks,
-AJ
http://MicrotechM1.blogspot.com
Lawrence, Dwight & Chuck...thank you!
The revelation that I can append 84CF to the 512-byte data block and the
4-byte block address, is golden, and this works!
The other thing to point out is that if you append the CRC to
your original block, the resulting CRC should be 0000.
e.g. 84 CF FF FF gives 0000
This means there's no need to do a specific 'CRC check', but
after your data block you keep feeding the CRC bytes in and
then check that the result is 0000.
It would have taken a lot longer for me to figure
out on my own that this
was a work-around, and at least one working interpretation of the
confusing
text I was quoting from the manuals.
I don't grasp all of the rest of what you shared fully, but now I at least
have an example of something that works to play with, and from there, I
hope to get more comfortable with this procedure, and write some programs
that do this checking to validate each block I read with my Universal QIC
reader system.
I've updated my web page about QIC-24 decoding CRC section to include this
knowledge:
http://mightyframe.blogspot.com/2015/08/qic-24-tape-data-block-format-decod…
When specifying a CRC it's always good to provide a short
example that can be used to validate an implementation - it
doesn't need to be a whole block, but just 10 or so bytes
together with the resulting CRC.
Thank you again!
-AJ
--
Lawrence Wilkinson lawrence at ljw.me.uk
The IBM 360/30 page http://www.ljw.me.uk/ibm360
9 Dec
9 Dec
11:38 p.m.
Just a quick update here, I believe that I have finally cracked the code on
the CRC polynomial/model for this enigmatic Kennedy 64XX QIC tape format.
Please see
http://microtechm1.blogspot.com/2015/09/kennedy-6450-tape-drive-data-format…
Here are the core findings:
For both 4096 and 8192-byte data blocks:
width=16 poly=0x8005 init=0x0002 refin=false refout=false xorout=0xfffd
check=0x2119 name=(none)
reveng -w 16 -p 8005 -i 0002 -x fffd -c -f Block.bin
--OR--
width=16 poly=0x8005 init=0x8001 refin=false refout=false xorout=0x7ffe
check=0x2119 name=(none)
reveng -w 16 -p 8005 -i 8001 -x 7ffe -c -f Block.bin
(For 4096-byte data blocks only?)
width=16 poly=0x8005 init=0xbffe refin=false refout=false xorout=0x0000
check=0x26e7 name=(none)
reveng -w 16 -p 8005 -i bffe -c -f Block.bin
(For 8192-byte data blocks only?)
width=16 poly=0x8005 init=0x5ffc refin=false refout=false xorout=0x0000
check=0x62e7 name=(none)
reveng -w 16 -p 8005 -i 5ffc -c -f Block.bin
CRC_RevEng was a BIG help, once I got more comfortable using it.
Has anyone here ever seen models like this for CRCs in any other media
storage tech? Maybe CRCs on open reel tape (if they had any?), or CRCs on
hard drives?
My grasp on CRC math and formulas is still at the novice level, however. I
am thrown by the fact that the derived model requires an XorOUT parameter
value in order to work properly on both the 4096 and the 8192-byte blocks
found on these tapes. My self-written program that decodes these tapes
doesn't have the ability to input an XorOUT value, mostly because I don't
really understand what it means.
I read in the CRC_RevEng Users Guide:
<http://reveng.sourceforge.net/readme.htm> "-x XOROUT: A hexadecimal digit
string specifying the value to be added to the final shift register value
before output."
I did find models with no XorOUT value required that work either for the
4096-byte blocks, and a different one for the 8192-byte blocks. I created a
work-around in my program by selecting a different model based on the size
of the block being analyzed.
Does anyone here have any insight as to why these models will consistently
work for one, but not the other, and vice-versa?
Again, thanks everyone for the great knowledgeable input. Your shared
knowledge has goten me to this point on these projects.
-AJ
http://MightyFrame.com
http://MicrotechM1.blogspot.com
On Mon, Nov 16, 2015 at 12:24 AM, Lawrence Wilkinson <ljw-cctech at ljw.me.uk>
wrote:
Ah, I thought you were writing a specific program, but
if you're
just documenting things then specifying the CRC polynomial and
the init value as all-ones should be sufficient.
The corresponding init parameter of reveng is "-i ffff".
On 15/11/15 23:52, Microtech Dart wrote:
--
Thanks,
-AJ
http://MicrotechM1.blogspot.com
Lawrence, Dwight & Chuck...thank you!
The revelation that I can append 84CF to the 512-byte data block and the
4-byte block address, is golden, and this works!
The other thing to point out is that if you append the CRC to
your original block, the resulting CRC should be 0000.
e.g. 84 CF FF FF gives 0000
This means there's no need to do a specific 'CRC check', but
after your data block you keep feeding the CRC bytes in and
then check that the result is 0000.
It would have taken a lot longer for me to figure
out on my own that this
was a work-around, and at least one working interpretation of the
confusing
text I was quoting from the manuals.
I don't grasp all of the rest of what you shared fully, but now I at least
have an example of something that works to play with, and from there, I
hope to get more comfortable with this procedure, and write some programs
that do this checking to validate each block I read with my Universal QIC
reader system.
I've updated my web page about QIC-24 decoding CRC section to include this
knowledge:
http://mightyframe.blogspot.com/2015/08/qic-24-tape-data-block-format-decod…
When specifying a CRC it's always good to provide a short
example that can be used to validate an implementation - it
doesn't need to be a whole block, but just 10 or so bytes
together with the resulting CRC.
Thank you again!
-AJ
--
Lawrence Wilkinson lawrence at ljw.me.uk
The IBM 360/30 page http://www.ljw.me.uk/ibm360
10 Dec
10 Dec
7:38 a.m.
I'm not familiar with a XorOut? I suspect it is to correct for
a different seed values at the tail end.
Does the program let you run it backwards?
As a 16 bit polynomial, it is practical to run it forward 2^16 bit times
to get it back to the seed value.
Dwight
________________________________________
From: cctalk <cctalk-bounces at classiccmp.org> on behalf of Microtech Dart
<microtechdart at gmail.com>
Sent: Wednesday, December 9, 2015 11:38 PM
To: General Discussion: On-Topic and Off-Topic Posts
Subject: Re: QIC Tape help with CRC - Dwight?
Just a quick update here, I believe that I have finally cracked the code on
the CRC polynomial/model for this enigmatic Kennedy 64XX QIC tape format.
Please see
http://microtechm1.blogspot.com/2015/09/kennedy-6450-tape-drive-data-format…
Here are the core findings:
For both 4096 and 8192-byte data blocks:
width=16 poly=0x8005 init=0x0002 refin=false refout=false xorout=0xfffd
check=0x2119 name=(none)
reveng -w 16 -p 8005 -i 0002 -x fffd -c -f Block.bin
--OR--
width=16 poly=0x8005 init=0x8001 refin=false refout=false xorout=0x7ffe
check=0x2119 name=(none)
reveng -w 16 -p 8005 -i 8001 -x 7ffe -c -f Block.bin
(For 4096-byte data blocks only?)
width=16 poly=0x8005 init=0xbffe refin=false refout=false xorout=0x0000
check=0x26e7 name=(none)
reveng -w 16 -p 8005 -i bffe -c -f Block.bin
(For 8192-byte data blocks only?)
width=16 poly=0x8005 init=0x5ffc refin=false refout=false xorout=0x0000
check=0x62e7 name=(none)
reveng -w 16 -p 8005 -i 5ffc -c -f Block.bin
CRC_RevEng was a BIG help, once I got more comfortable using it.
Has anyone here ever seen models like this for CRCs in any other media
storage tech? Maybe CRCs on open reel tape (if they had any?), or CRCs on
hard drives?
My grasp on CRC math and formulas is still at the novice level, however. I
am thrown by the fact that the derived model requires an XorOUT parameter
value in order to work properly on both the 4096 and the 8192-byte blocks
found on these tapes. My self-written program that decodes these tapes
doesn't have the ability to input an XorOUT value, mostly because I don't
really understand what it means.
I read in the CRC_RevEng Users Guide:
<http://reveng.sourceforge.net/readme.htm> "-x XOROUT: A hexadecimal digit
string specifying the value to be added to the final shift register value
before output."
I did find models with no XorOUT value required that work either for the
4096-byte blocks, and a different one for the 8192-byte blocks. I created a
work-around in my program by selecting a different model based on the size
of the block being analyzed.
Does anyone here have any insight as to why these models will consistently
work for one, but not the other, and vice-versa?
Again, thanks everyone for the great knowledgeable input. Your shared
knowledge has goten me to this point on these projects.
-AJ
http://MightyFrame.com
http://MicrotechM1.blogspot.com
On Mon, Nov 16, 2015 at 12:24 AM, Lawrence Wilkinson <ljw-cctech at ljw.me.uk>
wrote:
Ah, I thought you were writing a specific program, but
if you're
just documenting things then specifying the CRC polynomial and
the init value as all-ones should be sufficient.
The corresponding init parameter of reveng is "-i ffff".
On 15/11/15 23:52, Microtech Dart wrote:
--
Thanks,
-AJ
http://MicrotechM1.blogspot.com
Lawrence, Dwight & Chuck...thank you!
The revelation that I can append 84CF to the 512-byte data block and the
4-byte block address, is golden, and this works!
The other thing to point out is that if you append the CRC to
your original block, the resulting CRC should be 0000.
e.g. 84 CF FF FF gives 0000
This means there's no need to do a specific 'CRC check', but
after your data block you keep feeding the CRC bytes in and
then check that the result is 0000.
It would have taken a lot longer for me to figure
out on my own that this
was a work-around, and at least one working interpretation of the
confusing
text I was quoting from the manuals.
I don't grasp all of the rest of what you shared fully, but now I at least
have an example of something that works to play with, and from there, I
hope to get more comfortable with this procedure, and write some programs
that do this checking to validate each block I read with my Universal QIC
reader system.
I've updated my web page about QIC-24 decoding CRC section to include this
knowledge:
http://mightyframe.blogspot.com/2015/08/qic-24-tape-data-block-format-decod…
When specifying a CRC it's always good to provide a short
example that can be used to validate an implementation - it
doesn't need to be a whole block, but just 10 or so bytes
together with the resulting CRC.
Thank you again!
-AJ
--
Lawrence Wilkinson lawrence at ljw.me.uk
The IBM 360/30 page http://www.ljw.me.uk/ibm360
15 Nov
15 Nov
10:35 a.m.
I guess I should throw my own experience on this.
First off, it's useful to think about a CRC as just another checksum.
Instead of using a simple addition, however, a somewhat more complex
(table-drive) operation is used (that's your polynomial).
In other words, if S is the CRC sum and I is the current input, a CRC is
nothing more than S = sum(S,I) for every I. In the case of an
arithmetic checksum the function "sum" is a simple addition. In a CRC
it's more involved. You can see that the initial value of S is important.
In CRCs, this initial value can be anything, but usually, it's 0 or all
binary ones. Note that CRC calculation doesn't use arithmetic per se,
but rather boolean ops and shifts. The character of the operation is
dictated by the CRC polynomial.
I'll go out on a limb here and say that most often the polynomial used
in storage devices, where a 16-bit CRC is calculated is the standard
CCITT CRC-16 one. There are exceptions, but really, assuming that
polynomial is an excellent starting point.
Once you've selected a polynomial candidate, the second thing to do is
to determine exactly what the input to the CRC computation is. You'd
think, for example, that on most IBM-type floppy disks, that it's the
data in a sector--and you'd be wrong. Most floppy CRC computation
includes the data address mark (DAM) as well. So, in a typical 1.44M
IBM floppy, that data address mark would be hex "FB", followed by sector
data.
The third thing to consider is what order that bits are presented to the
CRC calculator--that is, high-order or low-order bit first. Floppy
disks mostly go one way and USRTs (synchronous comm) go the other way.
So you could have the same value treated two different ways and get two
completely different CRCs, based on the same polynomial. This is a real
gotcha when dealing with early disk formats, as many of those used an
inexpensive USRT or shift register rather than an expensive-for-the-time
LSI controller.
I find it useful to have a pair of simple test programs that, given a
CRC polynomial and initial value and an input file, produce a CRC for
comparison. One routine calculates little-bit endian and the other does
the same for big-endian calculations.
CRC_reveng is a great routine, but much of the time, it's overkill.
For whatever it's worth,
Chuck
3300
days inactive
3325
days old
8 comments
4 participants
participants (4)
-
cclist＠sydex.com -
dkelvey＠hotmail.com -
ljw-cctech＠ljw.me.uk -
microtechdart＠gmail.com