11 Dec
2014
11 Dec
'14
9:50 p.m.
I've been playing around with some SCSI tape drives and noticed
something peculiar--or not intuitive. Perhaps someone can explain it.
I'm doing a READ_6 (6 byte CDB) with the fixed bit 0 and a 65KB buffer.
So the sense data coming back returns the residual byte count, as it
should with the ILI bit set in the sense key.
So far, so good.
Now, when I come to a filenark, I'm expecting that the filemark bit (bit
7) will be set in the sense key. It doesn't happen, the result just
comes back as 0 bytes read and a normal (NO_SENSE) sense result. The
additional sense code and sense qualifier bytes are also 0.
Is this normal behavior? I'dve thought that the filemark bit would be
set in every case of a filemark being read. My X3T9 documents don't
mention this behavior, but maybe I missed something.
--Chuck
12 Dec
12 Dec
7:18 a.m.
On Thu, Dec 11, 2014 at 06:50:53PM -0800, Chuck Guzis wrote:
[...]
Now, when I come to a filenark, I'm expecting that
the filemark bit (bit 7)
will be set in the sense key. It doesn't happen, the result just comes back
as 0 bytes read and a normal (NO_SENSE) sense result. The additional sense
code and sense qualifier bytes are also 0.
Is this normal behavior? I'dve thought that the
filemark bit would be set in
every case of a filemark being read. My X3T9 documents don't mention this
behavior, but maybe I missed something.
My copy of the *draft* standard states "If the logical unit encounters a
filemark during a READ command, CHECK CONDITION status shall be returned and
the filemark and valid bits shall be set to one in the sense data" but perhaps
the final standard decided to be different for perverse reasons. Draft copies
of questionable provenance is what everybody's working from though, because the
final versions are $$$.
Note that there are *three* different types of end-of-X event with tape:
end-of-file at a filemark, end-of-medium indicating the logical end of the
recording, and end-of-tape where it's flapping off the reel (which is always
"fun" when it happens to QIC cartridges). If the tape has been reused, there
may be data between end-of-recording and end-of-tape, but you're probably not
going to get at it from SCSI-space.
The behaviour you describe dovetails nicely with the POSIX API which indicates
EOF by returning zero to a read() call. On Linux, the next read() call then
starts reading the next file. I had to check the man page to remind me what
happens at end-of-medium: it's indicated by two consecutive EOFs followed by
"an error" (not stated, but EIO seems likely), which can alternatively be
viewed as an empty file followed by something to break applications that didn't
take the hint the first time.
Anyway, this suggests an alternative API that your drive might be (incorrectly)
providing, and perhaps you will get two sequential zero-length reads and/or a
sense error and filemark bit only at end-of-medium.
I would certainly also check out the source for Linux's SCSI tape driver. The
Linux kernel takes a somewhat pragmatic approach in the face of dodgy hardware,
so there are loads of edge cases with accompanying rants explaining in
technical detail just why such-and-such a device is a steaming pile.
11:54 a.m.
On 12/12/2014 04:18 AM, Peter Corlett wrote:
My copy of the *draft* standard states "If the
logical unit encounters a
filemark during a READ command, CHECK CONDITION status shall be returned and
the filemark and valid bits shall be set to one in the sense data" but perhaps
the final standard decided to be different for perverse reasons. Draft copies
of questionable provenance is what everybody's working from though, because the
final versions are $$$.
Well, this is just odd--if I do my read not as variable length, but
rather using specified block size and the "fixed" bit (bit 0 of byte 1
in the READ command), I do indeed get the filemark indication.
I'll have a look at the *nix tape driver code, but AFAIK, the standard
tape handlers for Linux and Unix depend on the block length being
explicitly specified in a MODE SELECT command. When doing
variable-length reads, this block length is 0.
And yes, I can read past double filemarks and get to the previously
written data. Eventually, a blank media or EOM condition arises and
calls a halt to things. Recently, I ran into a case where the
double-filemark was an accident--a zero-length file was written, so the
disrespect for double filemarks is necessary.
I've got an old X3T9 spec here, as well as a few vendors' manuals and
the strange non-filemark behavior isn't detailed--however, it's simple
enough to translate this into the expected "filemark-sensed" code.
It's not a stumble, just curiosity.
--Chuck
12:10 p.m.
On 2014-12-12 17:54, Chuck Guzis wrote:
On 12/12/2014 04:18 AM, Peter Corlett wrote:
I don't know much of anything about the low level SCSI details here. So
I can't comment on that.
My copy of the *draft* standard states "If
the logical unit encounters a
filemark during a READ command, CHECK CONDITION status shall be
returned and
the filemark and valid bits shall be set to one in the sense data" but
perhaps
the final standard decided to be different for perverse reasons.
Draft copies
of questionable provenance is what everybody's working from though,
because the
final versions are $$$.
Well, this is just odd--if I do my read not as variable length, but
rather using specified block size and the "fixed" bit (bit 0 of byte 1
in the READ command), I do indeed get the filemark indication.
I'll have a look at the *nix tape driver code, but AFAIK, the standard
tape handlers for Linux and Unix depend on the block length being
explicitly specified in a MODE SELECT command. When doing
variable-length reads, this block length is 0. And yes, I can read past double filemarks and get to
the previously
written data. Eventually, a blank media or EOM condition arises and
calls a halt to things. Recently, I ran into a case where the
double-filemark was an accident--a zero-length file was written, so the
disrespect for double filemarks is necessary.
Two tape marks is just a convention used by some software to indicate
the logical EOT. It has no special meaning for the hardware. In fact,
standard ANSI formatted tapes can legally contain two consecutive tape
marks without it being the EOT. ANSI instead have a special record to
indicate EOT.
Johnny
12:29 p.m.
On 12/12/2014 09:10 AM, Johnny Billquist wrote:
Two tape marks is just a convention used by some
software to indicate
the logical EOT. It has no special meaning for the hardware. In fact,
standard ANSI formatted tapes can legally contain two consecutive tape
marks without it being the EOT. ANSI instead have a special record to
indicate EOT.
Old tapes are extremely variable on their use of ANSI labels, so when
archiving, I'll read as far as possible (i.e. either to EOT or a "blank
media" exception.
And then there's Unix/Linux and many other OSes that don't give a fig
about ANSI labels.
--Chuck
12:45 p.m.
On 2014-12-12 18:29, Chuck Guzis wrote:
On 12/12/2014 09:10 AM, Johnny Billquist wrote:
You mean old OSes...? :-)
Two tape marks is just a convention used by some
software to indicate
the logical EOT. It has no special meaning for the hardware. In fact,
standard ANSI formatted tapes can legally contain two consecutive tape
marks without it being the EOT. ANSI instead have a special record to
indicate EOT.
Old tapes are extremely variable on their use of ANSI labels, so when
archiving, I'll read as far as possible (i.e. either to EOT or a "blank
media" exception. And then there's Unix/Linux and many other OSes
that don't give a fig
about ANSI labels.
Even for OSes that do give a fig about ANSI labels, they are not the
bible. You can read the tapes without using that layer on any OS...
It's just another layer you can use in between, to give some more structure.
Johnny
1:32 p.m.
On 12/12/2014 09:45 AM, Johnny Billquist wrote:
You mean old OSes...? :-)
Maybe, but it's usually the case here that I can't render that opinion
(or usually even the age of the tape) until I've read and analyzed the
contents before I can determine the OS. In fact, on some tapes, the OS
remains completely unknown--just a bunch of ASCII files separated by
filemarks...
And what's your definition of "old"? :)
--Chuck
15 Dec
15 Dec
8:23 a.m.
Two tape marks is just a convention used by some
software to indicate
the logical EOT. It has no special meaning for the hardware.
Perhaps on some drives (such as 9-tracks). But I did run into one
drive, one of the relatively modern cartridge drives with tapes about
the size of a TK50, which had a relatively elaborate encoding scheme in
which EOT was different from two consecutive tapemarks, but it mapped
between the two in each direction and the drive refused to even try to
read past EOT. In order to recover data that had been written over,
the recommended procedure was to write enough to overwrite the EOT mark
and then power down the drive while it was writing more. (Recommended
by users, not the maker IIRC.)
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML mouse at rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
16 Dec
16 Dec
3:13 a.m.
On 2014-12-15 14:23, Mouse wrote:
Oh. Yuck... So ANSI tape format would be impossible on such a tape. That
is a very broken hardware.
Johnny
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: bqt at softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
Two tape marks
is just a convention used by some software to indicate
the logical EOT. It has no special meaning for the hardware.
Perhaps on some drives (such as 9-tracks). But I did run into one
drive, one of the relatively modern cartridge drives with tapes about
the size of a TK50, which had a relatively elaborate encoding scheme in
which EOT was different from two consecutive tapemarks, but it mapped
between the two in each direction and the drive refused to even try to
read past EOT. In order to recover data that had been written over,
the recommended procedure was to write enough to overwrite the EOT mark
and then power down the drive while it was writing more. (Recommended
by users, not the maker IIRC.) 
12 Dec
12 Dec
12:48 p.m.
On 12/12/14 8:54 AM, Chuck Guzis wrote:
On 12/12/2014 04:18 AM, Peter Corlett wrote:
This is bringing back a lot of bad memories..
You may want to look at
https://www.brouhaha.com/~eric/software/tapeutils/
and John Wilson's tapeio routines
I've run into problems with tapecopy where it gets stuck in an infinite loop writing
-1 byte
blocks if it ever runs into a bad block.
I'll have to look at the SCSI code I wrote back in 2001 for my direct SCSI tape reader
on MacOS
but I'm pretty sure I ended up having to read fixed-length 64k blocks there.
This gunk was creeping back up on my todo list as I really do need to retire my G3
powerbook that
I use for tape reading, mostly because I've been 'upgrading' my servers and
they no longer understand
AFP that OS9 uses.
My copy of the *draft* standard states "If
the logical unit encounters a
filemark during a READ command, CHECK CONDITION status shall be returned and
the filemark and valid bits shall be set to one in the sense data" but perhaps
the final standard decided to be different for perverse reasons. Draft copies
of questionable provenance is what everybody's working from though, because the
final versions are $$$.
Well, this is just odd--if I do my read not as variable length, but rather using
specified block size and the "fixed" bit (bit 0 of byte 1 in the READ command),
I do indeed get the filemark indication.
I'll have a look at the *nix tape driver code, but AFAIK, the standard tape handlers
for Linux and Unix depend on the block length being explicitly specified in a MODE SELECT
command. When doing
variable-length reads, this block length is 0.
1:41 p.m.
On 12/12/2014 09:48 AM, Al Kossow wrote:
This is bringing back a lot of bad memories..
You may want to look at
https://www.brouhaha.com/~eric/software/tapeutils/
and John Wilson's tapeio routines
I've seen those, but encountered issues with them and decided the only
route was to roll my own. Nothing new--I simply went back to my old
production code that I wrote around 1990 to handle SCSI tape drives in
general (DDS, DLT, SLT) and resurrected some things.
But, like a lot of code, my old code worked with tape of *known* format.
Working with completely unknown formats has been interesting, to say
the least.
Thankfully, I've been spared the issue of "long" tapes--that is, tapes
with physical blocks longer than 65KB--I know they're out there, though.
--Chuck
10:24 p.m.
On 12/12/2014 10:54 AM, Chuck Guzis wrote:
On 12/12/2014 04:18 AM, Peter Corlett wrote:
LOTS of SCSI devices violated the standard protocol in
interesting ways. My understanding
is there are two classes of SCSI drives. One class used a
scheme that was quite like old
9-track tapes, so that there was essentially no formatting
that wrapped around the user
data. So, there would be a physical gap with no magnetic
transitions between the
written user data blocks. There would be a CRC of some sort
at the end of the user
data. Higher-end drives wrapped the user data into a more
complex format, so each
block would have a block size and a sequence number at the
beginning, and maybe
a trailer of the same info with the CRC. Often in these
systems, there would be NO
physical gap. This increased data capacity, but made it
impossible to rewrite a single
tape block. The file mark was just a special block with
minimal or zero-length data
packet.
Jon
My copy of the *draft* standard states "If
the logical
unit encounters a
filemark during a READ command, CHECK CONDITION status
shall be returned and
the filemark and valid bits shall be set to one in the
sense data" but perhaps
the final standard decided to be different for perverse
reasons. Draft copies
of questionable provenance is what everybody's working
from though, because the
final versions are $$$.
Well, this is just odd--if I do my read not as variable
length, but rather using specified block size and the
"fixed" bit (bit 0 of byte 1 in the READ command), I do
indeed get the filemark indication.
I'll have a look at the *nix tape driver code, but AFAIK,
the standard tape handlers for Linux and Unix depend on
the block length being explicitly specified in a MODE
SELECT command. When doing variable-length reads, this
block length is 0.
And yes, I can read past double filemarks and get to the
previously written data. Eventually, a blank media or EOM
condition arises and calls a halt to things. Recently, I
ran into a case where the double-filemark was an
accident--a zero-length file was written, so the
disrespect for double filemarks is necessary.
I've got an old X3T9 spec here, as well as a few vendors'
manuals and the strange non-filemark behavior isn't
detailed--however, it's simple enough to translate this
into the expected "filemark-sensed" code.
It's not a stumble, just curiosity.
10:41 p.m.
On 12/12/2014 07:24 PM, Jon Elson wrote:
LOTS of SCSI devices violated the standard protocol in
interesting
ways. My understanding is there are two classes of SCSI drives. One
class used a scheme that was quite like old 9-track tapes, so that
there was essentially no formatting that wrapped around the user
data. So, there would be a physical gap with no magnetic transitions
between the written user data blocks. There would be a CRC of some
sort at the end of the user data. Higher-end drives wrapped the user
data into a more complex format, so each block would have a block
size and a sequence number at the beginning, and maybe a trailer of
the same info with the CRC. Often in these systems, there would be
NO physical gap. This increased data capacity, but made it
impossible to rewrite a single tape block. The file mark was just a
special block with minimal or zero-length data packet.
This is a bog-standard 1600 PE density 9 track tape on a 10.5" reel.
So no surprises in the physical format. Just some screwball controller
programming.
--Chuck
13 Dec
13 Dec
12:04 a.m.
On 12/12/2014 09:41 PM, Chuck Guzis wrote:
This is a bog-standard 1600 PE density 9 track tape on a
10.5" reel.
So no surprises in the physical format. Just some
screwball controller programming.
OK, that narrows things down QUITE a bit. I
worked with
some Exabyte
drives and others that had a lot of constraints on what you
could do,
and you could do some interesting things. I think you could
write
zero-length blocks, and they would show up just like that when
read. Clearly on 9-track, there SHOULD BE no such thing as a
zero-length block, but I suppose some drives could actually
write
such a thing at 1600. A preamble, CRC, LRCC and postamble, with
no data. But, that OUGHT to be an illegal command.
Jon
1:38 a.m.
On 12/12/2014 09:04 PM, Jon Elson wrote:
OK, that narrows things down QUITE a bit. I worked
with some Exabyte
drives and others that had a lot of constraints on what you could do,
and you could do some interesting things. I think you could write
zero-length blocks, and they would show up just like that when
read. Clearly on 9-track, there SHOULD BE no such thing as a
zero-length block, but I suppose some drives could actually write
such a thing at 1600. A preamble, CRC, LRCC and postamble, with
no data. But, that OUGHT to be an illegal command.
I seem to remember on certain tapes, that very short records are
discarded or reading; something like a "noise" threshold of 7 bytes.
But that could be for 7-track and 800 9-track. I don't recall--but
regardless, mistaking a 0-length record, should one occur, for a
filemark is not catastrophic.
--Chuck
1:57 p.m.
On 12/13/2014 12:38 AM, Chuck Guzis wrote:
On 12/12/2014 09:04 PM, Jon Elson wrote:
Yes, quite correct. Turning on or off the erase head
would
leave a "noise burst" that
could be picked up by the read amps. So, in NRZI mode, you
had to ignore them.
Good drives had something like a current ramp-up control to
eliminate these while
writing.
In PE mode, there would be a preamble and postamble of 40
transitions each,
so you have no way of mistaking a little noise for the
beginning of a valid block.
Jon
OK, that narrows things down QUITE a bit. I
worked with
some Exabyte
drives and others that had a lot of constraints on what
you could do,
and you could do some interesting things. I think you
could write
zero-length blocks, and they would show up just like that
when
read. Clearly on 9-track, there SHOULD BE no such thing
as a
zero-length block, but I suppose some drives could
actually write
such a thing at 1600. A preamble, CRC, LRCC and
postamble, with
no data. But, that OUGHT to be an illegal command.
I seem to remember on certain tapes, that very short
records are discarded or reading; something like a
"noise" threshold of 7 bytes. But that could be for
7-track and 800 9-track. I don't recall--but regardless,
mistaking a 0-length record, should one occur, for a
filemark is not catastrophic. 
2:36 a.m.
On Fri, Dec 12, 2014 at 11:04:20PM -0600, Jon Elson wrote:
I worked with some Exabyte
drives and others that had a lot of constraints on what you could do,
and you could do some interesting things.
My recollection with the Exabyte 8mm drives was that they would let you
write new data after the last thing written, but you couldn't backspace and
write over existing data, unless you were rewinding all the way to BOT. So
they were lousy at emulating 9-tracks (but some controllers fixed that by
wrapping everything in headers and writing their own suffix records that
said when to disregard the last N blocks, and they'd read ahead by some
known amount looking for that, to at least handle the common case of going
to LEOT, backspacing between the two tape marks, and writing a new file).
SCSI annoyances that I remember from the drivers I've written: when a
space with a negative count (so I mean a backspace) gets terminated early
(by a tape mark or BOT), the residual count returned can be positive *or*
negative, depending on the brand of the drive. Can't blame the mfgr
because the wording in the spec could arguably be read either way, but the
point is, take the absolute value before using it (e.g. to find the length
of a file you just skipped past).
Also some drives lose their minds (not necessarily giving a sane error
in the process) if you try to backspace past BOT, so if you're writing a
general driver where that might be attempted (as opposed to a single-purpose
utility that's only going to read the entire tape in one forward pass),
keep track of the tape position and don't let yourself do this.
And if you care about the DEC TK50Z-GA or TZ30 drives, their INQUIRY
commands return short/blank data (but with one of the reserved fields set
to ^X50 or ^X30, which is a useful clue). So a plain vanilla SCSI tape
driver might gag on them, but they work OK once you get past that (I
think you have to enable buffering on one model and/or the other before
writes work, or something, but that's it).
John Wilson
D Bit
2:02 p.m.
On 12/13/2014 01:36 AM, John Wilson wrote:
On Fri, Dec 12, 2014 at 11:04:20PM -0600, Jon Elson
wrote:
No, you
could commence writing after any existing record,
and the drive would take
care of the mess that was similar to a punch-in edit on
videotape, if you remember
what that looked like on the early VCR's. It would just
overwrite the following data
for about 2-3 seconds with full-width erase, and then pick
up writing in cleanly-erased
tape. I think some of the later drives might have been able
to do better with flying
erase heads. But, we generally never did fancy stuff with
them, just write a whole
tape at a time. Anyway, we never added records to an
existing file, but did often
add new files after the double file mark.
Jon
I worked with some Exabyte
drives and others that had a lot of constraints on what you could do,
and you could do some interesting things.
My recollection with the Exabyte 8mm
drives was that they would let you
write new data after the last thing written, but you couldn't backspace and
write over existing data, unless you were rewinding all the way to BOT. So
they were lousy at emulating 9-tracks (but some controllers fixed that by
wrapping everything in headers and writing their own suffix records that
said when to disregard the last N blocks, and they'd read ahead by some
known amount looking for that, to at least handle the common case of going
to LEOT, backspacing between the two tape marks, and writing a new file).
3506
days inactive
3510
days old
18 comments
7 participants
participants (7)
-
abuse＠cabal.org.uk -
aek＠bitsavers.org -
bqt＠update.uu.se -
cclist＠sydex.com -
elson＠pico-systems.com -
mouse＠Rodents-Montreal.ORG -
wilson＠dbit.com