20 Jan
2019
20 Jan
'19
6:55 a.m.
From: Fritz Mueller
it flagged a bunch of memory locations that
weren't reported by my much
simpler diagnostic (which only does all-ones/all-zeros passes looking for
stuck bits at this point.)
What is is complaining about?
The MAINDEC memory diagnostic is bulky and
complicated, and it takes
several minutes to re-download it after a power cycle, so it's not
exactly convenient to use while troubleshooting.
Would it be possible to put it on a disk and boot it from there? If it's in
some documented format (e.g. .LDA), I can easily produce a Unix disk with it
on, if that would help (although loading the image onto the physical pack
would take forever, I guess - although you could let it run overnight).
It's probably not worth trying to devise a way to load individual files onto
a Unix disk over the serial line until Unix is working reliably, so the
program can run under Unix (otherwise a stand-alone program would have to
include file-system code).
I'll probably be beefing up my smaller diagnostic
with a few more tests
(including parity).
One of the first things to add is to store each location's address in it during
a set-up pass, and check to see that it's still there during the checking pass.
Went ahead and tried both RSTS and Unix again after
the above repair,
and saw the same fault behaviors from both (sadness).
Yeah, sounds like you still have memory issues (per the diagnostic grumping).
I tried enabling trap on parity error in the MS11 CSR
before running my
diagnostic, but it didn't trap, even though it did flag parity error(s)
in the CSR. So maybe I *also* have a bug I haven't yet addressed in
parity handling within CPU.
Starting the CPU (i.e. 'START' switch) or an INIT instruction will clear
the 'trap enable' bit in the MS11-L CSR.
I'd modify your program to set it, and check to see if you're getting
parity error traps. (Clearly, if that hardware - either in the MS11-L,
or the CPU - isn't working you need to look at that first.)
some of the earlier ones support setting a bit to
determine whether
parity errors will halt or trap the CPU
Huh? I was just looking at parity in the MM11-L and MM11-U (to see if
parity needed to be enabled on them, or if it's always on by default),
and I didn't see that. Also, there's no way I know of, on the UNIBUS,
for anything to halt the CPU (the QBUS has such as line, but not the
UNIBUS). Which memory has this feature?
I'm curious how OS init code sniffs out what
memory CSRs there are,
determines their specific flavors and, in a heterogeneous system,
determines how much address space is under the auspice of each CSR?
Unix V6 does nothing at all with parity (doesn't enable it in memory modules,
although the memory that was extant at the time - MM11-S, MM11-U, etc - did
support it as an option).
If one turned it on, the code _would_ catch the trap and 'panic' (print a
message and halt operation). It would be pretty easy to modify the code to
send a signal to the process if it happened in User mode. I'm not sure there's
much to be done if it happens in Kernel mode.
V6 sizes memory by doing a read every 0100 bytes (of the xxxx00 byte), looking
for success or a trap. If that succeeds, it clears the 32. sequential words
starting at that address, and then tries the next 0100. (So if you modified
the code to enable parity traps, you wouldn't hsave to deal with bad parity
left over from random contents at power-on....)
The 11/45 prints show a jumper (W1, lower left of
sheet UBCB) that
looks like it would entirely disable Unibus parity error detection if
removed.
Yup, that's what it looks like to me too..
when I pulled and examined my UBC board (and also
looked over my spare)
no such jumper or any associated pads were anywhere to be found! So maybe
this was either added/removed from later etches of the UBC?
Well, if you have an M8106, you do have a KB11-A; in the later /45 CPU, the
KB11-D, that has been replaced by the M8119 - but that still has W1! (The
KB11-D prints are in MP00039, 11/55 Vol 1.) I looked on my M8119, and W1 is
indeed there - it's a 0-ohm 'resistor' (single black band) just less than
half-way up the 4th column of chips, with a '1' next to it in the etch. The
M8106 board layout drawing (a couple of pages back from UBCB) does show W1 -
upper left corner of the board, next to E84.
Noel
20 Jan
20 Jan
12:56 p.m.
Thanks, Paul and Noel, for the detailed responses per usual!
On Jan 20, 2019, at 6:55 AM, Noel Chiappa <jnc at
mercury.lcs.mit.edu> wrote:
What is [MAINDEC ZQMC] complaining about?
Looks like a few more flaky bits in a couple of additional banks. For those reading along
who may be unfamiliar with the MS11-L, it is laid out as 8 physical banks, each containing
18 16K x 1 DRAMS (16 data + 2 parity bits per word. So a flaky bit in a physical bank
implicates one particular chip.
Would it be possible to put [ZQMC] on a disk and boot
it from there?
I have thought about that... The most efficient way I think would be to work up a simple
LDA loader that would fit in a boot sector, and load a diagnostic from contiguous disk
starting at the second sector. It would then be easy to blast down just the boot sector
and a single desired diagnostic without imaging an entire pack.
One of the first things to add [to custom diagnostic]
is to store each location's address in it during a set-up pass, and check to see that
it's still there during the checking pass.
I did this last night, actually. I also added a "random" bits test that uses
the program image itself as a source sequence for words to write/compare.
The good news is that my enhanced diagnostics now detect failures in the same physical
banks and with the same bits as those flagged by the MAINDEC diagnostic. This was a good
lesson learned: all ones / all zeros is definitely not good enough when checking this sort
of thing!
Another thing I found interesting, though, is that the "random" test *also*
found a malfunctioning bit that the address test had missed. So ones/zeros and address
isn't really good enough, either.
I'm technically curious, now, about the failure modes of these sorts of DRAMS. I
guess in addition to stuck bits, there are also potential decode fails (show up on address
test, but not ones/zeros) and some errors that have history-dependence, perhaps internal
latches (show up on random data test, but not address or ones/zeros.) I'd guess also
there might be potential for crosstalk, noise, and "fading bit" type issues as
well? Will have to see after I make the next round of repairs if there are still
additional problems that the MAINDEC flags that my simplistic diag isn't shaking out.
I've also been somewhat surprised by the level of repair needed on this memory board.
So far, I've seen 6 failed 4116 out of an array of 144 total, so about a 4% failure
rate. Is this typical for vintage 4116, or did somebody leave my poor MS11 out in a
lightning storm? :-)
Starting the CPU (i.e. 'START' switch) or an
INIT instruction will clear
the 'trap enable' bit in the MS11-L CSR.
D'oh! Yes, thanks; I may very well have mucked that up. I'll give it another try
with a little more care later today.
Which memory has this [parity halt vs trap] feature?
Hmm, I saw this at least once when researching the variety of CSR formats yesterday
morning; I'll have to see if I can dig it up again today. Might be just a fastbus
thing? It's also hinted in paragraph 7.7.7 of the older KB11-A maintenance manual
(NOT the later edition that covers both KB11-A and KB11-D):
"The semiconductor memory control EHA and EHB (enable halt) flip-flops may be set
under program control to assert SMCB PE HALT L if a parity error is detected. This input
also asserts UBCB PARITY ERR SET L, which set the console flag and halts the CPU."
This particular text is removed from the later KB11-A,D maintenance manual, and the
description there seems to imply all reported parity conditions trap directly to 114. But
there aren't any details in this section concerning processor revision/version etc.
The logic design around all this is a bit complicated, and the fact that there are
apparent discrepancies between the texts, available prints, and the actual M8106 boards I
have on hand is not heartening!
The M8106 board layout drawing (a couple of pages back
from UBCB) does show W1 -
upper left corner of the board, next to E84.
Yup. And, surprisingly, neither one of my M8106 has either a jumper or the indicated
pull-up at that location! I'll try to send a pic later. The fact that W1 exists on
the M8119 is interesting; maybe the situation is that the prints are for later revisions,
and my actual M8106 are earlier? My /45 is a very early one -- serial 154!
cheers,
--FritzM.
1:47 p.m.
Some further info on the parity halt/trap thing:
Digging again this morning for what I saw yesterday, it does look like it's a MS11
fastbus thing. The EHA/EHB bits are described in the Jan 1973 version of the 11/45 MS11
maintenance manual (table 3-12, describing the parity register), but they have been
scrubbed out by the time of the Jan 1974 printing of the same manual!
Somewhere yesterday I saw an actual register layout drawing that had these bits, but
I'm not able to find it again this afternoon so far...
--FritzM.
2:03 p.m.
Further archeology has led me to the "PDP-11/45, 11/50 MOS Memory Troublshooting
Guide", the cover letter of which is quite amusing:
"...COMPANY CONFIDENTIAL, that is take [this guide] on site but do not let the
customer read it."
"Please do not send these to the Depot for repair!! Instead send them to me with all
the necessary paper work."
"This cover letter should not be kept with the guide; ..."
Oops on that last one :-) Somebody's job was on the line here one would guess.
This document is also great for have the most information compiled in on place that
I've been able to find so far on 11/45 ECOs! I've had a tough time trying to
track those down -- they don't seem to be included among the DEC-O-LOGs archived in
the usual places, and I haven't found anybody yet who has fiches or such containing
them.
--FritzM.
1:48 p.m.
On Jan 20, 2019, at 3:56 PM, Fritz Mueller via cctalk
<cctalk at classiccmp.org> wrote:
...
I'm technically curious, now, about the failure modes of these sorts of DRAMS. I
guess in addition to stuck bits, there are also potential decode fails (show up on address
test, but not ones/zeros) and some errors that have history-dependence, perhaps internal
latches (show up on random data test, but not address or ones/zeros.) I'd guess also
there might be potential for crosstalk, noise, and "fading bit" type issues as
well? Will have to see after I make the next round of repairs if there are still
additional problems that the MAINDEC flags that my simplistic diag isn't shaking out.
There are tests for crosstalk and decode problems, they may be marked as core memory tests
but they might work anyway. There is the "no duplicate address test" which
verifies that each word is accessed by exactly one address value. That would catch decode
issues -- for example, a stuck address bit or a stuck decoder would be caught quickly.
Not quite crosstalk, but there is the famous "core heating test" which hammers
on a specific set of addresses which are near each other in the core mat, to see if the
heat generated by those rapid accesses in a small physical area cause trouble.
paul
7:11 p.m.
On January 20, 2019 at 3:56 PM Fritz Mueller via
cctalk <cctalk at classiccmp.org> wrote:
I'm technically curious, now, about the failure modes of these sorts of DRAMS. I
guess in addition to stuck bits, there are also potential decode fails (show up on address
test, but not ones/zeros) and some errors that have history-dependence, perhaps internal
latches (show up on random data test, but not address or ones/zeros.) I'd guess also
there might be potential for crosstalk, noise, and "fading bit" type issues as
well? Will have to see after I make the next round of repairs if there are still
additional problems that the MAINDEC flags that my simplistic diag isn't shaking out.
Jack Ganssle wrote a couple of articles about memory testing and failure modes. They may
be useful or at least interesting.
http://www.ganssle.com/articles/aramrom.htm
http://www.ganssle.com/articles/ramtest.htm
Will
"He may look dumb but that's just a disguise."? -- Charlie Daniels
"The names of global variables should start with? ? // "? --?https://isocpp.org
26 Jan
26 Jan
1:22 p.m.
On Jan 20, 2019, at 7:11 PM, Will Cooke via cctalk
<cctalk at classiccmp.org> wrote:
Jack Ganssle wrote a couple of articles about memory testing and failure modes...
http://www.ganssle.com/articles/aramrom.htm
http://www.ganssle.com/articles/ramtest.htm
Thanks for the pointers, Will; I found the articles practical and useful. I have posted
the latest version of my diagnostic over on my project blog at fritzm.github.io.
The latest news is that I did repairs on all the implicated DRAMs on my MS11-L, and now
both my diagnostics and MAINDEC ZQMC are passing consistently. I am also now able to
successfully run the heavyweight KT11-C exerciser MAINDEC.
There is a lingering issue with parity abort handling -- it looks as if my 11/45 is a very
early model and as such my CPU boards are early etches and apparently have not had all
ECO's applied relating to parity. My CPU hardware will console halt on UNIBUS parity
error, rather than trap to 114. So MAINDECs etc that check for this trap fail (halt) on
my machine.
Well anyway, after all this, I still have the exact same failure mode with *both* RSTS/E
and V6 Unix! Sad panda...
I don't think this is directly related to the parity abort issue, since Noel informs
me that V6 Unix doesn't care at all, and I can also see under both OS's that after
the recent repairs no parity faults are occurring on the MS11 (it has an LED for that).
The KT11 seems really solid now, too. But I suppose it could be some other early-CPU
incompato?
I think this means I have to go back to working the problem from the other end -- analyze
the core/crash files for clues on exactly what it going wrong. Paul: I'll see about
retrieving a crash file from RSTS, now that I've cleaned up the memory system issues.
cheers,
--FritzM.
27 Jan
27 Jan
1:34 p.m.
On Jan 26, 2019, at 4:22 PM, Fritz Mueller via cctalk
<cctalk at classiccmp.org> wrote:
...
I don't think this is directly related to the parity abort issue, since Noel informs
me that V6 Unix doesn't care at all, and I can also see under both OS's that after
the recent repairs no parity faults are occurring on the MS11 (it has an LED for that).
The KT11 seems really solid now, too. But I suppose it could be some other early-CPU
incompato?
I admit that I didn't read in depth, but I have the impression that RSTS tries to
identify the parity CSRs by forcing wrong parity and seeing which CSR reports the issue.
So if you have a CPU ECO issue that causes trap to 114 not to work, I don't see how
you could get RSTS to start at all.
Can you query the parity information (in defaults->memory->parity) and post the
result?
I think this means I have to go back to working the
problem from the other end -- analyze the core/crash files for clues on exactly what it
going wrong. Paul: I'll see about retrieving a crash file from RSTS, now that
I've cleaned up the memory system issues.
Great. You can feed it to a matching system in SIMH and run ANALYS on it. Did I
mentioned I looked at SDA, the interactive analyzer I wrote in Forth? It won't work
with that, since it's dependent on the data structures of a particular release. I
started that work long after RSTS V6C, so it looks like ANALYS is all you get.
paul
3:06 p.m.
On Jan 27, 2019, at 1:34 PM, Paul Koning
<paulkoning at comcast.net> wrote:
I have the impression that RSTS tries to identify the parity CSRs by forcing wrong parity
and seeing which CSR reports the issue. So if you have a CPU ECO issue that causes trap
to 114 not to work, I don't see how you could get RSTS to start at all.
Can you query the parity information (in defaults->memory->parity) and post the
result?
I'm pretty sure my machine can't take a trap to 114, because I have verified that
it is missing (at least) the backplane wire that delivers UBCB PE TRAP (L) to the M8100 as
well as R17 and the decodes at E7 and E11 on the M8100 that are necessary to deliver
constant 114 to the TV input of K1MX.
However, RSTS init defaults->memory->parity reports (correctly):
0K: 00000000 - 00757777 ( 124K) : 00
I'll keep you posted about what I find out from ANALYS!
thanks,
--FritzM.
6:27 p.m.
On Jan 27, 2019, at 3:06 PM, Fritz Mueller <fritzm
at fritzm.org> wrote:
I'll keep you posted about what I find out from ANALYS!
Posted below. I've elided the core dump here so not to spam everybody; will send
off-list.
Note that I only copied over the blocks of [0,1]CRASH.SYS, and overlaid them on a copy of
the disk image that I already had on my machine, then booted that under SIMH to run
ANALYS. Is copying over [0,1]CRASH.SYS enough, or need I take the time to image the
entire disk after the dump?
========================================================
ANALYS V06C-03 RSTS V06C-03 FRITZTS
Input <[0,1]CRASH.SYS>?
Output <KB:ANALYS.DMP>?
Crash Error Log Filename <$ERRCRS.FIL>?
Analysis of (0,1)CRASH.SYS/SIL:SY0:(0,1)RSTS.SIL
Taken on 27-Jan-89 at 06:16 PM
Certain Critical values are inaccurate
Octal Dump of Status will not be attempted
Further LOGINS were Disabled
Crash Dump Status From 27-Jan-89 at 05:37 PM, Up: 3:16
Job Who Where What Size State Run-Time Priority RTS
1 1,2 KB0 2/28K RN 2.5 0 BASIC
Run-Time Systems:
Name Ext Size Users Comments
BASIC BAC 14(16)K 1 Loading, KBM, CSZ
Busy Devices: None
Disk Structure:
Disk Open Free Cluster Errors Name Comments
DK0 0 1599 1 0 RSTS6C Pub, DLW
Small Large Jobs Hung TTY'S Errors
166 1 1/1 0 616
Message Receivers:
Name Job Msgs Max Senders
ERRLOG 0 3 40 Priv
CORE DUMP OF MONITOR
ERRDIS Full Report (All Types) taken on 27-Jan-89, 06:16 PM
Input File: $ERRCRS.FIL Output File: KB:ANALYS.DMP
Requested Date/Time Range:
First Error through Last Error
$ERRCRS.FIL will not be Zeroed upon completion
************************************************************************
PF PowerFail/Strtup Seq #1 Occurred on 27-Jan-89 at 05:34:01 PM
Detailed Description:
---------------------
R0 000000
R1 000000
R2 000000
R3 000000
R4 000000
R5 000000
Virtual PC 040222
Physical PC 00040222
Processor Status 034140
Stack Pointer 001770
(SP) 000077
(SP+2) 000000
(PC-6) 000361
(PC-4) 000000
(PC-2) 000000
(PC) 012705
CPU ID 0
CPU ERR 000000
************************************************************************
CK RTS Declared Seq #2 Occurred on 27-Jan-89 at 05:34:01 PM
User Description:
-----------------
Job Number 1
KB Number 0
Account [1,2]
Program Name
User Job Physical Addr. 00230000
User Job Size 2K
Control Parameters 200
RTS Name BASIC
RTS Physical Address 00124000
Detailed Description:
---------------------
R0 000000
R1 000000
R2 000000
R3 000000
R4 000000
R5 000000
Virtual PC 000000
Physical PC 00230000
Processor Status 174000
Stack Pointer 000374
(SP) 000001
(SP+2) 174000
(PC-6) 177777
(PC-4) 000000
(PC-2) 177777
(PC) 000000
CPU ID 0
CPU ERR 000000
************************************************************************
CK RTS Declared Seq #3 Occurred on 27-Jan-89 at 05:34:02 PM
Repeat Count: 102
User Description:
-----------------
Job Number 1
KB Number 0
Account [1,2]
Program Name
User Job Physical Addr. 00230000
User Job Size 2K
Control Parameters 200
RTS Name BASIC
RTS Physical Address 00124000
Detailed Description:
---------------------
R0 000000
R1 000000
R2 000000
R3 000000
R4 000000
R5 000000
Virtual PC 000000
Physical PC 00230000
Processor Status 174000
Stack Pointer 000374
(SP) 000002
(SP+2) 174000
(PC-6) 177777
(PC-4) 000000
(PC-2) 177777
(PC) 000000
CPU ID 0
CPU ERR 000000
Reported Date/Time Range:
27-Jan-89, 05:34:01 PM through 27-Jan-89, 05:34:02 PM
File Totals Follow:
RTS Declared -- 104 Errors Logged out of 104 Received
PowerFail/Strtup -- 1 Error Logged out of 1 Received
2 out of 30 Blocks have been used in $ERRCRS.FIL
Ready
28 Jan
28 Jan
6:50 a.m.
On Jan 27, 2019, at 9:27 PM, Fritz Mueller via cctalk
<cctalk at classiccmp.org> wrote:
Just the crash file. However...
On Jan 27, 2019, at 3:06 PM, Fritz Mueller
<fritzm at fritzm.org> wrote:
I'll keep you posted about what I find out from ANALYS!
Posted below. I've elided the core dump here so not to spam everybody; will send
off-list.
Note that I only copied over the blocks of [0,1]CRASH.SYS, and overlaid them on a copy of
the disk image that I already had on my machine, then booted that under SIMH to run
ANALYS. Is copying over [0,1]CRASH.SYS enough, or need I take the time to image the
entire disk after the dump?
========================================================
ANALYS V06C-03 RSTS V06C-03 FRITZTS
Input <[0,1]CRASH.SYS>?
Output <KB:ANALYS.DMP>?
Crash Error Log Filename <$ERRCRS.FIL>?
Analysis of (0,1)CRASH.SYS/SIL:SY0:(0,1)RSTS.SIL
Taken on 27-Jan-89 at 06:16 PM
Certain Critical values are inaccurate
Octal Dump of Status will not be attempted
That's puzzling. How did you generate the crash dump? I remember that message but
not what causes it (if I ever knew).
Perhaps it wants to read the defaults. If so, copying the monitor SIL file would take
care of that.
Is the system on which you're running this ANALYS run using the same monitor as the
failed system? Maybe that's the issue. If ANALYS is using current monitor offsets to
find data structures in the dump file, that won't work if you're analyzing a dump
from a different system.
paul
7:06 a.m.
On Jan 28, 2019, at 9:50 AM, Paul Koning via cctalk
<cctalk at classiccmp.org> wrote:
Looking at the 10.1 source of ANALYS: it defaults to the installed SIL (of the running
system). So if your dump came from a different system, that won't be correct. In
V10.1 there is a switch to supply the correct SIL name. You might try that for your run.
Specify "/SIL:yourmonitor.SIL" on the "input" prompt. Or, if needed,
"crashfile/SIL:monitor" if you need to supply a different dump file name as
well.
So you do need the correct SIL file. As-built should be sufficient (I don't believe
it cares about the defaults, so the stuff INIT saves into the DEFALT section of the
monitor SIL should not be needed).
paul
On Jan 27, 2019, at 9:27 PM, Fritz Mueller via
cctalk <cctalk at classiccmp.org> wrote:
...
ANALYS V06C-03 RSTS V06C-03 FRITZTS
Input <[0,1]CRASH.SYS>?
Output <KB:ANALYS.DMP>?
Crash Error Log Filename <$ERRCRS.FIL>?
Analysis of (0,1)CRASH.SYS/SIL:SY0:(0,1)RSTS.SIL
Taken on 27-Jan-89 at 06:16 PM
Certain Critical values are inaccurate
Octal Dump of Status will not be attempted
That's puzzling. How did you generate the crash dump? I remember that message but
not what causes it (if I ever knew).
Perhaps it wants to read the defaults. If so, copying the monitor SIL file would take
care of that.
Is the system on which you're running this ANALYS run using the same monitor as the
failed system? Maybe that's the issue. If ANALYS is using current monitor offsets to
find data structures in the dump file, that won't work if you're analyzing a dump
from a different system.
8:06 a.m.
On Jan 28, 2019, at 7:06 AM, Paul Koning
<paulkoning at comcast.net> wrote:
So you do need the correct SIL file. As-built should be sufficient (I don't believe
it cares about the defaults, so the stuff INIT saves into the DEFALT section of the
monitor SIL should not be needed).
Well, it *should* be the same -- I just overwrote the sectors for CRASH.SYS on a copy of
the image file I used to create the RK05.
I'll guess I'll try to transfer the full image back from the machine and see if
that fares any better...
thanks,
--FritzM.
2155
days inactive
2163
days old
12 comments
4 participants
participants (4)
-
fritzm＠fritzm.org -
jnc＠mercury.lcs.mit.edu -
paulkoning＠comcast.net -
wrcooke＠wrcooke.net