Cool. I don't even know what the heck an entry
mask is.
You might want to read up on VAX machine/assembly language, then,
before you try to dig too much deeper into picking apart this binary
blob. You can learn a lot without, it's true, but it will help
substantially.
As for entry masks in particular...the VAX has two different kinds of
routine calls.
There's the simple sort, which call with BSBB, BSBW, or JSB, and return
with RSB. These are pretty close to the kind of call most RISC
hardware has, a call being "do a jump, but push what would otherwise be
the next instruction's PC on the stack first" and a return being "pull
an address off the stack and jump to it". (The differences among the
three call instructions is how the target instruction is addressed.)
I'll call these `JSB calls'. Arguments, if any, can be passed pretty
much anywhere: in registers, on the stack, in fixed locations, whatever
the caller and the callee agree on.
There's also the complex sort, which call with CALLS or CALLG and
return with RET. These not only transfer flow-of-control but do a
bunch of other stuff, too, notably meddling with AP and FP. They take
the argument list in memory - on the stack for CALLS, anywhere in
memory for CALLG. CALLS also removes the args from the stack
automatically on return - the stack frame created by CALL[SG] includes
a bit that tells RET whether to pop the arglist. I'll call these
`CALLx calls'.
JSB calls are substantially cheaper, but provide less assist. Most VAX
toolchain and development software (compilers, debuggers, etc) assumes
CALLx calls; in particular, stack frames are generally assumed to look
like what CALLS/CALLG generate.
As for entry masks, one of the things CALLS/CALLG do is save
callee-used registers (and potentially, do a few other things, such as
turning integer-overflow traps on or off). The entry mask says what
registers to save on entry (and restore on exit); it's a 16-bit
bitmask. (There are a few bits that would otherwise correspond to
registers that it doesn't make any sense to save and restore, such as
PC; these are the bits that control things like integer overflow trap
enables.) The entry point address passed to CALLS/CALLG is that of the
entry mask; execution starts two bytes further along. When running for
the VAX, the e command disassembles two bytes as an entry mask (there's
a note late in the README about this).
There are two occurences of "0c 00 10 20" in
the code, at 200468b2
and 20046932.
Each of those appears to be real. I'm currently annotating every
mfpr/mtpr instruction, or at least those with constant register number
operands, as to which register it's referring to; once that's done I'll
search out all the 201000xx references and annotate them. I'll make
the disassembler save file and the resulting text version available....
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML mouse at
rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B