What about the fuzzball? I don't know much about
it, but IIRC it's free.
I don' know much about it either, and I've not looked at it for years,
I just remembered that there's also TSX-11+. That was originally based on
RT too, but they got tired of paying the DEC license fees and just rewrote
all the RT11 bits that were necessary to make it work. I think Lyle has
secured permission for hobbyists to use TSX, but I don't know the terms.
Bob