No, RSTS is RDS-1; it's a file structure all its
own, totally
unrelated to any other PDP-11 OS. (It's a distant relative of the
TSS-8 file system, though.)
The trick with interleaving ODS-2 and ISO9660 is to use the first
64 blocks or so that ISO9660 leaves undefined as the "in" for ODS-2.
Then you get N blocks of ISO9660 data and meta-data. The ISO9660
blocks get collected into BADBLK.SYS in ODS-2.
So the two filesystems side-step each other. If you want to share
data (which I did when I was building CDs with PDFs and index HTMLs)
then that is possible, but slightly harder.
As long as RDS-1 does not absolutely need block 64 and some way
onwards, then it may be possible to logically interleave it and
ISO9660.
I've often wondered whether it would be possible to build a
disk that could boot VAX, Alpha (OpenVMS) and PDP-11 (RSX-11).
It would be an interesting exercise to try sometime, and
probably a lot easier to test now that we have SIMH!
Antonio