If should be possible to detect if the server is running EE and have a subsystem step fail the boot if the SM is enabled.

This seems like something the ee subsystem could do. Probably it can try and load jakarta.annotation.ManagedBean and if it can't check for the SM being enabled (or vice versa).

This can't be foolproof, and I don't think it needs to be. Documentation is the primary way to address the EE 11 and the SM don't mix. But we can help by detecting the common case. And to the extent the ee subsystem can't work with jakarta.annotations, then we know anything that needs the ee subsystem also can't work. Which is likely most anything 'EE'.