It should be possible to detect if the server is running EE 11 and have a subsystem step fail the boot if the SM is enabled.

This seems like something the ee subsystem could do. Probably it can try and load jakarta.annotation.ManagedBean and if it can't then check for the SM being enabled (or vice versa).

This can't be foolproof, and I don't think it needs to be. Documentation is the primary way to address that EE 11 and the SM don't mix. But we can help by detecting the common case. And to the extent the ee subsystem can't work with jakarta.annotations, then we know anything that needs the ee subsystem also can't work. Which is likely most anything 'EE'.

Context: an element of the EE 11 release plan was removal of SM support, and AFAIK all specs that have any reference to SM have removed it, and in cases where deprecated-for-removal SM types appeared in APIs they have been removed. The standard API artifacts from Jakarta may or may not have removed internal uses of SM types (e.g. privileged blocks). Accordingly, some impl projects that are closely tied to those spec versions are likely beginning to remove SM support. For example, I know Weld made at least one privileged block removal in a 6.0 beta. In this environment WildFly can't let users believe that running with an SM will work well, and in the security area 'maybe' isn't a good answer, so we should reject its use.