The issue came up after upgrading from Wildfly 14.0.1 to 17.0.1 (also tested 19.1.0) with JDK 1.8 and OpenJDK 1.11.
When using the Nashorn scripting engine inside of a deployed WAR file, it will fail with random "ClassNotFound" exceptions, however, at the beginning of the log the following can be seen:

ERROR [stderr] (Thread-213) java.lang.LinkageError: loader 'platform' attempted duplicate abstract class definition for jdk.nashorn.internal.codegen.types.Type. (jdk.nashorn.internal.codegen.types.Type is in module jdk.scripting.nashorn of loader 'platform')

followed for example by this exception:
ERROR [stderr] (Thread-205) Exception in thread "Thread-205" java.lang.NoClassDefFoundError: Could not initialize class jdk.nashorn.internal.runtime.linker.Bootstrap
ERROR [stderr] (Thread-205) at jdk.scripting.nashorn/jdk.nashorn.internal.runtime.Context.<init>(Context.java:644)
ERROR [stderr] (Thread-205) at jdk.scripting.nashorn/jdk.nashorn.internal.runtime.Context.<init>(Context.java:569)
ERROR [stderr] (Thread-205) at jdk.scripting.nashorn/jdk.nashorn.api.scripting.NashornScriptEngine$1.run(NashornScriptEngine.java:132)
ERROR [stderr] (Thread-205) at jdk.scripting.nashorn/jdk.nashorn.api.scripting.NashornScriptEngine$1.run(NashornScriptEngine.java:128)
ERROR [stderr] (Thread-205) at java.base/java.security.AccessController.doPrivileged(Native Method)
ERROR [stderr] (Thread-205) at jdk.scripting.nashorn/jdk.nashorn.api.scripting.NashornScriptEngine.<init>(NashornScriptEngine.java:128)
ERROR [stderr] (Thread-205) at jdk.scripting.nashorn/jdk.nashorn.api.scripting.NashornScriptEngineFactory.getScriptEngine(NashornScriptEngineFactory.java:154)
ERROR [stderr] (Thread-205) at java.scripting/javax.script.ScriptEngineManager.getEngineByName(ScriptEngineManager.java:241)

I have checked the base module.xml file for the java scripting exports and found it to be identical between all 3 Wildfly versions. Could it be possible that the class loader really tries to load the same class twice? The app has multiple threads and many of them are using the ScriptEngine, so from my perspective I cannot say if the initialization is happening multiple times at once.
The failing statement is:
ScriptEngine engine = manager.getEngineByExtension("js");