While switching from a Java 8 based deployment to Java 11 of RHSSO, we noticed an increase in memory usage and that some objects were long lived.

Our RHSSO deployment makes use of script mappers. We ultimately have determined that the Nashorn engine in Java 11 is creating many weak/soft references and are not culled with mixed GC as they were with Java 8. This ultimately results in the application being required to perform a Full GC or otherwise risk memory exhaustion.

There are also some other changes to Nashorn in Java 11 that also create additional classloaders and classes with each instantiation of the engine (a factor of 3).

Additionally, we determined this is also an issue to some degree with Java 8 - it went unnoticed previously as the Full GC cycles did not occur (Java 8 was able to reclaim the space during mixed gc), but could be observed by the rather fast allocation and promotion of new objects from young to old gen, but subsequently cleared by mixed GC. By turning off the script mappers, we could observe the young and old gen growth rates slowed significantly.

There are some recommendations about sharing the ScriptEngine created initially (I believe RHSSO/Keycloak creates it with each request), which could reduce some of the resource consumption. I am unsure if that would solve the weak/soft reference issues in Java 11 as those could be from one of many bugs open for Java 11 + Nashorn.