1. Problem description:

 Critical issue of heap usage continuously growing and exhausting available heap memory in production. As you can see in below heap dump org.infinispan.container.impl.DefaultDataContainer is growing as the time passes and not garbage collected by GC. This increase in heap happening after upgrading from Wildfly 10.1 to Wildfly 18.0.1. There is no change in JDK in both cases ( Wildfly 10.1 and Wildfly 18.0.1).

 2. Web application environment

• OS: RHEL 7.5
• Wildfly 18.0.1
• JDK 1.8
• Wildfly is being run in cluster mode
• Integrated with Keycloak 3.4.3 for SSO ( SAML)
• Enabled Wildfly clustering mode
• G1GC garbage collector used. And 20gigs of heap allocated ( -Xmx)
• Environment of Web App on Wildfly 10.1 is same ( except Wildfly 18) with same hardware
• Web application uses Spring web framework and security

 3. Heap dump analysis

• We have few web users logging in and every second and external application consuming few API’s exposed by same application.
• As you can see infinispan.container.impl.DefaultDataContainer Has already grown 14gigs.
• Please note that web application does not use infinispan directly.
• The heap does not grow in wildfly 10.1 and its normal ( garbage gets collected and size gets reduced after GC)

Please find the snapshot as attached.