-
Bug
-
Resolution: Done
-
Undefined
-
None
-
False
-
-
False
-
-
Before reporting an issue
[x] I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.
Area
infinispan
Describe the bug
Hello,
I am working on a standalone Keycloak node in production mode (run with "start" and not "start-dev"). Since upgrading from version 23.0.7 (JDK 17) to 26.0.8 (JDK 21) then to 26.1.4 (JDK 21), I am having thread accumulation issues.
I do indeed see a thread accumulation with this stack trace:
"at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method) at java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:221) at java.util.concurrent.CompletableFuture$Signaller.block(java.base@21.0.6/CompletableFuture.java:1864) at org.keycloak.models.sessions.infinispan.changes.JpaChangesPerformer.applyChanges(JpaChangesPerformer.java:109) at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsChangelogBasedTransaction$$Lambda/0x00007f7798135aa8.accept(Unknown Source) at java.lang.Iterable.forEach(java.base@21.0.6/Iterable.java:75) at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsChangelogBasedTransaction.commitImpl(PersistentSessionsChangelogBasedTransaction.java:222) at org.keycloak.models.AbstractKeycloakTransaction.commit(AbstractKeycloakTransaction.java:46) at org.keycloak.services.DefaultKeycloakTransactionManager.lambda$commitWithTracing$0(DefaultKeycloakTransactionManager.java:169) at org.keycloak.services.DefaultKeycloakTransactionManager$$Lambda/0x00007f7797f31b70.accept(Unknown Source) at org.keycloak.quarkus.runtime.tracing.OTelTracingProvider.trace(OTelTracingProvider.java:117) at org.keycloak.tracing.TracingProvider.trace(TracingProvider.java:174) at org.keycloak.services.DefaultKeycloakTransactionManager.commitWithTracing(DefaultKeycloakTransactionManager.java:168) at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:146) at org.keycloak.services.DefaultKeycloakSession.closeTransactionManager(DefaultKeycloakSession.java:396) at org.keycloak.services.DefaultKeycloakSession.close(DefaultKeycloakSession.java:361) at org.keycloak.models.KeycloakBeanProducer_ProducerMethod_getKeycloakSession_XoSEUTXOsE3bpqXlGMAykCiECUM_ClientProxy.close(Unknown Source) at org.keycloak.quarkus.runtime.transaction.TransactionalSessionHandler.close(TransactionalSessionHandler.java:60) at org.keycloak.quarkus.runtime.integration.jaxrs.CloseSessionFilter.closeSession(CloseSessionFilter.java:67) at org.keycloak.quarkus.runtime.integration.jaxrs.CloseSessionFilter.filter(CloseSessionFilter.java:63) at org.jboss.resteasy.reactive.server.handlers.ResourceResponseFilterHandler.handle(ResourceResponseFilterHandler.java:25) at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:150) at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147) at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:635) at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516) at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521) at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11) at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)"
It seems to be an Infinispan problem but I'm not sure. When using a standalone node, do I have to specify "cache=local" at startup from now on ?
I'm getting this issue with one of my environment but not all my standalone environments so I'm not able to reproduce it easily.
If it can help, I have metrics and telemetry features enabled, but not histograms metrics :
health-enabled=true metrics-enabled=true cache-metrics-histograms-enabled=false http-metrics-histograms-enabled=false tracing-enabled=true tracing-endpoint=$PROM_SERVER tracing-service-name=$SERIVCE_NAME
Version
26.1.4
Regression
[x] The issue is a regression
Expected behavior
Not having a thread accumulation
Actual behavior
JVM threads only increase at night and at some point I can log in with my password, but on the OTP login page I get a timeout. If I don't restart the service it becomes completely unavailable after a while.
How to Reproduce?
Standalone node started with "start" command without 'cache' configuration.
Anything else?
No response
- links to
