Before reporting an issue

[X] I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

Area

storage

Describe the bug

Keycloak preloads remote sessions from a remote Infinispan.

When analyzing concurrency tests, where some nodes arrive and leave, this can lead to a putAll() operation to deadlock as seen in the test log.

https://github.com/keycloak/keycloak/blob/833bf9864356abe6f2c9f672edf1438b8635f48c/model/infinispan/src/main/java/org/keycloak/models/sessions/infinispan/remotestore/RemoteCacheSessionsLoader.java#L144

Given the latest setup configurations in the KCB project, instead of preloading the entries Keycloak could rely on the Infinispan fetching the entries from the remote store automatically when needed, saving startup time.

Version

main

Expected behavior

Use the remote store as persistence store as described in the KCB project.

Actual behavior

With nodes coming and going, putAll might touch the same keys, and then deadlock.

How to Reproduce?

Happens rarely in tests, but when it happens it deadlocks. Full log attached, here the relevant part: logs.txt.gz

```
jgroups-17,node-24(site-id=site-2) (TIMED_WAITING): locked on java.util.concurrent.CompletableFuture$Signaller@2b0c90de waiting for 25160 ms blocked for 0 ms
at java.base@17.0.9/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@17.0.9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
at java.base@17.0.9/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1866)
at java.base@17.0.9/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
at java.base@17.0.9/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
at java.base@17.0.9/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1939)
at java.base@17.0.9/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
at app//org.infinispan.commons.util.concurrent.CompletableFutures.await(CompletableFutures.java:130)
at app//org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:36)
at app//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:249)
at app//org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:323)
at app//org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:111)
at app//org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:93)
at app//org.infinispan.cache.impl.CacheImpl.putAll(CacheImpl.java:1361)
at app//org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:562)
at app//org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:567)
at app//org.infinispan.cache.impl.EncoderCache.putAll(EncoderCache.java:792)
at app//org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:144)
at app//org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:47)
at app//org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker$1.run(SessionInitializerWorker.java:71)
at app//org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:262)
at app//org.keycloak.models.utils.KeycloakModelUtils$$Lambda$1281/0x00007f5254689dc0.run(Unknown Source)
at app//org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:384)
at app//org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:261)
at app//org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:67)
at app//org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:34)
at app//org.infinispan.manager.impl.ReplicableManagerFunctionCommand.invokeAsync(ReplicableManagerFunctionCommand.java:45)
at app//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ReplicableCommandRunner.run(GlobalInboundInvocationHandler.java:256)
at app//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleReplicableCommand(GlobalInboundInvocationHandler.java:175)
at app//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:114)
at app//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1548)
at app//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1476)
at app//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$1(JGroupsTransport.java:1695)
at app//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks$$Lambda$2528/0x00007f5254ed8f80.accept(Unknown Source)
at java.base@17.0.9/java.lang.Iterable.forEach(Iterable.java:75)
at app//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1687)
at app//org.jgroups.JChannel.up(JChannel.java:749)
at app//org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:939)
at app//org.jgroups.protocols.FRAG2.up(FRAG2.java:161)
at app//org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
at app//org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
at app//org.jgroups.protocols.pbcast.GMS.up(GMS.java:859)
at app//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
```

Anything else?

Implementation idea:

• Remove all pre-loading of remote sessions, possibly hide it behind a flag so people can enable it for one or two versions of Keycloak
• While testing this, also revisit RemoteCacheSessionListener to instead of fetching changed entries from the remote cache and putting them into Keycloak, it would just forget the entries in the local cache.