Uploaded image for project: 'Red Hat build of Keycloak'
  1. Red Hat build of Keycloak
  2. RHBK-3592

First JDBC_PING initialization happens in the JTA transaction context [GHI#43335]

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      Before reporting an issue

      [x] I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

      Area

      infinispan

      Describe the bug

      Due to this the statements for JDBC_PING are bound to the current JTA transaction that is about to migrate the realm. This then leads to locks on the table, that eventually leads to deadlocks.

      Version

      main

      Regression

      [ ] The issue is a regression

      Expected behavior

      The JTA should be suspended while initializing JDBC_PING

      Actual behavior

      The JTA is active. This leads to the following behavior - possibly only when a migration happens:

      • Migration is starting
      • JGroups is initializing
      • JGroups locks the tables
      • Deadlock occurs, transaction is rolled back
      • Migration fails.

      Possibly causing problems in setups like https://github.com/keycloak/keycloak/discussions/43194#discussioncomment-14629927

      How to Reproduce?

      Running KC 26.2 in parallel with the KC 26.4. KC 26.2 will run the clear command which will lead to a deadlock with multiple deletes of the KC 26.4 server.

      Note that there must be some migration that have not been run. So possibly delete the 26.4.x or the 999.0.0 entry from the MIGRATION_MODEL table.

      Similar setup

      Similar problem seems to happen when upgrading from 26.3.0 to 26.3.5. 

      WARNING: With HTTPS not enabled, `proxy-headers` unset, and `hostname-strict=false`, the server is running in an insecure context. Secure contexts are required for full functionality, including cross-origin cookies. Also if you are using a proxy, requests from the proxy to the server will fail CORS checks with 403s because the wrong origin will be determined. Make sure `proxy-headers` are configured properly.
2025-10-13 10:22:14,433 INFO [org.keycloak.spi.infinispan.impl.embedded.JGroupsConfigurator] (main) JGroups JDBC_PING discovery enabled.
2025-10-13 10:22:14,450 INFO [org.keycloak.spi.infinispan.impl.embedded.JGroupsConfigurator] (main) JGroups Encryption enabled (mTLS).
2025-10-13 10:22:14,553 INFO [org.infinispan.CONTAINER] (main) Virtual threads support enabled
2025-10-13 10:22:14,622 INFO [org.keycloak.jgroups.certificates.CertificateReloadManager] (main) Starting JGroups certificate reload manager
2025-10-13 10:22:14,740 INFO [org.infinispan.CONTAINER] (main) ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller'
2025-10-13 10:22:14,935 INFO [org.infinispan.CLUSTER] (main) ISPN000078: Starting JGroups channel `ISPN` with stack `jdbc-ping`
2025-10-13 10:22:14,936 INFO [org.jgroups.JChannel] (main) local_addr: 00000000-0000-0000-0000-000000000006, name: keycloak-3-25727
2025-10-13 10:22:14,946 INFO [org.jgroups.protocols.FD_SOCK2] (main) server listening on *:57800
2025-10-13 10:22:16,959 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 0
2025-10-13 10:22:18,966 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 1
2025-10-13 10:22:20,972 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 2
2025-10-13 10:22:22,979 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 3
2025-10-13 10:22:24,985 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 4
2025-10-13 10:22:26,991 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 5
2025-10-13 10:22:28,997 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 6
2025-10-13 10:22:31,003 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 7
2025-10-13 10:22:33,009 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 8
2025-10-13 10:22:35,014 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: JOIN(keycloak-3-25727) sent to keycloak-0-51280 timed out (after 2000 ms), on try 9
2025-10-13 10:22:35,015 WARN [org.jgroups.protocols.pbcast.GMS] (main) keycloak-3-25727: too many JOIN attempts (10): becoming singleton
2025-10-13 10:22:35,043 INFO [org.infinispan.CLUSTER] (main) ISPN000094: Received new cluster view for channel ISPN: [keycloak-3-25727|0] (1) [keycloak-3-25727]
2025-10-13 10:22:35,045 INFO [org.keycloak.jgroups.certificates.CertificateReloadManager] (main) Reloading JGroups Certificate
2025-10-13 10:22:35,088 INFO [org.infinispan.CLUSTER] (main) ISPN000079: Channel `ISPN` local address is `keycloak-3-25727`, physical addresses are `[10.130.0.124:7800]`
2025-10-13 10:22:35,458 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-3-25727, Site name: null
2025-10-13 10:22:36,219 INFO [io.quarkus] (main) Keycloak 26.3.5 on JVM (powered by Quarkus 3.20.3) started in 25.986s. Listening on: http://0.0.0.0:8080. Management interface listening on http://0.0.0.0:9000.
2025-10-13 10:22:36,219 INFO [io.quarkus] (main) Profile prod activated.
2025-10-13 10:22:36,219 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, narayana-jta, opentelemetry, reactive-routes, rest, rest-jackson, smallrye-context-propagation, smallrye-health, vertx]
2025-10-13 10:22:36,244 INFO [org.infinispan.CLUSTER] (main) ISPN000080: Disconnecting JGroups channel `ISPN`
2025-10-13 10:22:36,253 INFO [org.keycloak.jgroups.certificates.CertificateReloadManager] (main) Stopping JGroups certificate reload manager
2025-10-13 10:22:36,255 INFO [com.arjuna.ats.jbossatx] (main) ARJUNA032014: Stopping transaction recovery manager
2025-10-13 10:22:36,267 INFO [io.quarkus] (main) Keycloak stopped in 0.043s

      Anything else?

      I'll create a PR

              Unassigned Unassigned
              pvlha Pavel Vlha
              Keycloak SRE
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: