Uploaded image for project: 'Red Hat build of Keycloak'
  1. Red Hat build of Keycloak
  2. RHBK-3243

MigrationModel duplicate entry on Recreate Upgrade in Cluster with 2+ nodes [GHI#39866]

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      Before reporting an issue

      [x] I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

      Area

      No response

      Describe the bug

      When upgrading to a newer Keycloak(26.1.3 -> 26.2.3) in recreate and Cluster with more than 1 node, the DefaultMigrationManager runs migrate in each node. Specifically it runs
      model.setStoredVersion(currentVersion.toString()); in more than 1 node.
      This results in MigrationModelEntity table to have more than 1 row on the same version and the migration was ran more than once.
      Also a custom theme was used. This affects the login(welcome/ui) pages UI to not load properly due to 404 in UI static resources fetching. In the requests with the 404, it can be seen that the Id from the MigrationModelEntity table is used so when trying to change it to the other id it works(if the request goes to the same node that created this row).
      Each node is using a different Id from the MigrationModelEntity table in MigrationModelAdapter.latest

      Maybe this is a missing configuration or something of this sort, but in previous upgrades(to different verisons) of same scenario it wasn't reproduced.

      Version

      26.2.3

      Regression

      [x] The issue is a regression

      Expected behavior

      MigrationModelEntity table to have a single row for each version.
      When upgrading with Recreate + Cluster with more than 1 node, should run these migrations once and should be aware to handle this case of race.

      Actual behavior

      MigrationModelEntity table have more than 1 row for the version you upgraded to.
      When upgrading with Recreate + Cluster with more than 1 node, runs these migrations more than once without logic of locking.

      How to Reproduce?

      • Set up Keycloak in version 26.1.3 or lower in Cluster with more than 1 node(can use 4 for easier reproduction)
      • Start an upgrade of the environment to 26.2.3, Should be Recreate upgrade(add Debug logs for better visibility).
      • Once the upgrade is completed, can be seen in MigrationModelEntity table that we have more than 1 row for 26.2.3 version
        As this is some kind of race condition, can be a bit of flaky reproduction

      Anything else?

      Logs from 2 different pods:

      local-keycloak-56bdb855f8-gc5jh local-keycloak 2025-05-19 13:00:41,596 INFO  [org.keycloak.storage.datastore.DefaultMigrationManager] (main) Migrating older model to 26.2.0 
      local-keycloak-56bdb855f8-mw2r8 local-keycloak 2025-05-19 13:00:37,876 DEBUG [org.keycloak.quarkus.runtime.storage.database.jpa.QuarkusJpaConnectionProviderFactory] (main) Calling migrateModel 
      local-keycloak-56bdb855f8-mw2r8 local-keycloak 2025-05-19 13:00:38,580 INFO  [org.keycloak.storage.datastore.DefaultMigrationManager] (main) Migrating older model to 26.2.0
      

              Unassigned Unassigned
              pvlha Pavel Vlha
              Keycloak Cloud Native
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: