Narrative

If a user enables a feature during a rolling update, for example enabling/disabling persistent-user-sesssions, it’s possible for an inconsistent state to exist between the Keycloak nodes with/without the feature enabled #29561. Whilst discussing mitigation strategies it became clear that Keycloak currently has no mechanism in the server or operator to guard against rolling updates of incompatible configurations. Consequently, it’s possible for users to rollout configurations that will result in undefined behaviour during and/or after the update has completed. Best case scenario the incompatibilities lead to exceptions that make the underlying issue clear, however in the worst case users may run into an untested state that leads to subtle race conditions over time that lead to customer cases that are very hard to identify and debug. 

It should be possible for individual Features to explicitly declare whether a configuration change is compatible with a rolling upgrade and require the cluster to be shutdown during upgrades if it is not.

RHBK-2130 added a framework based around the update-compatibility command and CompatibilityMetadataProvider allowing individual components within Keycloak to declare whether changes can be safely deployed as part of a rolling upgrade.

Currently CompatibilityMetadataProvider implementations are only provided for the Keycloak and Infinispan versions, with rolling upgrades only allowed if these remain the same.

Value proposition

• Preflight check on the CLI: Ensuring that configuration updates allow a rolling update will provide customers quicker feedback that should help identify issues with upgrades in development before they are rolled out into production.
• Safe operations with Keycloak Operator: Preventing known incompatible mixed configuration states will simplify debugging of RH customer support cases, as users should be prevented from entering untested states. 

Acceptance criteria

• Implement CompatibilityMetadataProvider for all supported Features that might prevent a rolling update
• Implement CompatibilityMetadataProvider for CLI args not encapsulated by a specific feature, but which could impact compatibility.
• Document features/CLI args that can be enabled/disabled with rolling updates
• Document best practices/process for ensuring compatibility in order to enable other Keycloak teams to correctly leverage these capabilities going forward.
• Default to rolling if there is no obvious evidence against it. Try to catch the 20% of the incompatible options that annoy 80% of the users.