Goal:

Achieving true zero downtime is tricky because moving new passwords for different servers may require specialized steps, potentially involving data plane deployments and restarting processes, which may cause downtime if systems cannot reread configs without service disruption.

The infrastructure for managing credentials is provided in Keystone, but service-specific steps are necessary for zero or minimal downtime. It is up to the different RAs to review their services and ensure they can use application credentials and the automatic rotation feature without downtime, possibly requiring documentation or programmatic changes.

Details

Zero Downtime Password Rotation (ZDPR) provides an alternative to Keystone service users + passwords by using Application Credentials (AppCreds) for service-to-service authentication.

• KeystoneApplicationCredential CRDs are created by openstack-operator based on OpenStackControlPlane.spec.applicationCredential (global + per-service overrides).

• keystone-operator watches these CRDs, creates/rotates AppCreds in Keystone, and stores AC_ID / AC_SECRET in a Kube Secret (ac-<service>-secret).

• Each service-operator consumes that secret and renders auth_type = v3applicationcredential. If the secret is missing or malformed, operators fall back to password auth.

Rotation is driven entirely by the keystone-operator using expirationDays and gracePeriodDays. During the grace window, both the old and new AppCreds are valid. This provides a window during which any steps that the <service> needs to take to start using the new AppCred can be performed. This may include, for instance, a dataplane deployment.

This ticket is for <DFG> to determine the steps needed to adopt this mechanism for <service>.

Specifically:

• Review the ZDPR Design Doc and relevant service patch (if already available)
• Confirm that the <service>- operator watches the AppCred Secret (ac-<service>-secret)
  Renders auth_type=v3applicationcredential and uses AC_ID / AC_SECRET
• Triggers a rolling update of pods when the AppCred Secret changes
• Falls back cleanly to password auth when the AppCred Secret is missing, or incomplete
• Identify that all <service> components using Keystone for auth are using app cred authentication when available and are able to switch between both auth methods
• Identify the steps required for all <service> components to use the new appCred. Ideally, this could happen automatically with no downtime. In the worst case, this will require a manual procedure that should be documented.
• Confirm that redeploying with new credentials behaves as expected and is compatible with the grace period model

Relevant links:
https://docs.google.com/document/d/11myws61iGFfRtAqGGxTcfTbmsBkXs3t_PSZRUybtTss/edit?tab=t.0

https://github.com/openstack-k8s-operators/keystone-operator/pull/567

[WIP] https://github.com/openstack-k8s-operators/openstack-operator/pull/1430

Acceptance Criteria:

• Ensure the service can use application credentials and the automatic rotation(ZDPR) feature without downtime.
• As the outcome of this ticket, please provide information whether the service is able to receive application credential implementation with current patches provided, and if not, what are the steps that need to be done in order to do so.

Open questions:

<-- Capture any open questions and resolutions relating to the goal/acceptance criteria - remove these notes before saving -->

• Any additional details, questions or decisions that need to be made/addressed
• …