It is not clear at this time if it is possible to do both: a) ensure the private key never leaves the cluster, and b) ensure zero downtime for authentication to the API while transitioning to this new key. Deleting the next-bound-service-account-signing-key will trigger it to regenerate a new one. It then appears to start automatically rolling out to the cluster. It is therefore possible for pods to start using tokens signed by the new key before it is added to the issuer file, which would cause authentication failures. For example, a reboot of all nodes would cause all pods to get new tokens using the new keys.

The documentation in my current PR ensures the private key never leaves the cluster while assuming a risk of the authentication failing. Ideally, we want the cluster to generate a new keypair in a way that would allow us to add the public key to the issuer file before the cluster starts using it. However, adding such functionality is beyond the scope of the cloud-credential-operator.

I discussion with rhn-support-sdodson, he identified that one requirement of this doc should probably be to not have the next-bound-service-account-signing-key public key leave the OpenShift cluster. This will require changing the procedure in the current PR to change from creating a key-pair locally to deleting the next-bound-service-account-signing-key secret and downloading only the public key after it is regenerated by the cluster. The challenge here is that, it appears the next-bound-service-account-signing-key is partially deployed to the cluster automatically after the secret is modified. As a result, this may cause issues with authentication until the new key is added to the issuer's keys.json. I believe an example of this can be seen in OCPBUGS-31433.