-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.17.z, 4.19.0
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
Moderate
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Machine-api-controller does not reconcile automatically after a vCenter password change.
Version-Release number of selected component (if applicable):
Openshift Container Platform 4.17.31 and 4.19.0
How reproducible:
All the time
Steps to Reproduce:
Followed Red Hat Openshift Documentation Section for vCenter: "Rotating cloud provider credentials manually":
- Start with a fresh cluster.
- Change vCenter Password.
- Change `vsphere-creds` secret to the new password.
- (Optional just to follow the documentation) Force a KCM redeploy with:
oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' --type=merge
Actual results:
The machine-controller container logs shows a `401 Unauthorized` error that only stops after a manual delete of the pod:
$ oc logs -n openshift-machine-api -f machine-api-controllers-78586bb75-b4zpp -c machine-controller [...] E0701 08:25:42.021995 1 actuator.go:63] "master-node" error: master-node: reconciler failed to Update machine: failed to reconcile tags: POST https://VCENTER-URL/rest/com/vmware/cis/session: 401 Unauthorized
Expected results:
The machine-api operator should restart the pod to update the new credentials and evade possible vCenter account lockups due to security policies.
Workaround:
An identified workaround is to manually delete the pod so it restarts with the new credentials:
oc delete pod -n openshift-machine-api -l k8s-app=controller