Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-58399

Machine API controller doesn't use new vCenter credentials

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem: 

      Machine-api-controller does not reconcile automatically after a vCenter password change.

      Version-Release number of selected component (if applicable):

      Openshift Container Platform 4.17.31 and 4.19.0

      How reproducible:

      All the time

      Steps to Reproduce:

      Followed Red Hat Openshift Documentation Section for vCenter: "Rotating cloud provider credentials manually":

      1. Start with a fresh cluster.
      2. Change vCenter Password.
      3. Change `vsphere-creds` secret to the new password.
      4. (Optional just to follow the documentation) Force a KCM redeploy with:
        oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' --type=merge

      Actual results: 

      The machine-controller container logs shows a `401 Unauthorized` error that only stops after a manual delete of the pod:

       

      $ oc logs -n openshift-machine-api -f machine-api-controllers-78586bb75-b4zpp -c machine-controller 
      [...]
      E0701 08:25:42.021995       1 actuator.go:63] "master-node" error: master-node: reconciler failed to Update machine: failed to reconcile tags: POST https://VCENTER-URL/rest/com/vmware/cis/session: 401 Unauthorized

       

      Expected results: 

      The machine-api operator should restart the pod to update the new credentials and evade possible vCenter account lockups due to security policies.

      Workaround:

      An identified workaround is to manually delete the pod so it restarts with the new credentials:

       

      oc delete pod -n openshift-machine-api -l k8s-app=controller

       

              raryan@redhat.com Rachel Ryan
              rhn-support-tdomingu Tomas Dominguez
              None
              None
              Zhaohua Sun Zhaohua Sun
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: