Major Description of problem:
The storage CO goes degraded complaining about incorrect credentials for vcenter login
Version-Release number of selected component (if applicable):
Vsphere UPI 4.12.34
How reproducible:
In customer's environment (100%)
Steps to Reproduce:
The storage CO is in degraded state with the below error: ~~~ storage 4.12.34 True True True 13h VSphereCSIDriverOperatorCRDegraded: VMwareVSphereControllerDegraded: error logging into vcenter: ServerFaultCode: Cannot complete login due to an incorrect user name or password. ~~~ - The vsphere-problem-detector pod logs do not show `complete login due to an incorrect user name or password.` error. - The credentials are absolutely correct as customer can login to vcenter using the same credentials. - Tried creating a new PVC and mount it to pod using thin-csi, this confirmed that the vmdk file is created on the vcenter side for the PV/PVC. That means the credentials and the permissions are OK. This also confirms that no issue with credentials. - During cluster installation, they used vcenter IP address since vcenter FQDN was not available that time. On remote, saw that while logging into the vcenter the login page shows vcenter FQDN and after login it shows vcenter IP address. - This thing is now correct and vcenter IP and FQDNs are not flipping now. - The cluster is configured with vcenter FQDN now. - The debug logs for storages/cluster and clustercsidriver/csi.vsphere.vmware.com are enabled, but they did not help with much information. - Tried putting vcenter password in single inverted commas, but the vsphere-csi-driver-controller pod started going in crashloopbackoff state. - Tried putting vcenter password in double inverted commas, which also caused failures.
Actual results:
The storage CO seems like reporting false positive message. The storage CO degraded saying that cannot login to vcenter, but the username/password is correct, and assigning new PV/PVC works. The vsphere-csi-driver-operator pod is not happy.
Expected results:
The storage CO should not report the Degraded state.
This is preventing customer to upgrade the cluster to 4.14 version.
