Ensure proper cloud provider configuration

When running the in-tree storage driver with an external CCM running, we need to ensure that the flags to the kubelet, kube-controller-manager, and kube-apiserver are expressed properly. The ideal state for the flags when in-tree driver is enabled during migration is as follows:

kubelet - "--cloud-provider=vsphere"

kube-controller-manager - "--cloud-provider=external --external-cloud-volume-plugin=vsphere"

kube-apiserver - "--cloud-provider=external"

for new cluster installations we expect the following flags:

kubelet - "--cloud-provider=external"

kube-controller-manager - "--cloud-provider=external --external-cloud-volume-plugin=vsphere"

kube-apiserver - "--cloud-provider=external"
 

Our current thinking is that most of these flags will be set the proper way with the exception of the kubelet flag. That will require a patch to the MCO to ensure that only the kubelet is taking the vsphere cloud provider as a flag option.