Customer performed a disaster recovery test. As part of the test, one nodepool was forcefully shut down in vCenter.

After restarting the shutdown nodes some statefulset pods wanted to start again but got stuck in an MountVolume.SetUp failed Error. 3 out of the 5 statefulset replicas got killed by the node shutdown. 1 of those 3 pods was able to start successfully after the restart of the nodes without any problems, but 2 out of the 3 restarted pods were stuck in ContainerCreating state with MountVolume.SetUp failed Error.

Event of one of the pods stuck in ContainerCreating state:

MountVolume.SetUp failed for volume "pvc-dea66279-a255-4879-9a84-2e39f05593ea" : rpc error: code = FailedPrecondition desc = volume ID: "[VSAN-EXAMPLE-OPENSHIFT] 9f581864-2871-a9ee-a6ba-9e66fb700256/_002c/962e41a3d16040a39d198677ef61f0f1.vmdk" does not appear staged to "/var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/80848e86228674b6011b910e5bf771a188cb7d242cef8cae93e3efddd155bc17/globalmount"

We were able to fix the mount error with a pod restart (waited at least 20 minutes to see if the error fixes itself, but as nothing happened we killed the pods)

OpenShift Container Platform 4.14.37

On customer side, only in one disaster recovery test

    1. Run OCP 4.14 on vSphere. Forcefully stop multiple nodes where Pods of a StatefulSet are running
    2. Restart the nodes
    3. Observe Pod status
    

Not all Pods start, some fail with a MountVolume.SetUp failed Error

All Pods start as expected

- must-gather and csi-driver logs are available in Support Case