https://sippy.dptools.openshift.org/sippy-ng/jobs/4.12/analysis?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22periodic-ci-openshift-release-master-nightly-4.12-e2e-vsphere-csi%22%7D%5D%7D

This job is now down around 3% pass rate.

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.12-informing#periodic-ci-openshift-release-master-nightly-4.12-e2e-vsphere-csi shows it is always the same 4-5 storage tests, they seem to always time out out cleaning up the persistent volume after 5 mins. The pvcs appear to stick around after and appear in must-gather.

4.11 job appears pretty similar in testgrid.

Looking specifically at https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.12-e2e-vsphere-csi/1552615656082378752

We found some interesting logs in namespaces/openshift-cluster-csi-drivers/pods/vmware-vsphere-csi-driver-controller-5db4fb5869-bchfp/csi-resizer/csi-resizer/logs/current.log

2022-07-28T12:21:08.444275635Z E0728 12:21:08.444226       1 controller.go:286] Error syncing PVC: resize volume "pvc-b0d82d9a-1433-4d62-9b6a-ea6b8413e9a1" by resizer "csi.vsphere.vmware.com" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Though this is relevant to the likely goal of the test, rather than cleaning up the PV.

Is something going on with the vsphere cluster and storage?

cc rhn-support-rvanderp