Description of problem:

When CSI migration for vSphere CSI driver is enabled using TechPreviewNoUpgrade feature gate while there are some Deployments that use in-tree PVs or inline volumes, some of these Deployments don't survive and they keep their pods ContainerCreating forever.

Version-Release number of selected component (if applicable):

4.12.0-0.ci.test-2023-01-10-153025-ci

How reproducible: sometimes, under 50%

Steps to Reproduce:

Same as https://bugzilla.redhat.com/show_bug.cgi?id=2031045#c8

Actual results:

Some Pods are ContanerCreating

Expected results:

All Pods running

Additional info:

See https://bugzilla.redhat.com/show_bug.cgi?id=2031045#c18 for must-gather. 

A pod is stuck at attaching volume with this error:

rpc error: code = Internal desc = failed to set keepAfterDeleteVm control flag for VolumeID "438ac617-b70e-4154-b090-1d11282fe099"

VolumeAttachment.Name: csi-6b77a4227c694303af20f9f93a514726545e2af02032036a663208321a599fbb
nodeName: jsafrane-vk95d-worker-bdm8z
persistentVolumeName: pvc-a5992118-b04c-4cdd-834d-d27ad54f4907

Note that I added extra logs to vsphere-syncer.