kubevirt-csi unable to unpublish volumes in the event a VM is unexpectedly destroyed

kubevirt csi is unable to unpublish a volume in the event that the VM/VMI that the volume was published on unexpectedly disappears. This situation can occur for many reasons. Someone could forcibly delete the VM, an replace update could occur that destroys a VM before it can unpublish a volume, a VM node can become unresponsible and the capi machine controller will delete it, and other scenarios like this.

When this situation occurs, the PVC within the guest will never get deleted properly. Kubevirt csi will report the following error.

 
I0531 13:07:51.338413 1 controller.go:264] Detaching DataVolume pvc-4c4d4744-8a04-4df1-964b-d4eac90a93a2 from Node ID fc3ad096-53f0-535d-bbd8-45a3ab3803d1
E0531 13:07:51.349493 1 server.go:124] /csi.v1.Controller/ControllerUnpublishVolume returned with error: rpc error: code = NotFound desc = failed to find VM with domain.firmware.uuid 5cb46a00-2b8b-509b-b32b-39d1bab4e8b5
 

To resolve this, the kubevirt-csi controller needs to gracefully handle unpublishing a volume when the VM and VMI associated with the volume no longer exists.