Evaluate and recommend solutions for migrating vSphere volumes between datastores

It is often requested feature that, for various reasons customers are interested in migrating their existing OCP PV's backing volumes to a different datastore.

Migrating VMs to different datastore is while out of scope, we should evaluate what happens if someone migrates a VM to a different datastore while dynamically attached volumes were still attached to the VM.

When it comes to Openshift - usually couple of solutions are often mentioned:

1. Using Migration Toolkit (MTC)

This solution although developed and supported by redhat requires creation of parallel cluster and then moving stuff. Storage team has not evaluated or tested this solution.

2. Migration of VMDK files inside kubevol folder

This was a possible solution for in-tree volumes, where one can take all volumes and their associated workloads(pods) offline and then move entire kubevols folder to new datastore.

Before doing this though - one must choose to use Retain volume deletion policy for PV and then backup associated PVC and PVs. All PVCs should be deleted, so as only PVs are left.

After volumes are moved, we need to change corresponding PV YAML to reflect new path and then create them. We also need to re-create associated PVCs and then scale the workloads.

This solution is no longer as simple/possible after CSI volumes. But I think we should still test this one.

3. Another option is to use CNS manager - https://github.com/vmware-samples/cloud-native-storage-self-service-manager

This should allow migration of CSI volumes between datastores. There is https://core.vmware.com/blog/whats-new-tanzu-cns-vsphere-80-u2 feature of tanzu which may use it.

Exit Critirea:

We need to evaluate all possible solutions for volume migration and then make recommendation based on conclusions.

If necessary - this spike should also be used for recommending additional tooling that needs to be developed to support this feature.

Please not development of actual tooling is actually out of scope for this spike.