Description of problem:

The customer is requesting us to be able to restore the VMs in the event of a cluster failure. This means that we will need to make use of the Data Mover feature so that we can keep the snapshots protected. We were able to make use of the documentation, i.e. https://docs.openshift.com/container-platform/4.13/backup_and_restore/application_backup_and_restore/backing_up_and_restoring/backing-up-applications.html#oadp-using-data-mover-for-csi-snapshots_backing-up-applications and the information in the demo video (https://www.youtube.com/watch?v=iyoxuP2xb2E) to successfully backup and restore a mssql stateful application but the same steps failed for the OCP-V VM.

Similar to what was mentioned in https://access.redhat.com/support/cases/#/case/03544060, we were able to send the data over to S3 but the VM was unable to find the boot disk during the boot up process (upon restore). We also found that the content backed up in the S3 bucket was very small in size.

[root@ocp4-bastion tmp]# du -sh centos-test-vm/
420K centos-test-vm/
[root@ocp4-bastion tmp]# du -sh test-backup/
96K test-backup/
[root@ocp4-bastion tmp]#

We reached out to CNV and OADP Product Team and got to know about https://issues.redhat.com/browse/OADP-2144 and https://github.com/openshift/oadp-operator/blob/master/docs/examples/datamover_advanced_voloptions.md  

We tried the methods over there but continue to get the following errors while trying to get pass the "WaitingForPluginOperations" state for the VM backup.

failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd": rpc error: code = InvalidArgument desc = multi node access modes are only supported on rbd `block` type volumes

Have tried multiple versions of OADP definitions (see attached) and are getting similar errors.

Version-Release number of selected component (if applicable):

OCP 4.12 with OADP 1.2

How reproducible:

100% reproducible

Steps to Reproduce:

• Refer to attachments for the YAML configuration Files
• See below for steps

[root@ocp4-bastion ~]# oc apply -f oadp.yaml
dataprotectionapplication.oadp.openshift.io/openshift-dpa created
[root@ocp4-bastion ~]# oc apply -f centos-vm-backup.yaml
backup.velero.io/centos-test-vm created
[root@ocp4-bastion ~]# ./datamover_resources.sh -b -d
Get Backups:NAME             STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
centos-test-vm   WaitingForPluginOperations   0        3          2023-06-29 15:46:58 +0000 UTC   29d       openshift-dpa-1    <none>Total Snapshots:  3
Total OADP Snapshots:  1
Total SnapshotContents:  6Total VSB:  1
Completed:  0
InProgress:  0
SnapshotBackupDone:  0VSB STATUS
Completed:  0
Processing:  1
Queued:  0
No resources foundTotal ReplicationSources:  0VOLUME SNAPSHOT CONTENTS ******
Name:  snapcontent-14ec19e0-59e3-41ad-9067-a95b757e589f  ReadyToUse:  true  creationTime:  2023-06-29T15:30:02Z
Name:  snapcontent-3370894a-90be-430a-b5ca-7ba1fa033144  ReadyToUse:  true  creationTime:  2023-06-29T15:39:36Z
Name:  snapcontent-5a2b4834-6f9d-46de-b83a-f9b2a70f1ec6  ReadyToUse:  true  creationTime:  2023-06-29T09:33:26Z
Name:  snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4  ReadyToUse:  true  creationTime:  2023-06-29T15:47:05Z
Name:  snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-clone  ReadyToUse:  true  creationTime:  2023-06-29T15:47:24Z
Name:  snapcontent-f1a3179a-8aec-416e-b541-a085eb4bba1b  ReadyToUse:  true  creationTime:  2023-06-29T15:11:04ZREPLICATION SOURCE ******
[root@ocp4-bastion ~]# oc get pvc -n openshift-adp
NAME                                                   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc   Pending                                      ocs-storagecluster-ceph-rbd   19m
[root@ocp4-bastion ~]# oc describe pvc snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc -n openshift-adp
Name:          snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc
Namespace:     openshift-adp
StorageClass:  ocs-storagecluster-ceph-rbd
Status:        Pending
Volume:
Labels:        datamover.oadp.openshift.io/vsb=vsb-hj88d
Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
DataSource:
  APIGroup:  snapshot.storage.k8s.io
  Kind:      VolumeSnapshot
  Name:      snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-clone-volumesnapshot
Used By:     vsb-hj88d-pod
Events:
  Type     Reason                Age                   From                                                                                                                Message
  ----     ------                ----                  ----                                                                                                                -------
  Normal   PVCReconciled         19m                   VolumeSnapshotBackup-Controller                                                                                     performed created on PVC snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc
  Normal   ExternalProvisioning  4m47s (x65 over 19m)  persistentvolume-controller                                                                                         waiting for a volume to be created, either by external provisioner "openshift-storage.rbd.csi.ceph.com" or manually created by system administrator
  Normal   Provisioning          113s (x14 over 19m)   openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-6f6c684477-tbj9v_5b66cd26-d4c7-4b17-a870-22bd45585156  External provisioner is provisioning volume for claim "openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc"
  Warning  ProvisioningFailed    113s (x14 over 19m)   openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-6f6c684477-tbj9v_5b66cd26-d4c7-4b17-a870-22bd45585156  failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd": rpc error: code = InvalidArgument desc = multi node access modes are only supported on rbd `block` type volumes}}
{{[root@ocp4-bastion ~]# oc get pod -n openshift-adp
NAME                                                READY   STATUS    RESTARTS   AGE
openshift-adp-controller-manager-7946f6bd45-fhn5c   1/1     Running   0          8h
velero-5875cf7c9-jjsnk                              1/1     Running   0          22m
volume-snapshot-mover-5bcbc8c79d-bmbhv              1/1     Running   0          22m
vsb-hj88d-pod                                       0/1     Pending   0          21m
[root@ocp4-bastion ~]# oc get backup -n openshift-adp
NAME             AGE
centos-test-vm   22m
[root@ocp4-bastion ~]# oc get volumesnapshots
NAME                                                              READYTOUSE   SOURCEPVC                         SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS                            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
velero-centos-stream9-hfyop0pl7wg2sfw1-8576v                      true         centos-stream9-hfyop0pl7wg2sfw1                           30Gi          ocs-storagecluster-rbdplugin-snapclass   snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4   22m            22m
vmsnapshot-2f08d6aa-521b-434c-a0cb-42a59f5c0f9a-volume-rootdisk   true         centos-stream9-hfyop0pl7wg2sfw1                           30Gi          ocs-storagecluster-rbdplugin-snapclass   snapcontent-5a2b4834-6f9d-46de-b83a-f9b2a70f1ec6   6h36m          6h36m
[root@ocp4-bastion ~]# oc get volumesnapshotcontents
NAME                                                     READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER                               VOLUMESNAPSHOTCLASS                      VOLUMESNAPSHOT                                                          VOLUMESNAPSHOTNAMESPACE   AGE
snapcontent-14ec19e0-59e3-41ad-9067-a95b757e589f         true         32212254720   Retain           openshift-storage.rbd.csi.ceph.com   ocs-storagecluster-rbdplugin-snapclass   velero-centos-stream9-hfyop0pl7wg2sfw1-9grbj                            test-backup               40m
snapcontent-3370894a-90be-430a-b5ca-7ba1fa033144         true         32212254720   Retain           openshift-storage.rbd.csi.ceph.com   ocs-storagecluster-rbdplugin-snapclass   velero-centos-stream9-hfyop0pl7wg2sfw1-7qdh5                            test-backup               30m
snapcontent-5a2b4834-6f9d-46de-b83a-f9b2a70f1ec6         true         32212254720   Retain           openshift-storage.rbd.csi.ceph.com   ocs-storagecluster-rbdplugin-snapclass   vmsnapshot-2f08d6aa-521b-434c-a0cb-42a59f5c0f9a-volume-rootdisk         test-backup               6h36m
snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4         true         32212254720   Retain           openshift-storage.rbd.csi.ceph.com   ocs-storagecluster-rbdplugin-snapclass   velero-centos-stream9-hfyop0pl7wg2sfw1-8576v                            test-backup               23m
snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-clone   true         0             Retain           openshift-storage.rbd.csi.ceph.com   ocs-storagecluster-rbdplugin-snapclass   snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-clone-volumesnapshot   openshift-adp             22m
snapcontent-f1a3179a-8aec-416e-b541-a085eb4bba1b         true         32212254720   Retain           openshift-storage.rbd.csi.ceph.com   ocs-storagecluster-rbdplugin-snapclass   velero-centos-stream9-hfyop0pl7wg2sfw1-29n56                            test-backup               59m
[root@ocp4-bastion ~]# oc get vm -n test-backup
NAME                              AGE     STATUS    READY
centos-stream9-hfyop0pl7wg2sfw1   6h40m   Running   True
[root@ocp4-bastion ~]#

Actual results:

Unable to backup OCP-V VM using Data Mover

Expected results:

Able to perform full backup of VM for recovery in case of cluster failure

Additional info:

Fault replicated in RHPDS environment with catalog, OpenShift AIO (Equinix Metal) with OpenShift Virtualization Lab

 
[root@ocp4-bastion ~]# oc logs  deployment.apps/volume-snapshot-mover -n openshift-adp
...2023-06-29T16:17:14Z    INFO    cloned PVC openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc is not in bound state    {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup":{"name":"vsb-hj88d","namespace":"test-backup"}, "namespace": "test-backup", "name": "vsb-hj88d", "reconcileID": "a0a34e99-15f9-4d9d-a43a-ca012512bcf8", "vsb": "test-backup/vsb-hj88d"}{}
2023-06-29T16:17:19Z    INFO    cloned PVC openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc is not in bound state    {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup":{"name":"vsb-hj88d","namespace":"test-backup"}, "namespace": "test-backup", "name": "vsb-hj88d", "reconcileID": "16a9faf7-a62f-4ce0-a560-cdb40bdf7e26", "vsb": "test-backup/vsb-hj88d"}{}
2023-06-29T16:17:24Z    INFO    cloned PVC openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc is not in bound state    {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup":{"name":"vsb-hj88d","namespace":"test-backup"}, "namespace": "test-backup", "name": "vsb-hj88d", "reconcileID": "7879887c-6b37-424f-be8c-daa131eaf414", "vsb": "test-backup/vsb-hj88d"}{}
2023-06-29T16:17:29Z    INFO    cloned PVC openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc is not in bound state    {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup":{"name":"vsb-hj88d","namespace":"test-backup"}, "namespace": "test-backup", "name": "vsb-hj88d", "reconcileID": "c9c27526-fbd9-4916-8b2a-e19e6dca55b9", "vsb": "test-backup/vsb-hj88d"}{}
2023-06-29T16:17:34Z    INFO    cloned PVC openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc is not in bound state    {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup":{"name":"vsb-hj88d","namespace":"test-backup"}, "namespace": "test-backup", "name": "vsb-hj88d", "reconcileID": "bb9e7288-2873-409e-bf76-c64a25c2f007", "vsb": "test-backup/vsb-hj88d"}{}
2023-06-29T16:17:39Z    INFO    cloned PVC openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc is not in bound state    {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup":{"name":"vsb-hj88d","namespace":"test-backup"}, "namespace": "test-backup", "name": "vsb-hj88d", "reconcileID": "63818def-8241-40e4-90c0-4f3cb094b2f6", "vsb": "test-backup/vsb-hj88d"}{}
2023-06-29T16:17:44Z    INFO    cloned PVC openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc is not in bound state    {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup":{"name":"vsb-hj88d","namespace":"test-backup"}, "namespace": "test-backup", "name": "vsb-hj88d", "reconcileID": "5ccda74b-6881-443b-8e3d-2299ea32e9e4", "vsb": "test-backup/vsb-hj88d"}{}
2023-06-29T16:17:49Z    INFO    cloned PVC openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc is not in bound state    {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup":{"name":"vsb-hj88d","namespace":"test-backup"}, "namespace": "test-backup", "name": "vsb-hj88d", "reconcileID": "918d07d6-3bca-4bf4-b4d7-f467a2b283ac", "vsb": "test-backup/vsb-hj88d"}{}
2023-06-29T16:17:54Z    INFO    cloned PVC openshift-adp/snapcontent-f0affa00-35ec-48b1-a72c-b55de3e09ab4-pvc is not in bound state    {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup":{"name":"vsb-hj88d","namespace":"test-backup"}, "namespace": "test-backup", "name": "vsb-hj88d", "reconcileID": "5b1a5fbf-964b-4674-a5b0-c4a3d705c356", "vsb": "test-backup/vsb-hj88d"}{}