Description of problem:

While running DM restore using CephRBD, all pods were created but not all PVs were created.  Getting errors like BZ 219598, but not errors found in 'csi-rbdplugin-provisioner' pod.

Backup of 100 PVS were completed, during restore 1-3 PVs were missing

Version-Release number of selected component (if applicable):

ODF 4.12.9

ODP 4.12.2
OADP 1.2.0-69
Using CephRBD

How reproducible:

Steps to Reproduce:
1. Create NS with 100PVs (6GB size , 2GB usage) - CephRBD
2. Running DM backup (20cc) - Completed
3. Delete NS , VSB , VS , VSC
4. Running DM restore (20cc)

Actual results:

Restore Failed. Missing PVs

Expected results:

Restore Completed, all PVs are created and pods are running

Additional info:

Errors:
    Operation Error:       Asynchronous action timed out

  Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-25f078e3-b8e0-4012-babf-9ad9c200ba99:
    Backup Item Action Plugin:  velero.io/vsm-volumesnapshotcontent-backupper
    Operation ID:               perf-busy-data-cephrbd-100pods/vsb-d2wn4
    Items to Update:
                           volumesnapshotbackups.datamover.oadp.openshift.io perf-busy-data-cephrbd-100pods/vsb-d2wn4
    Phase:                 Failed
    Operation Error:       Asynchronous action timed out
    Progress description:  Phase: SnapshotBackupDone BatchingStatus: Completed
    Created:               2023-05-17 21:36:35 +0000 UTC
    Started:               2023-05-17 21:36:35 +0000 UTC
    Updated:               2023-05-17 22:36:47 +0000 UTC

    perf-busy-data-cephrbd-100pods:  error preparing volumesnapshots.snapshot.storage.k8s.io/perf-busy-data-cephrbd-100pods/velero-pvc-busy-data-rbd-100pods-1-pg7bb: rpc error: code = Unknown desc = timed out waiting for the condition
                                     error preparing volumesnapshots.snapshot.storage.k8s.io/perf-busy-data-cephrbd-100pods/velero-pvc-busy-data-rbd-100pods-11-9fg4q: rpc error: code = Unknown desc = timed out waiting for the condition
                                     error preparing persistentvolumeclaims/perf-busy-data-cephrbd-100pods/pvc-busy-data-rbd-100pods-1: rpc error: code = Unknown desc = Failed to get Volumesnapshot perf-busy-data-cephrbd-100pods/velero-pvc-busy-data-rbd-100pods-1-pg7bb to restore PVC perf-busy-data-cephrbd-100pods/pvc-busy-data-rbd-100pods-1: volumesnapshots.snapshot.storage.k8s.io "velero-pvc-busy-data-rbd-100pods-1-pg7bb" not found
                                     error preparing persistentvolumeclaims/perf-busy-data-cephrbd-100pods/pvc-busy-data-rbd-100pods-11: rpc error: code = Unknown desc = Failed to get Volumesnapshot perf-busy-data-cephrbd-100pods/velero-pvc-busy-data-rbd-100pods-11-9fg4q to restore PVC perf-busy-data-cephrbd-100pods/pvc-busy-data-rbd-100pods-11: volumesnapshots.snapshot.storage.k8s.io "velero-pvc-busy-data-rbd-100pods-11-9fg4q" not found

time="2023-05-18T12:36:07Z" level=error msg="Timed out awaiting reconciliation of volumesnapshotrestoreList" cmd=/plugins/velero-plugin-for-vsm logSource="/remote-source/app/internal/util/util.go:393" pluginName=velero-plugin-for-vsm restore=openshift-adp/dm-restore-rbd-100pvss-20cc-iter2

time="2023-05-18T12:52:00Z" level=error msg="Timed out awaiting reconciliation of volumesnapshotrestoreList" cmd=/plugins/velero-plugin-for-vsm logSource="/remote-source/app/internal/util/util.go:393" pluginName=velero-plugin-for-vsm restore=openshift-adp/dm-restore-rbd-100pvss-20cc-iter2

time="2023-05-18T13:03:32Z" level=error msg="Namespace perf-busy-data-cephrbd-100pods, resource restore error: error preparing volumesnapshots.snapshot.storage.k8s.io/perf-busy-data-cephrbd-100pods/velero-pvc-busy-data-rbd-100pods-1-pg7bb: rpc error: code = Unknown desc = timed out waiting for the condition" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:498" restore=openshift-adp/dm-restore-rbd-100pvss-20cc-iter2
time="2023-05-18T13:03:32Z" level=error msg="Namespace perf-busy-data-cephrbd-100pods, resource restore error: error preparing volumesnapshots.snapshot.storage.k8s.io/perf-busy-data-cephrbd-100pods/velero-pvc-busy-data-rbd-100pods-11-9fg4q: rpc error: code = Unknown desc = timed out waiting for the condition" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:498" restore=openshift-adp/dm-restore-rbd-100pvss-20cc-iter2
time="2023-05-18T13:03:32Z" level=error msg="Namespace perf-busy-data-cephrbd-100pods, resource restore error: error preparing persistentvolumeclaims/perf-busy-data-cephrbd-100pods/pvc-busy-data-rbd-100pods-1: rpc error: code = Unknown desc = Failed to get Volumesnapshot perf-busy-data-cephrbd-100pods/velero-pvc-busy-data-rbd-100pods-1-pg7bb to restore PVC perf-busy-data-cephrbd-100pods/pvc-busy-data-rbd-100pods-1: volumesnapshots.snapshot.storage.k8s.io \"velero-pvc-busy-data-rbd-100pods-1-pg7bb\" not found" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:498" restore=openshift-adp/dm-restore-rbd-100pvss-20cc-iter2
time="2023-05-18T13:03:32Z" level=error msg="Namespace perf-busy-data-cephrbd-100pods, resource restore error: error preparing persistentvolumeclaims/perf-busy-data-cephrbd-100pods/pvc-busy-data-rbd-100pods-11: rpc error: code = Unknown desc = Failed to get Volumesnapshot perf-busy-data-cephrbd-100pods/velero-pvc-busy-data-rbd-100pods-11-9fg4q to restore PVC perf-busy-data-cephrbd-100pods/pvc-busy-data-rbd-100pods-11: volumesnapshots.snapshot.storage.k8s.io \"velero-pvc-busy-data-rbd-100pods-11-9fg4q\" not found" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:498" restore=openshift-adp/dm-restore-rbd-100pvss-20cc-iter2