Description of problem:

While running datamover backup , it failed with "PartiallyFailed" status.

The failed backup uses Ceph-FS storage.

The env. running backup ~30 hours and the backups namespace with CephFS & CephRBD storage. The backups become slower each time.

Version-Release number of selected component (if applicable):

OCP 4.12.9

ODF 4.12.2
OADP 1.2.0-48

  features:
    dataMover:
      credentialName: restic-secret
      enable: true
      maxConcurrentBackupVolumes: "7"
      volumeOptions:
        sourceVolumeOptions:
          accessMode: ReadOnlyMany
          cacheAccessMode: ReadWriteMany
          cacheStorageClassName: ocs-storagecluster-cephfs
          moverSecurityContext: true
          storageClassName: ocs-storagecluster-cephfs-shallow

How reproducible:

Steps to Reproduce:
1. Create ns with 50 PV with data (6GB PV size,  2GB usage) - CephFS
2. Running datamover backup with maxConcurrentBackupVolumes: "7"
3. Test failed

Actual results:

Test Failed with "runtime error: invalid memory address or nil pointer dereference" error

Expected results:

Backup completed

Additional info:

Before the backup ended with "PartiallyFailed", the CR status was 'FinalizingPartiallyFailed'

error="error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=perf-busy-data-cephfs-50pods, name=velero-pvc-busy-data-fs-50pods-33-xzjzv): rpc error: code = Aborted desc = plugin panicked: runtime error: invalid memory address or nil pointer dereference
 

Backups Duration:

Attched 2 jpeg files