Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-849

DataMover: restore PartiallyFails randomly with "ReplicationDestination.volsync.backube xxxx not found" error

    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • Proposed
    • Yes

      Description of problem: effectively from 1.1.1 (not sure exactly in which build it was first introduced), restore started to fail randomly with "ReplicationDestination.volsync.backube xxxx not found" error (where xxx is the name of the replicationdestination CR), although it looks like the ReplicationDestination was created eventually.

      Also looks like the CSI volumesnapshot fails with the following error:

      'Failed to check and update snapshot content: failed to list snapshot
              for content velero-velero-mysql-tz5f4-4hq58: "rpc error: code = Internal desc
              = Could not list snapshots: InvalidParameterValue: Value ( 0 ) for parameter
              maxResults is invalid. Expecting a value greater than 5.\n\tstatus code: 400,
              request id: 655bbfa8-c7e0-4311-900b-092f2e617f8d"'

       

      Please note that the restore can also pass for the same application.

      Version-Release number of selected component (if applicable):

      1.1.1, build oadp-operator-bundle-container-1.1.1-21

       

      How reproducible: happens a lot, not sure exactly how much.

      Steps to Reproduce:
      1. Create a backup of a stateful application with datamover for PV backup
      2. Make sure backup completes sucessfully - no errors on volmesnapshots nor on VSB (VSB is in "Completed" phase)
      3. Delete the application namespace
      4. Once the namespace is removed, create a restore of the backup

      Actual results:

      Restore may fail with the following errors:

      VSR:

      [mperetz@fedora oadp-e2e-qe]$   oc get vsr -A -o yaml
      apiVersion: v1
      items:
      - apiVersion: datamover.oadp.openshift.io/v1alpha1
        kind: VolumeSnapshotRestore
        metadata:
          creationTimestamp: "2022-10-11T10:29:27Z"
          generateName: vsr-
          generation: 1
          labels:
            velero.io/persistent-volume-claim-name: mysql
            velero.io/restore-name: mysql-ad93ad8a-494e-11ed-b0c4-902e163f806c
          name: vsr-mjp78
          namespace: mysql-persistent
          resourceVersion: "65364"
          uid: 47455522-fc52-44d1-a14c-f35205a7389e
        spec:
          protectedNamespace: openshift-adp
          resticSecretRef:
            name: ts-dpa-1-volsync-restic
          volumeSnapshotMoverBackupRef:
            resticrepository: s3:s3.amazonaws.com/oadpbucket145568/openshift-adp/snapcontent-4131629e-a252-44ba-8ccf-99553ea06a7d-pvc
            sourcePVCData:
              name: mysql
              size: 2Gi
              storageClassName: gp2-csi
            volumeSnapshotClassName: example-snapclass
        status:
          conditions:
          - lastTransitionTime: "2022-10-11T10:29:27Z"
            message: ReplicationDestination.volsync.backube "vsr-mjp78-rep-dest" not found
            reason: Error
            status: "False"
            type: Reconciled
          phase: Failed
      kind: List
      metadata:
        resourceVersion: ""

      VolumeSnapshot:

      [mperetz@fedora oadp-e2e-qe]$ oc get volumesnapshot -A -o yaml
      apiVersion: v1
      items:
      - apiVersion: snapshot.storage.k8s.io/v1
        kind: VolumeSnapshot
        metadata:
          annotations:
            velero.io/csi-driver-name: ebs.csi.aws.com
            velero.io/csi-volumesnapshot-handle: snap-0e464beebb6c4c180
            velero.io/csi-vsc-deletion-policy: Retain
            velero.io/vsi-volumesnapshot-restore-size: 2Gi
          creationTimestamp: "2022-10-11T10:29:35Z"
          finalizers:
          - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
          generation: 1
          labels:
            velero.io/backup-name: mysql-ad93ad8a-494e-11ed-b0c4-902e163f806c
            velero.io/restore-name: mysql-ad93ad8a-494e-11ed-b0c4-902e163f806c
          name: velero-mysql-cwwp5
          namespace: mysql-persistent
          resourceVersion: "67546"
          uid: 0276f2a6-4d17-4dee-a0c6-1fa8f9b33d76
        spec:
          source:
            volumeSnapshotContentName: velero-velero-mysql-cwwp5-j4qdt
          volumeSnapshotClassName: example-snapclass
        status:
          boundVolumeSnapshotContentName: velero-velero-mysql-cwwp5-j4qdt
          error:
            message: 'Failed to check and update snapshot content: failed to list snapshot
              for content velero-velero-mysql-cwwp5-j4qdt: "rpc error: code = Internal desc
              = Could not list snapshots: InvalidParameterValue: Value ( 0 ) for parameter
              maxResults is invalid. Expecting a value greater than 5.\n\tstatus code: 400,
              request id: f18b6673-526e-49db-bff3-5635de93d5c7"'
            time: "2022-10-11T10:31:47Z"
          readyToUse: false
      - apiVersion: snapshot.storage.k8s.io/v1
        kind: VolumeSnapshot
        metadata:
          creationTimestamp: "2022-10-11T10:29:43Z"
          finalizers:
          - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
          generation: 1
          labels:
            app.kubernetes.io/created-by: volsync
          name: volsync-vsr-mjp78-rep-dest-dest-20221011102943
          namespace: openshift-adp
          ownerReferences:
          - apiVersion: volsync.backube/v1alpha1
            blockOwnerDeletion: true
            controller: true
            kind: ReplicationDestination
            name: vsr-mjp78-rep-dest
            uid: 466d66c3-b544-4460-bca8-9f383f80352d
          resourceVersion: "66319"
          uid: 236a5c9a-f9a3-4a9e-a83b-a2e7ed279cb3
        spec:
          source:
            persistentVolumeClaimName: volsync-vsr-mjp78-rep-dest-dest
          volumeSnapshotClassName: example-snapclass
        status:
          boundVolumeSnapshotContentName: snapcontent-236a5c9a-f9a3-4a9e-a83b-a2e7ed279cb3
          creationTime: "2022-10-11T10:29:45Z"
          readyToUse: true
          restoreSize: 2Gi
      kind: List
      metadata:
        resourceVersion: ""

      Expected results: Restore should pass

       

      Additional info:

            [OADP-849] DataMover: restore PartiallyFails randomly with "ReplicationDestination.volsync.backube xxxx not found" error

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory, and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2022:8634

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:8634

            Maya Peretz mentioned this issue in a merge request of app-mig / oadp-e2e-qe on branch bug_849:

            Test to verify OADP-611, OADP-1016 and OADP-849

            GitLab CEE Bot added a comment - Maya Peretz mentioned this issue in a merge request of app-mig / oadp-e2e-qe on branch bug_849 : Test to verify OADP-611 , OADP-1016 and OADP-849

            Wes Hayutin added a comment - mperetz@redhat.com rhn-support-prajoshi   back on_qa w/ https://jenkins-cvp-5d6ccd93bed8bd3809ddcb5d.apps.ocp-c1.prod.psi.redhat.com/job/cvp-redhat-operator-bundle-image-validation-test/29/artifact/cvp-test-report.html    

            Emily McMullan added a comment - New PR for issue: https://github.com/migtools/volume-snapshot-mover/pull/165  

            Maya Peretz added a comment -

            Just updating here too what was said in the email thread for better tracking:

            -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

            Build oadp-oadp-operator-bundle:1.1.1-37, using custom image quay.io/emcmulla/data-mover:fix849 for VSM:

            We hit 2 problems:

             
            1. VSR fails due to this error from the ReplicationDestination - it looks random:

            - apiVersion: volsync.backube/v1alpha1
              kind: ReplicationDestination
              metadata:
                creationTimestamp: "2022-11-02T08:52:17Z"
                generation: 1
                labels:
                  datamover.oadp.openshift.io/vsr: vsr-wg2xr
                name: vsr-wg2xr-rep-dest
                namespace: openshift-adp
                resourceVersion: "237332"
                uid: 6ad8fae7-4f46-45b9-94a8-f915be2b56fb
              spec: {}
              status:
                conditions:
                - lastTransitionTime: "2022-11-02T08:52:17Z"
                  message: a replication method must be specified
                  reason: Error
                  status: "False"
                  type: Synchronizing
            kind: List
            metadata:
              resourceVersion: ""

             
            2. A nil pointer exception in the VSM pod:

            2022-11-02T08:33:09Z DEBUG events Normal {"object": {"kind":"Secret","namespace":"openshift-adp","name":"vsr-x52x5-secret","uid":"69993dd0-245b-4220-b617-7b1db9fcf4fa","apiVersion":"v1","resourceVersion":"210567"}, "reason": "ResticSecretRestoreReconciled", "message": "created restic secret vsr-x52x5-secret"}
            2022-11-02T08:33:09Z INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "volumesnapshotrestore", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotRestore", "volumeSnapshotRestore": {"name":"vsr-x52x5","namespace":"nginx-j2"}, "namespace": "nginx-j2", "name": "vsr-x52x5", "reconcileID": "9025799d-fbd0-41df-8ed7-4e8ca3cbec59"}
            panic: runtime error: invalid memory address or nil pointer dereference [recovered]
            panic: runtime error: invalid memory address or nil pointer dereference
            [signal SIGSEGV: segmentation violation code=0x1 addr=0x1b0 pc=0x136ebf6]
            
            goroutine 484 [running]:
            sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
            /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:118 +0x1f4
            panic({0x1508540, 0x23c7060})
            /usr/lib/golang/src/runtime/panic.go:838 +0x207
            github.com/konveyor/volume-snapshot-mover/controllers.(*VolumeSnapshotRestoreReconciler).WaitForVolSyncSnapshotContentToBeReady(0xc000785440, {{0x1913c60?, 0xc002984c00?}, 0x11e5efd?})
            /opt/app-root/src/volume-snapshot-mover/controllers/volumesnapshot.go:278 +0x176
            github.com/konveyor/volume-snapshot-mover/controllers.ReconcileBatch({{0x1913c60?, 0xc002984c00?}, 0xc002984ba0?}, {0xc0008e9c40, 0x5, 0xc00297cc70?})
            /opt/app-root/src/volume-snapshot-mover/controllers/common.go:84 +0x78
            github.com/konveyor/volume-snapshot-mover/controllers.(*VolumeSnapshotRestoreReconciler).Reconcile(0xc000785440, {0x1912468?, 0xc002984ba0}, {{{0xc00297cc80, 0x8}, {0xc00297cc70, 0x9}}})
            /opt/app-root/src/volume-snapshot-mover/controllers/volumesnapshotrestore_controller.go:102 +0x49a
            sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x19123c0?, {0x1912468?, 0xc002984ba0?}, {{{0xc00297cc80?, 0x1614120?}, {0xc00297cc70?, 0x4041f4?}}})
            /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121 +0xc8
            sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00024d360, {0x19123c0, 0xc0004dc300}, {0x155bce0?, 0xc000bbc340?})
            /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320 +0x33c
            sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00024d360, {0x19123c0, 0xc0004dc300})
            /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 +0x1d9
            sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
            /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 +0x85
            created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
            /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:230 +0x325

             

            Maya Peretz added a comment - Just updating here too what was said in the email thread for better tracking: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Build oadp-oadp-operator-bundle:1.1.1-37, u sing custom image quay.io/emcmulla/data-mover:fix849 for VSM: We hit 2 problems:   1. VSR fails due to this error from the ReplicationDestination - it looks random: - apiVersion: volsync.backube/v1alpha1   kind: ReplicationDestination   metadata:     creationTimestamp: "2022-11-02T08:52:17Z"     generation: 1     labels:       datamover.oadp.openshift.io/vsr: vsr-wg2xr     name: vsr-wg2xr-rep-dest     namespace: openshift-adp     resourceVersion: "237332"     uid: 6ad8fae7-4f46-45b9-94a8-f915be2b56fb   spec: {}   status:     conditions:     - lastTransitionTime: "2022-11-02T08:52:17Z"       message: a replication method must be specified       reason: Error       status: "False"       type: Synchronizing kind: List metadata:   resourceVersion: ""   2. A nil pointer exception in the VSM pod: 2022-11-02T08:33:09Z DEBUG events Normal { "object" : { "kind" : "Secret" , "namespace" : "openshift-adp" , "name" : "vsr-x52x5-secret" , "uid" : "69993dd0-245b-4220-b617-7b1db9fcf4fa" , "apiVersion" : "v1" , "resourceVersion" : "210567" }, "reason" : "ResticSecretRestoreReconciled" , "message" : "created restic secret vsr-x52x5-secret" } 2022-11-02T08:33:09Z INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference { "controller" : "volumesnapshotrestore" , "controllerGroup" : "datamover.oadp.openshift.io" , "controllerKind" : "VolumeSnapshotRestore" , "volumeSnapshotRestore" : { "name" : "vsr-x52x5" , "namespace" : "nginx-j2" }, "namespace" : "nginx-j2" , "name" : "vsr-x52x5" , "reconcileID" : "9025799d-fbd0-41df-8ed7-4e8ca3cbec59" } panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x1b0 pc=0x136ebf6] goroutine 484 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:118 +0x1f4 panic({0x1508540, 0x23c7060}) /usr/lib/golang/src/runtime/panic.go:838 +0x207 github.com/konveyor/volume-snapshot-mover/controllers.(*VolumeSnapshotRestoreReconciler).WaitForVolSyncSnapshotContentToBeReady(0xc000785440, {{0x1913c60?, 0xc002984c00?}, 0x11e5efd?}) /opt/app-root/src/volume-snapshot-mover/controllers/volumesnapshot.go:278 +0x176 github.com/konveyor/volume-snapshot-mover/controllers.ReconcileBatch({{0x1913c60?, 0xc002984c00?}, 0xc002984ba0?}, {0xc0008e9c40, 0x5, 0xc00297cc70?}) /opt/app-root/src/volume-snapshot-mover/controllers/common.go:84 +0x78 github.com/konveyor/volume-snapshot-mover/controllers.(*VolumeSnapshotRestoreReconciler).Reconcile(0xc000785440, {0x1912468?, 0xc002984ba0}, {{{0xc00297cc80, 0x8}, {0xc00297cc70, 0x9}}}) /opt/app-root/src/volume-snapshot-mover/controllers/volumesnapshotrestore_controller.go:102 +0x49a sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x19123c0?, {0x1912468?, 0xc002984ba0?}, {{{0xc00297cc80?, 0x1614120?}, {0xc00297cc70?, 0x4041f4?}}}) /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121 +0xc8 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00024d360, {0x19123c0, 0xc0004dc300}, {0x155bce0?, 0xc000bbc340?}) /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320 +0x33c sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00024d360, {0x19123c0, 0xc0004dc300}) /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 +0x1d9 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 +0x85 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /opt/app-root/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:230 +0x325  

            Wes Hayutin added a comment -

            Wes Hayutin added a comment - emcmulla@redhat.com and spampatt@redhat.com additionally posted https://github.com/migtools/volume-snapshot-mover/pull/164

            Sonia Garudi added a comment - - edited

            Issue is still seen with OADP 1.1.1-36 on P platform as well -

            [root@rdr-sonioadp-bf9a-lon06-bastion-0 ~]#  oc get restore.velero.io/mysql-restore-849 -n openshift-adp -ojsonpath={.status.phase}
            PartiallyFailed
            [root@rdr-sonioadp-bf9a-lon06-bastion-0 ~]# oc get vsr -n mysql-persistent vsr-crxwc -ojsonpath={.status}
            {"conditions":[{"lastTransitionTime":"2022-11-01T07:22:52Z","message":"ReplicationDestination.volsync.backube \"vsr-crxwc-rep-dest\" not found","reason":"Error","status":"False","type":"Reconciled"}],"phase":"Failed"} 

            Sonia Garudi added a comment - - edited Issue is still seen with OADP 1.1.1-36 on P platform as well - [root@rdr-sonioadp-bf9a-lon06-bastion-0 ~]#  oc get restore.velero.io/mysql-restore-849 -n openshift-adp -ojsonpath={.status.phase} PartiallyFailed [root@rdr-sonioadp-bf9a-lon06-bastion-0 ~]# oc get vsr -n mysql-persistent vsr-crxwc -ojsonpath={.status} { "conditions" :[{ "lastTransitionTime" : "2022-11-01T07:22:52Z" , "message" : "ReplicationDestination.volsync.backube \" vsr-crxwc-rep-dest\ " not found" , "reason" : "Error" , "status" : "False" , "type" : "Reconciled" }], "phase" : "Failed" }

            We believe that there is a race condition between the vsm get call https://github.com/migtools/volume-snapshot-mover/blob/master/controllers/replicationdestination.go#L124 for replicationDestination and its creation. This seems like its not 100% reproducible. 

            Shubham Pampattiwar added a comment - We believe that there is a race condition between the vsm get call https://github.com/migtools/volume-snapshot-mover/blob/master/controllers/replicationdestination.go#L124 for replicationDestination and its creation. This seems like its not 100% reproducible. 

            Daniel Gur added a comment -

            Proposing Release Blocker - as we should not release regressions

            Daniel Gur added a comment - Proposing Release Blocker - as we should not release regressions

            Maya Peretz added a comment - - edited

            issue is still there on build 1.1.1-36:

            - apiVersion: datamover.oadp.openshift.io/v1alpha1
              kind: VolumeSnapshotRestore
              metadata:
                creationTimestamp: "2022-10-30T13:19:15Z"
                generateName: vsr-
                generation: 1
                labels:
                  velero.io/persistent-volume-claim-name: cassandra-data-cassandra-0
                  velero.io/restore-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c
                name: vsr-tqxlm
                namespace: cassandra-ns
                resourceVersion: "270595"
                uid: 2dc265e4-ea3c-42f0-9d8a-7495c5d7a722
              spec:
                protectedNamespace: openshift-adp
                resticSecretRef:
                  name: ts-dpa-1-volsync-restic
                volumeSnapshotMoverBackupRef:
                  resticrepository: s3:s3.amazonaws.com/oadpbucket151037/openshift-adp/snapcontent-0e22175a-e4d2-4c7f-b02d-4e6f87eecd63-pvc
                  sourcePVCData:
                    name: cassandra-data-cassandra-0
                    size: 1Gi
                    storageClassName: gp2-csi
                  volumeSnapshotClassName: example-snapclass
              status:
                conditions:
                - lastTransitionTime: "2022-10-30T13:19:16Z"
                  message: ReplicationDestination.volsync.backube "vsr-tqxlm-rep-dest" not found
                  reason: Error
                  status: "False"
                  type: Reconciled
                phase: Failed
            kind: List
            metadata:
              resourceVersion: ""
            [mperetz@fedora oadp-e2e-qe]$ PRESTAGE_INDEX_CONTAINS=$(oc get catsrc prestage-operators -n openshift-marketplace -ojsonpath='{.spec.image}' | grep -Eo "iib:[0-9]+")
            NVR=$(curl -k https://datagrepper.engineering.redhat.com/raw\?topic\=/topic/VirtualTopic.eng.ci.redhat-container-image.index.built\&contains\=${PRESTAGE_INDEX_CONTAINS}\&rows_per_page\=1 | jq -r '.raw_messages[0].msg.artifact.nvr')
            echo $NVR
              % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                             Dload  Upload   Total   Spent    Left  Speed
            100  3383  100  3383    0     0   2911      0  0:00:01  0:00:01 --:--:--  2911
            oadp-operator-bundle-container-1.1.1-36
            [mperetz@fedora oadp-e2e-qe]$ 
             

            cluster used: AWS

            application used: https://gitlab.cee.redhat.com/app-mig/oadp-e2e-qe/-/tree/master/sample-applications/ansible/roles/ocp-cassandra

             

            DPA:

            apiVersion: v1
            items:
            - apiVersion: oadp.openshift.io/v1alpha1
              kind: DataProtectionApplication
              metadata:
                creationTimestamp: "2022-10-30T13:03:02Z"
                generation: 1
                name: ts-dpa
                namespace: openshift-adp
                resourceVersion: "261775"
                uid: 8696402b-4451-49f0-a5de-9d2335e03548
              spec:
                backupLocations:
                - velero:
                    config:
                      region: us-east-2
                    credential:
                      key: cloud
                      name: cloud-credentials
                    default: true
                    objectStorage:
                      bucket: oadpbucket151037
                      prefix: velero-e2e-29f2f374-5853-11ed-8778-902e163f806c
                    provider: aws
                configuration:
                  restic:
                    enable: true
                    podConfig:
                      resourceAllocations: {}
                  velero:
                    defaultPlugins:
                    - openshift
                    - aws
                    - kubevirt
                    - csi
                features:
                  dataMover:
                    enable: true
                podDnsConfig: {}
                snapshotLocations: []
              status:
                conditions:
                - lastTransitionTime: "2022-10-30T13:03:02Z"
                  message: Reconcile complete
                  reason: Complete
                  status: "True"
                  type: Reconciled
            kind: List
            metadata:
              resourceVersion: ""
             

            Volumesnapshots:

            [mperetz@fedora oadp-e2e-qe]$ oc get vs -A -o yaml
            apiVersion: v1
            items:
            - apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                annotations:
                  velero.io/csi-driver-name: ebs.csi.aws.com
                  velero.io/csi-volumesnapshot-handle: snap-03441902518cdfd0e
                  velero.io/csi-vsc-deletion-policy: Retain
                  velero.io/vsi-volumesnapshot-restore-size: 1Gi
                creationTimestamp: "2022-10-30T13:19:29Z"
                finalizers:
                - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
                generation: 1
                labels:
                  velero.io/backup-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c
                  velero.io/restore-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c
                name: velero-cassandra-data-cassandra-0-qr5vz
                namespace: cassandra-ns
                resourceVersion: "278926"
                uid: 571636fd-7a1c-4a91-b38b-65f3ded28dd4
              spec:
                source:
                  volumeSnapshotContentName: velero-velero-cassandra-data-cassandra-0-qr5vz-qrnxl
                volumeSnapshotClassName: example-snapclass
              status:
                boundVolumeSnapshotContentName: velero-velero-cassandra-data-cassandra-0-qr5vz-qrnxl
                error:
                  message: 'Failed to check and update snapshot content: failed to list snapshot
                    for content velero-velero-cassandra-data-cassandra-0-qr5vz-qrnxl: "rpc error:
                    code = Internal desc = Could not list snapshots: InvalidParameterValue: Value
                    ( 0 ) for parameter maxResults is invalid. Expecting a value greater than
                    5.\n\tstatus code: 400, request id: 1be39728-5ab0-47fe-a3af-3f9c66acce36"'
                  time: "2022-10-30T13:28:13Z"
                readyToUse: false
            - apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                annotations:
                  velero.io/csi-driver-name: ebs.csi.aws.com
                  velero.io/csi-volumesnapshot-handle: snap-02dfac73656cef3ac
                  velero.io/csi-vsc-deletion-policy: Retain
                  velero.io/vsi-volumesnapshot-restore-size: 1Gi
                creationTimestamp: "2022-10-30T13:19:32Z"
                finalizers:
                - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
                generation: 1
                labels:
                  velero.io/backup-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c
                  velero.io/restore-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c
                name: velero-cassandra-data-cassandra-1-2s65n
                namespace: cassandra-ns
                resourceVersion: "278921"
                uid: c69ee28d-17b9-4fb5-afb9-4e4dc0eabbb2
              spec:
                source:
                  volumeSnapshotContentName: velero-velero-cassandra-data-cassandra-1-2s65n-6kc2t
                volumeSnapshotClassName: example-snapclass
              status:
                boundVolumeSnapshotContentName: velero-velero-cassandra-data-cassandra-1-2s65n-6kc2t
                error:
                  message: 'Failed to check and update snapshot content: failed to list snapshot
                    for content velero-velero-cassandra-data-cassandra-1-2s65n-6kc2t: "rpc error:
                    code = Internal desc = Could not list snapshots: InvalidParameterValue: Value
                    ( 0 ) for parameter maxResults is invalid. Expecting a value greater than
                    5.\n\tstatus code: 400, request id: 820f2cba-ed25-417a-be29-141da993eeac"'
                  time: "2022-10-30T13:28:12Z"
                readyToUse: false
            - apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                annotations:
                  velero.io/csi-driver-name: ebs.csi.aws.com
                  velero.io/csi-volumesnapshot-handle: snap-0a4fda618e0ca1f5f
                  velero.io/csi-vsc-deletion-policy: Retain
                  velero.io/vsi-volumesnapshot-restore-size: 1Gi
                creationTimestamp: "2022-10-30T13:19:35Z"
                finalizers:
                - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
                generation: 1
                labels:
                  velero.io/backup-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c
                  velero.io/restore-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c
                name: velero-cassandra-data-cassandra-2-99wnj
                namespace: cassandra-ns
                resourceVersion: "278912"
                uid: 9708dc68-70b8-4a1b-8939-01b3b75cfd4a
              spec:
                source:
                  volumeSnapshotContentName: velero-velero-cassandra-data-cassandra-2-99wnj-q7n7l
                volumeSnapshotClassName: example-snapclass
              status:
                boundVolumeSnapshotContentName: velero-velero-cassandra-data-cassandra-2-99wnj-q7n7l
                error:
                  message: 'Failed to check and update snapshot content: failed to list snapshot
                    for content velero-velero-cassandra-data-cassandra-2-99wnj-q7n7l: "rpc error:
                    code = Internal desc = Could not list snapshots: InvalidParameterValue: Value
                    ( 0 ) for parameter maxResults is invalid. Expecting a value greater than
                    5.\n\tstatus code: 400, request id: c27c4ccf-19cc-43a3-bcec-3bf6f839ff17"'
                  time: "2022-10-30T13:28:12Z"
                readyToUse: false
            - apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                creationTimestamp: "2022-10-30T12:22:08Z"
                finalizers:
                - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
                generation: 1
                labels:
                  datamover.oadp.openshift.io/vsb: vsb-djs2j
                name: snapcontent-7e8c2bd2-3e63-4ef2-afe0-d4ecbb7bc753-clone-volumesnapshot
                namespace: openshift-adp
                resourceVersion: "240890"
                uid: e3951105-96f9-4673-90a5-505598db1676
              spec:
                source:
                  volumeSnapshotContentName: snapcontent-7e8c2bd2-3e63-4ef2-afe0-d4ecbb7bc753-clone
              status:
                boundVolumeSnapshotContentName: snapcontent-7e8c2bd2-3e63-4ef2-afe0-d4ecbb7bc753-clone
                creationTime: "2022-10-30T12:20:42Z"
                readyToUse: true
                restoreSize: 1Gi
            - apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                creationTimestamp: "2022-10-30T13:19:39Z"
                finalizers:
                - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
                generation: 1
                labels:
                  app.kubernetes.io/created-by: volsync
                name: volsync-vsr-fj2r9-rep-dest-dest-20221030131939
                namespace: openshift-adp
                ownerReferences:
                - apiVersion: volsync.backube/v1alpha1
                  blockOwnerDeletion: true
                  controller: true
                  kind: ReplicationDestination
                  name: vsr-fj2r9-rep-dest
                  uid: 5febec5c-5104-4888-b240-dcb7c11568af
                resourceVersion: "272229"
                uid: 8546f62c-78b0-45fa-9b6e-91c208182b17
              spec:
                source:
                  persistentVolumeClaimName: volsync-vsr-fj2r9-rep-dest-dest
                volumeSnapshotClassName: example-snapclass
              status:
                boundVolumeSnapshotContentName: snapcontent-8546f62c-78b0-45fa-9b6e-91c208182b17
                creationTime: "2022-10-30T13:19:43Z"
                readyToUse: true
                restoreSize: 1Gi
            - apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                creationTimestamp: "2022-10-30T13:19:39Z"
                finalizers:
                - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
                generation: 1
                labels:
                  app.kubernetes.io/created-by: volsync
                name: volsync-vsr-qmvkx-rep-dest-dest-20221030131939
                namespace: openshift-adp
                ownerReferences:
                - apiVersion: volsync.backube/v1alpha1
                  blockOwnerDeletion: true
                  controller: true
                  kind: ReplicationDestination
                  name: vsr-qmvkx-rep-dest
                  uid: 1e261dbc-c367-4cd0-95c6-360ef6c069cf
                resourceVersion: "272019"
                uid: 03b3d494-2d5b-43d8-af37-6d33b7627ead
              spec:
                source:
                  persistentVolumeClaimName: volsync-vsr-qmvkx-rep-dest-dest
                volumeSnapshotClassName: example-snapclass
              status:
                boundVolumeSnapshotContentName: snapcontent-03b3d494-2d5b-43d8-af37-6d33b7627ead
                creationTime: "2022-10-30T13:19:43Z"
                readyToUse: true
                restoreSize: 1Gi
            - apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                creationTimestamp: "2022-10-30T13:19:29Z"
                finalizers:
                - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
                generation: 1
                labels:
                  app.kubernetes.io/created-by: volsync
                name: volsync-vsr-tqxlm-rep-dest-dest-20221030131929
                namespace: openshift-adp
                ownerReferences:
                - apiVersion: volsync.backube/v1alpha1
                  blockOwnerDeletion: true
                  controller: true
                  kind: ReplicationDestination
                  name: vsr-tqxlm-rep-dest
                  uid: e927c49e-9e33-4c6d-8371-edd16dc9e30d
                resourceVersion: "271648"
                uid: 2dd68986-851f-4684-849f-bcbbab9f15a3
              spec:
                source:
                  persistentVolumeClaimName: volsync-vsr-tqxlm-rep-dest-dest
                volumeSnapshotClassName: example-snapclass
              status:
                boundVolumeSnapshotContentName: snapcontent-2dd68986-851f-4684-849f-bcbbab9f15a3
                creationTime: "2022-10-30T13:19:30Z"
                readyToUse: true
                restoreSize: 1Gi
            kind: List
            metadata:
              resourceVersion: ""
             

             

            Maya Peretz added a comment - - edited issue is still there on build 1.1.1-36: - apiVersion: datamover.oadp.openshift.io/v1alpha1   kind: VolumeSnapshotRestore   metadata:     creationTimestamp: "2022-10-30T13:19:15Z"     generateName: vsr-     generation: 1     labels:       velero.io/persistent-volume-claim-name: cassandra-data-cassandra-0       velero.io/restore-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c     name: vsr-tqxlm     namespace: cassandra-ns     resourceVersion: "270595"     uid: 2dc265e4-ea3c-42f0-9d8a-7495c5d7a722   spec:     protectedNamespace: openshift-adp     resticSecretRef:       name: ts-dpa-1-volsync-restic     volumeSnapshotMoverBackupRef:       resticrepository: s3:s3.amazonaws.com/oadpbucket151037/openshift-adp/snapcontent-0e22175a-e4d2-4c7f-b02d-4e6f87eecd63-pvc       sourcePVCData:         name: cassandra-data-cassandra-0         size: 1Gi         storageClassName: gp2-csi       volumeSnapshotClassName: example-snapclass   status:     conditions:     - lastTransitionTime: "2022-10-30T13:19:16Z"       message: ReplicationDestination.volsync.backube "vsr-tqxlm-rep-dest" not found       reason: Error       status: "False"       type: Reconciled     phase: Failed kind: List metadata:   resourceVersion: "" [mperetz@fedora oadp-e2e-qe]$ PRESTAGE_INDEX_CONTAINS=$(oc get catsrc prestage-operators -n openshift-marketplace -ojsonpath='{.spec.image}' | grep -Eo "iib:[0-9]+") NVR=$(curl -k https://datagrepper.engineering.redhat.com/raw\?topic\=/topic/VirtualTopic.eng.ci.redhat-container-image.index.built\&contains\=${PRESTAGE_INDEX_CONTAINS}\&rows_per_page\=1 | jq -r '.raw_messages[0].msg.artifact.nvr') echo $NVR   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed 100  3383  100  3383    0     0   2911      0  0:00:01  0:00:01 --:--:--  2911 oadp-operator-bundle-container-1.1.1-36 [mperetz@fedora oadp-e2e-qe]$  cluster used: AWS application used: https://gitlab.cee.redhat.com/app-mig/oadp-e2e-qe/-/tree/master/sample-applications/ansible/roles/ocp-cassandra   DPA: apiVersion: v1 items: - apiVersion: oadp.openshift.io/v1alpha1   kind: DataProtectionApplication   metadata:     creationTimestamp: "2022-10-30T13:03:02Z"     generation: 1     name: ts-dpa     namespace: openshift-adp     resourceVersion: "261775"     uid: 8696402b-4451-49f0-a5de-9d2335e03548   spec:     backupLocations:     - velero:         config:           region: us-east-2         credential:           key: cloud           name: cloud-credentials         default: true         objectStorage:           bucket: oadpbucket151037           prefix: velero-e2e-29f2f374-5853-11ed-8778-902e163f806c         provider: aws     configuration:       restic:         enable: true         podConfig:           resourceAllocations: {}       velero:         defaultPlugins:         - openshift         - aws         - kubevirt         - csi     features:       dataMover:         enable: true     podDnsConfig: {}     snapshotLocations: []   status:     conditions:     - lastTransitionTime: "2022-10-30T13:03:02Z"       message: Reconcile complete       reason: Complete       status: "True"       type: Reconciled kind: List metadata:   resourceVersion: "" Volumesnapshots: [mperetz@fedora oadp-e2e-qe]$ oc get vs -A -o yaml apiVersion: v1 items: - apiVersion: snapshot.storage.k8s.io/v1   kind: VolumeSnapshot   metadata:     annotations:       velero.io/csi-driver-name: ebs.csi.aws.com       velero.io/csi-volumesnapshot-handle: snap-03441902518cdfd0e       velero.io/csi-vsc-deletion-policy: Retain       velero.io/vsi-volumesnapshot-restore-size: 1Gi     creationTimestamp: "2022-10-30T13:19:29Z"     finalizers:     - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection     generation: 1     labels:       velero.io/backup-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c       velero.io/restore-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c     name: velero-cassandra-data-cassandra-0-qr5vz     namespace: cassandra-ns     resourceVersion: "278926"     uid: 571636fd-7a1c-4a91-b38b-65f3ded28dd4   spec:     source:       volumeSnapshotContentName: velero-velero-cassandra-data-cassandra-0-qr5vz-qrnxl     volumeSnapshotClassName: example-snapclass   status:     boundVolumeSnapshotContentName: velero-velero-cassandra-data-cassandra-0-qr5vz-qrnxl     error:       message: 'Failed to check and update snapshot content: failed to list snapshot         for content velero-velero-cassandra-data-cassandra-0-qr5vz-qrnxl: "rpc error:         code = Internal desc = Could not list snapshots: InvalidParameterValue: Value         ( 0 ) for parameter maxResults is invalid. Expecting a value greater than         5.\n\tstatus code: 400, request id: 1be39728-5ab0-47fe-a3af-3f9c66acce36"'       time: "2022-10-30T13:28:13Z"     readyToUse: false - apiVersion: snapshot.storage.k8s.io/v1   kind: VolumeSnapshot   metadata:     annotations:       velero.io/csi-driver-name: ebs.csi.aws.com       velero.io/csi-volumesnapshot-handle: snap-02dfac73656cef3ac       velero.io/csi-vsc-deletion-policy: Retain       velero.io/vsi-volumesnapshot-restore-size: 1Gi     creationTimestamp: "2022-10-30T13:19:32Z"     finalizers:     - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection     generation: 1     labels:       velero.io/backup-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c       velero.io/restore-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c     name: velero-cassandra-data-cassandra-1-2s65n     namespace: cassandra-ns     resourceVersion: "278921"     uid: c69ee28d-17b9-4fb5-afb9-4e4dc0eabbb2   spec:     source:       volumeSnapshotContentName: velero-velero-cassandra-data-cassandra-1-2s65n-6kc2t     volumeSnapshotClassName: example-snapclass   status:     boundVolumeSnapshotContentName: velero-velero-cassandra-data-cassandra-1-2s65n-6kc2t     error:       message: 'Failed to check and update snapshot content: failed to list snapshot         for content velero-velero-cassandra-data-cassandra-1-2s65n-6kc2t: "rpc error:         code = Internal desc = Could not list snapshots: InvalidParameterValue: Value         ( 0 ) for parameter maxResults is invalid. Expecting a value greater than         5.\n\tstatus code: 400, request id: 820f2cba-ed25-417a-be29-141da993eeac"'       time: "2022-10-30T13:28:12Z"     readyToUse: false - apiVersion: snapshot.storage.k8s.io/v1   kind: VolumeSnapshot   metadata:     annotations:       velero.io/csi-driver-name: ebs.csi.aws.com       velero.io/csi-volumesnapshot-handle: snap-0a4fda618e0ca1f5f       velero.io/csi-vsc-deletion-policy: Retain       velero.io/vsi-volumesnapshot-restore-size: 1Gi     creationTimestamp: "2022-10-30T13:19:35Z"     finalizers:     - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection     generation: 1     labels:       velero.io/backup-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c       velero.io/restore-name: cassandra-e2e-2c11fd29-5853-11ed-8778-902e163f806c     name: velero-cassandra-data-cassandra-2-99wnj     namespace: cassandra-ns     resourceVersion: "278912"     uid: 9708dc68-70b8-4a1b-8939-01b3b75cfd4a   spec:     source:       volumeSnapshotContentName: velero-velero-cassandra-data-cassandra-2-99wnj-q7n7l     volumeSnapshotClassName: example-snapclass   status:     boundVolumeSnapshotContentName: velero-velero-cassandra-data-cassandra-2-99wnj-q7n7l     error:       message: 'Failed to check and update snapshot content: failed to list snapshot         for content velero-velero-cassandra-data-cassandra-2-99wnj-q7n7l: "rpc error:         code = Internal desc = Could not list snapshots: InvalidParameterValue: Value         ( 0 ) for parameter maxResults is invalid. Expecting a value greater than         5.\n\tstatus code: 400, request id: c27c4ccf-19cc-43a3-bcec-3bf6f839ff17"'       time: "2022-10-30T13:28:12Z"     readyToUse: false - apiVersion: snapshot.storage.k8s.io/v1   kind: VolumeSnapshot   metadata:     creationTimestamp: "2022-10-30T12:22:08Z"     finalizers:     - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection     generation: 1     labels:       datamover.oadp.openshift.io/vsb: vsb-djs2j     name: snapcontent-7e8c2bd2-3e63-4ef2-afe0-d4ecbb7bc753-clone-volumesnapshot     namespace: openshift-adp     resourceVersion: "240890"     uid: e3951105-96f9-4673-90a5-505598db1676   spec:     source:       volumeSnapshotContentName: snapcontent-7e8c2bd2-3e63-4ef2-afe0-d4ecbb7bc753-clone   status:     boundVolumeSnapshotContentName: snapcontent-7e8c2bd2-3e63-4ef2-afe0-d4ecbb7bc753-clone     creationTime: "2022-10-30T12:20:42Z"     readyToUse: true     restoreSize: 1Gi - apiVersion: snapshot.storage.k8s.io/v1   kind: VolumeSnapshot   metadata:     creationTimestamp: "2022-10-30T13:19:39Z"     finalizers:     - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection     generation: 1     labels:       app.kubernetes.io/created-by: volsync     name: volsync-vsr-fj2r9-rep-dest-dest-20221030131939     namespace: openshift-adp     ownerReferences:     - apiVersion: volsync.backube/v1alpha1       blockOwnerDeletion: true       controller: true       kind: ReplicationDestination       name: vsr-fj2r9-rep-dest       uid: 5febec5c-5104-4888-b240-dcb7c11568af     resourceVersion: "272229"     uid: 8546f62c-78b0-45fa-9b6e-91c208182b17   spec:     source:       persistentVolumeClaimName: volsync-vsr-fj2r9-rep-dest-dest     volumeSnapshotClassName: example-snapclass   status:     boundVolumeSnapshotContentName: snapcontent-8546f62c-78b0-45fa-9b6e-91c208182b17     creationTime: "2022-10-30T13:19:43Z"     readyToUse: true     restoreSize: 1Gi - apiVersion: snapshot.storage.k8s.io/v1   kind: VolumeSnapshot   metadata:     creationTimestamp: "2022-10-30T13:19:39Z"     finalizers:     - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection     generation: 1     labels:       app.kubernetes.io/created-by: volsync     name: volsync-vsr-qmvkx-rep-dest-dest-20221030131939     namespace: openshift-adp     ownerReferences:     - apiVersion: volsync.backube/v1alpha1       blockOwnerDeletion: true       controller: true       kind: ReplicationDestination       name: vsr-qmvkx-rep-dest       uid: 1e261dbc-c367-4cd0-95c6-360ef6c069cf     resourceVersion: "272019"     uid: 03b3d494-2d5b-43d8-af37-6d33b7627ead   spec:     source:       persistentVolumeClaimName: volsync-vsr-qmvkx-rep-dest-dest     volumeSnapshotClassName: example-snapclass   status:     boundVolumeSnapshotContentName: snapcontent-03b3d494-2d5b-43d8-af37-6d33b7627ead     creationTime: "2022-10-30T13:19:43Z"     readyToUse: true     restoreSize: 1Gi - apiVersion: snapshot.storage.k8s.io/v1   kind: VolumeSnapshot   metadata:     creationTimestamp: "2022-10-30T13:19:29Z"     finalizers:     - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection     generation: 1     labels:       app.kubernetes.io/created-by: volsync     name: volsync-vsr-tqxlm-rep-dest-dest-20221030131929     namespace: openshift-adp     ownerReferences:     - apiVersion: volsync.backube/v1alpha1       blockOwnerDeletion: true       controller: true       kind: ReplicationDestination       name: vsr-tqxlm-rep-dest       uid: e927c49e-9e33-4c6d-8371-edd16dc9e30d     resourceVersion: "271648"     uid: 2dd68986-851f-4684-849f-bcbbab9f15a3   spec:     source:       persistentVolumeClaimName: volsync-vsr-tqxlm-rep-dest-dest     volumeSnapshotClassName: example-snapclass   status:     boundVolumeSnapshotContentName: snapcontent-2dd68986-851f-4684-849f-bcbbab9f15a3     creationTime: "2022-10-30T13:19:30Z"     readyToUse: true     restoreSize: 1Gi kind: List metadata:   resourceVersion: ""  

              spampatt@redhat.com Shubham Pampattiwar
              mperetz@redhat.com Maya Peretz
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: