-
Bug
-
Resolution: Done-Errata
-
Blocker
-
OADP 1.2.0
-
False
-
-
False
-
oadp-operator-bundle-container-1.2.1-4
-
ToDo
-
-
-
0
-
0
-
Very Likely
-
0
-
None
-
Unset
-
Unknown
-
No
Description of problem:
Create a DPA CR with maxConcurrentRestoreVolumes set as 1. I observed multiple VSR resource had batchingStatus as "Processing". Velero should only allow processing 1 VSR at a time.
Version-Release number of selected component (if applicable):
OADP 1.2.0 (brew.registry.redhat.io/rh-osbs/iib:474028)
How reproducible:
Always(Consistently)
Since stage build its intermittent. its around 25% reproducible.
Steps to Reproduce:
1. Create stateful app
$ oc create -f https://raw.githubusercontent.com/openshift/oadp-operator/master/tests/e2e/sample-applications/minimal-8csivol/list.yaml
2. Create a dpa with datamover enabled and set maxConcurrentRestoreVolumes as 1.
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: ts namespace: openshift-adp spec: backupLocations: - velero: config: profile: default region: us-east-2 credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: oadpbucket196925 prefix: velero provider: aws configuration: velero: defaultPlugins: - openshift - aws - csi - vsm features: dataMover: enable: true maxConcurrentBackupVolumes: "1" maxConcurrentRestoreVolumes: "1"
3. Create a backup
4. Delete app ns
5. Restore app namespace
6. Observe VSR status.batchingStatus
Actual results:
Multiple VolumesnapshotRestore resources had batchingStatus as processing.
$ oc get vsr -n minimal-8csivol -o yaml apiVersion: v1 items: - apiVersion: datamover.oadp.openshift.io/v1alpha1 kind: VolumeSnapshotRestore metadata: creationTimestamp: "2023-04-18T10:23:26Z" finalizers: - oadp.openshift.io/oadp-datamover generateName: vsr- generation: 1 labels: velero.io/backup-name: test-datamover-batching velero.io/persistent-volume-claim-name: volume6 velero.io/restore-name: test-batching-restore name: vsr-bhq4x namespace: minimal-8csivol resourceVersion: "221882" uid: 0bc931e9-e890-4c96-9399-77024385ae6a spec: protectedNamespace: openshift-adp resticSecretRef: name: ts-1-volsync-restic volumeSnapshotMoverBackupRef: resticrepository: s3:s3.amazonaws.com/oadpbucket196925/openshift-adp/snapcontent-6d5f30b6-222a-45a0-9999-76ed6d0092c6-pvc sourcePVCData: name: volume6 size: 1Gi storageClassName: gp2-csi volumeSnapshotClassName: example-snapclass status: batchingStatus: Queued conditions: - lastTransitionTime: "2023-04-18T10:23:26Z" message: 'Operation cannot be fulfilled on volumesnapshotrestores.datamover.oadp.openshift.io "vsr-bhq4x": the object has been modified; please apply your changes to the latest version and try again' reason: Error status: "False" type: Reconciled replicationDestinationData: {} - apiVersion: datamover.oadp.openshift.io/v1alpha1 kind: VolumeSnapshotRestore metadata: creationTimestamp: "2023-04-18T10:23:21Z" finalizers: - oadp.openshift.io/oadp-datamover generateName: vsr- generation: 1 labels: velero.io/backup-name: test-datamover-batching velero.io/persistent-volume-claim-name: volume2 velero.io/restore-name: test-batching-restore managedFields: - apiVersion: datamover.oadp.openshift.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: {} v:"oadp.openshift.io/oadp-datamover": {} manager: manager operation: Update time: "2023-04-18T10:23:21Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/persistent-volume-claim-name: {} f:velero.io/restore-name: {} f:spec: .: {} f:protectedNamespace: {} f:resticSecretRef: .: {} f:name: {} f:volumeSnapshotMoverBackupRef: .: {} f:resticrepository: {} f:sourcePVCData: .: {} f:name: {} f:size: {} f:storageClassName: {} f:volumeSnapshotClassName: {} manager: velero-plugin-for-vsm operation: Update time: "2023-04-18T10:23:21Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:status: .: {} f:batchingStatus: {} f:completionTimestamp: {} f:conditions: {} f:phase: {} f:replicationDestinationData: .: {} f:completionTimestamp: {} f:name: {} f:startTimestamp: {} f:startTimestamp: {} manager: manager operation: Update subresource: status time: "2023-04-18T10:24:02Z" name: vsr-d4l9h namespace: minimal-8csivol resourceVersion: "222654" uid: 394d091b-c21d-491e-86fd-4ea0c7ad4714 spec: protectedNamespace: openshift-adp resticSecretRef: name: ts-1-volsync-restic volumeSnapshotMoverBackupRef: resticrepository: s3:s3.amazonaws.com/oadpbucket196925/openshift-adp/snapcontent-f9618a70-c52e-4c70-b800-3fc86abc1883-pvc sourcePVCData: name: volume2 size: 1Gi storageClassName: gp2-csi volumeSnapshotClassName: example-snapclass status: batchingStatus: Completed completionTimestamp: "2023-04-18T10:24:02Z" conditions: - lastTransitionTime: "2023-04-18T10:23:31Z" message: Reconcile complete reason: Complete status: "True" type: Reconciled phase: SnapshotRestoreDone replicationDestinationData: completionTimestamp: "2023-04-18T10:23:40Z" name: vsr-d4l9h-rep-dest startTimestamp: "2023-04-18T10:23:26Z" startTimestamp: "2023-04-18T10:23:26Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 kind: VolumeSnapshotRestore metadata: creationTimestamp: "2023-04-18T10:23:29Z" finalizers: - oadp.openshift.io/oadp-datamover generateName: vsr- generation: 1 labels: velero.io/backup-name: test-datamover-batching velero.io/persistent-volume-claim-name: volume1 velero.io/restore-name: test-batching-restore managedFields: - apiVersion: datamover.oadp.openshift.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: {} v:"oadp.openshift.io/oadp-datamover": {} manager: manager operation: Update time: "2023-04-18T10:23:29Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/persistent-volume-claim-name: {} f:velero.io/restore-name: {} f:spec: .: {} f:protectedNamespace: {} f:resticSecretRef: .: {} f:name: {} f:volumeSnapshotMoverBackupRef: .: {} f:resticrepository: {} f:sourcePVCData: .: {} f:name: {} f:size: {} f:storageClassName: {} f:volumeSnapshotClassName: {} manager: velero-plugin-for-vsm operation: Update time: "2023-04-18T10:23:29Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:status: .: {} f:batchingStatus: {} f:conditions: {} f:phase: {} f:replicationDestinationData: .: {} f:name: {} f:startTimestamp: {} f:startTimestamp: {} manager: manager operation: Update subresource: status time: "2023-04-18T10:23:59Z" name: vsr-g2xwt namespace: minimal-8csivol resourceVersion: "222619" uid: 0e6df353-b8be-473a-b4c1-51cca83a8fef spec: protectedNamespace: openshift-adp resticSecretRef: name: ts-1-volsync-restic volumeSnapshotMoverBackupRef: resticrepository: s3:s3.amazonaws.com/oadpbucket196925/openshift-adp/snapcontent-d4eb65b2-61bb-4302-8f0c-c9b94fc50b41-pvc sourcePVCData: name: volume1 size: 1Gi storageClassName: gp2-csi volumeSnapshotClassName: example-snapclass status: batchingStatus: Processing conditions: - lastTransitionTime: "2023-04-18T10:23:59Z" message: Reconcile complete reason: Complete status: "True" type: Reconciled phase: InProgress replicationDestinationData: name: vsr-g2xwt-rep-dest startTimestamp: "2023-04-18T10:23:54Z" startTimestamp: "2023-04-18T10:23:54Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 kind: VolumeSnapshotRestore metadata: creationTimestamp: "2023-04-18T10:23:39Z" finalizers: - oadp.openshift.io/oadp-datamover generateName: vsr- generation: 1 labels: velero.io/backup-name: test-datamover-batching velero.io/persistent-volume-claim-name: volume7 velero.io/restore-name: test-batching-restore name: vsr-j8g2d namespace: minimal-8csivol resourceVersion: "222689" uid: d54dbce3-ddd8-478b-bb07-4c00120437e9 spec: protectedNamespace: openshift-adp resticSecretRef: name: ts-1-volsync-restic volumeSnapshotMoverBackupRef: resticrepository: s3:s3.amazonaws.com/oadpbucket196925/openshift-adp/snapcontent-3a19ac93-3378-4068-ae3c-ef46fafc11bb-pvc sourcePVCData: name: volume7 size: 1Gi storageClassName: gp2-csi volumeSnapshotClassName: example-snapclass status: batchingStatus: Processing conditions: - lastTransitionTime: "2023-04-18T10:23:39Z" message: 'Operation cannot be fulfilled on volumesnapshotrestores.datamover.oadp.openshift.io "vsr-j8g2d": the object has been modified; please apply your changes to the latest version and try again' reason: Error status: "False" type: Reconciled replicationDestinationData: {} startTimestamp: "2023-04-18T10:24:04Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 kind: VolumeSnapshotRestore metadata: creationTimestamp: "2023-04-18T10:23:34Z" finalizers: - oadp.openshift.io/oadp-datamover generateName: vsr- generation: 1 labels: velero.io/backup-name: test-datamover-batching velero.io/persistent-volume-claim-name: volume3 velero.io/restore-name: test-batching-restore name: vsr-mvgcq namespace: minimal-8csivol resourceVersion: "222622" uid: 52e6eddd-948f-4747-bf94-dd8cde672bf1 spec: protectedNamespace: openshift-adp resticSecretRef: name: ts-1-volsync-restic volumeSnapshotMoverBackupRef: resticrepository: s3:s3.amazonaws.com/oadpbucket196925/openshift-adp/snapcontent-bb5af687-38a1-4a69-92bb-5fd3bd44791c-pvc sourcePVCData: name: volume3 size: 1Gi storageClassName: gp2-csi volumeSnapshotClassName: example-snapclass status: batchingStatus: Processing conditions: - lastTransitionTime: "2023-04-18T10:23:59Z" message: Reconcile complete reason: Complete status: "True" type: Reconciled phase: InProgress replicationDestinationData: name: vsr-mvgcq-rep-dest startTimestamp: "2023-04-18T10:23:54Z" startTimestamp: "2023-04-18T10:23:54Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 kind: VolumeSnapshotRestore metadata: creationTimestamp: "2023-04-18T10:23:31Z" finalizers: - oadp.openshift.io/oadp-datamover generateName: vsr- generation: 1 labels: velero.io/backup-name: test-datamover-batching velero.io/persistent-volume-claim-name: volume5 velero.io/restore-name: test-batching-restore name: vsr-p6kzc namespace: minimal-8csivol resourceVersion: "221967" uid: ace50c58-9eef-4922-9ddc-dee5ad2f6366 spec: protectedNamespace: openshift-adp resticSecretRef: name: ts-1-volsync-restic volumeSnapshotMoverBackupRef: resticrepository: s3:s3.amazonaws.com/oadpbucket196925/openshift-adp/snapcontent-cbbc580d-e853-451c-8164-bde03fad74ac-pvc sourcePVCData: name: volume5 size: 1Gi storageClassName: gp2-csi volumeSnapshotClassName: example-snapclass status: batchingStatus: Queued replicationDestinationData: {} startTimestamp: "2023-04-18T10:23:31Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 kind: VolumeSnapshotRestore metadata: creationTimestamp: "2023-04-18T10:23:36Z" finalizers: - oadp.openshift.io/oadp-datamover generateName: vsr- generation: 1 labels: velero.io/backup-name: test-datamover-batching velero.io/persistent-volume-claim-name: volume8 velero.io/restore-name: test-batching-restore name: vsr-phdr9 namespace: minimal-8csivol resourceVersion: "222062" uid: 62ba2c96-d1c9-457b-8f46-58aed9044561 spec: protectedNamespace: openshift-adp resticSecretRef: name: ts-1-volsync-restic volumeSnapshotMoverBackupRef: resticrepository: s3:s3.amazonaws.com/oadpbucket196925/openshift-adp/snapcontent-b2a182a5-88ba-4496-8270-c5c4244291a9-pvc sourcePVCData: name: volume8 size: 1Gi storageClassName: gp2-csi volumeSnapshotClassName: example-snapclass status: batchingStatus: Queued replicationDestinationData: {} startTimestamp: "2023-04-18T10:23:36Z" - apiVersion: datamover.oadp.openshift.io/v1alpha1 kind: VolumeSnapshotRestore metadata: creationTimestamp: "2023-04-18T10:23:24Z" finalizers: - oadp.openshift.io/oadp-datamover generateName: vsr- generation: 1 labels: velero.io/backup-name: test-datamover-batching velero.io/persistent-volume-claim-name: volume4 velero.io/restore-name: test-batching-restore name: vsr-qnsdh namespace: minimal-8csivol resourceVersion: "222462" uid: bcb8e0de-fb67-496a-9abe-2924c5e320b3 spec: protectedNamespace: openshift-adp resticSecretRef: name: ts-1-volsync-restic volumeSnapshotMoverBackupRef: resticrepository: s3:s3.amazonaws.com/oadpbucket196925/openshift-adp/snapcontent-0dbaa823-a231-4a01-aa0a-888d644e9338-pvc sourcePVCData: name: volume4 size: 1Gi storageClassName: gp2-csi volumeSnapshotClassName: example-snapclass status: batchingStatus: Processing conditions: - lastTransitionTime: "2023-04-18T10:23:54Z" message: Reconcile complete reason: Complete status: "True" type: Reconciled phase: InProgress replicationDestinationData: name: vsr-qnsdh-rep-dest startTimestamp: "2023-04-18T10:23:49Z" startTimestamp: "2023-04-18T10:23:49Z"
Expected results:
There should only 1 VSR resource in "processing" status at a time.
Additional info:
$ oc get pod volume-snapshot-mover-75b8799bbf-gkh82 -o yaml containers: - env: - name: DATAMOVER_CONCURRENT_BACKUP value: "1" - name: DATAMOVER_CONCURRENT_RESTORE value: "1"
- links to
- mentioned on
(7 mentioned on)
