-
Bug
-
Resolution: Won't Do
-
Normal
-
OADP 1.2.0
-
1
-
False
-
-
False
-
oadp-operator-bundle-container-1.2.4-2
-
ToDo
-
-
-
0
-
0.000
-
Very Likely
-
0
-
None
-
Unset
-
Unknown
-
No
Description of problem:
In our testing we have observed that the VolumeSnapshotContent sometime takes more than 10 mins to become ready. Most of the time we hit timeout issue because of this. This issue is intermittent. Attached error below:-
error: message: 'Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a: "snapshot controller failed to update snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a\": the object has been modified; please apply your changes to the latest version and try again"' time: "2023-05-24T12:02:15Z"
Version-Release number of selected component (if applicable):
OADP 1.2.0 - 79
How reproducible:
Intermittent
Steps to Reproduce:
1. Deploy an stateful application
2. Execute a backup with DataMover/CSI
Actual results:
VolumeSnapshotContent takes more than 10mins to become ready due it failed to remove the annotation.
$ oc get vsc snapcontent-b572e497-ab51-4938-9115-2a8759e7d5dc
NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT VOLUMESNAPSHOTNAMESPACE AGE
snapcontent-b572e497-ab51-4938-9115-2a8759e7d5dc false 1073741824 Retain ebs.csi.aws.com example-snapclass velero-volume6-4gq4w minimal-8csivol 10m
Expected results:
VolumeSnapshotContent shouldn't take too much time to become ready.
Additional info:
oc get vsc snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a -o yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: creationTimestamp: "2023-05-24T12:02:10Z" finalizers: snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection generation: 1 labels: velero.io/backup-name: test-backup1 managedFields: apiVersion: snapshot.storage.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: {} v:"snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection": {} f:spec: .: {} f:deletionPolicy: {} f:driver: {} f:source: .: {} f:volumeHandle: {} f:volumeSnapshotClassName: {} f:volumeSnapshotRef: .: {} f:apiVersion: {} f:kind: {} f:name: {} f:namespace: {} f:resourceVersion: {} f:uid: {} manager: snapshot-controller operation: Update time: "2023-05-24T12:02:10Z" apiVersion: snapshot.storage.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: .: {} f:creationTime: {} f:error: .: {} f:message: {} f:time: {} f:readyToUse: {} f:restoreSize: {} f:snapshotHandle: {} manager: csi-snapshotter operation: Update subresource: status time: "2023-05-24T12:02:15Z" apiVersion: snapshot.storage.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:labels: .: {} f:velero.io/backup-name: {} manager: velero-plugin-for-csi operation: Update time: "2023-05-24T12:02:15Z" name: snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a resourceVersion: "209174" uid: 3ae91fce-9078-46ba-9af7-8483d32fd0d5 spec: deletionPolicy: Retain driver: ebs.csi.aws.com source: volumeHandle: vol-039e6374bdf0922bd volumeSnapshotClassName: example-snapclass volumeSnapshotRef: apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot name: velero-volume3-57wqb namespace: minimal-8csivol resourceVersion: "209053" uid: 4125d62b-dab7-4d7d-af91-7b5c3dbee52a status: creationTime: 1684929730484000000 error: message: 'Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a: "snapshot controller failed to update snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a\": the object has been modified; please apply your changes to the latest version and try again"' time: "2023-05-24T12:02:15Z" readyToUse: false restoreSize: 1073741824 snapshotHandle: snap-0ffb8994814f826a0
Slack discussion:- https://redhat-internal.slack.com/archives/C0144ECKUJ0/p1684931431092739
- links to
-
RHBA-2023:122996 OpenShift API for Data Protection (OADP) 1.2.4 security and bug fix update
- mentioned on