Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-2038

VolumeSnapshotContent resource takes long time to become ready

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Normal Normal
    • OADP 1.2.6
    • OADP 1.2.0
    • data-mover
    • 1
    • False
    • Hide

      None

      Show
      None
    • False
    • oadp-operator-bundle-container-1.2.4-2
    • ToDo
    • 0
    • 0.000
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • No

      Description of problem:

      In our testing we have observed that the VolumeSnapshotContent sometime takes more than 10 mins to become ready. Most of the time we hit timeout issue because of this. This issue is intermittent. Attached error below:-

      error:
      message: 'Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a: "snapshot controller failed to update snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a\": the object has been modified; please apply your changes to the latest version and try again"'
      time: "2023-05-24T12:02:15Z"

      Version-Release number of selected component (if applicable):
      OADP 1.2.0 - 79

       

       

      How reproducible:
      Intermittent

       

      Steps to Reproduce:
      1. Deploy an stateful application

      2. Execute a backup with DataMover/CSI

      Actual results:

      VolumeSnapshotContent takes more than 10mins to become ready due it failed to remove the annotation.

      $ oc get vsc snapcontent-b572e497-ab51-4938-9115-2a8759e7d5dc 
      NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT VOLUMESNAPSHOTNAMESPACE AGE
      snapcontent-b572e497-ab51-4938-9115-2a8759e7d5dc false 1073741824 Retain ebs.csi.aws.com example-snapclass velero-volume6-4gq4w minimal-8csivol 10m

       

      Expected results:

      VolumeSnapshotContent shouldn't take too much time to become ready.

       

      Additional info:

      oc get vsc snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a -o yaml
      apiVersion: snapshot.storage.k8s.io/v1
      kind: VolumeSnapshotContent
      metadata:
      creationTimestamp: "2023-05-24T12:02:10Z"
      finalizers:
      
      snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
      generation: 1
      labels:
      velero.io/backup-name: test-backup1
      managedFields: apiVersion: snapshot.storage.k8s.io/v1
      fieldsType: FieldsV1
      fieldsV1:
      f:metadata:
      f:finalizers:
      .: {}
      v:"snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection": {}
      f:spec:
      .: {}
      f:deletionPolicy: {}
      f:driver: {}
      f:source:
      .: {}
      f:volumeHandle: {}
      f:volumeSnapshotClassName: {}
      f:volumeSnapshotRef:
      .: {}
      f:apiVersion: {}
      f:kind: {}
      f:name: {}
      f:namespace: {}
      f:resourceVersion: {}
      f:uid: {}
      manager: snapshot-controller
      operation: Update
      time: "2023-05-24T12:02:10Z" apiVersion: snapshot.storage.k8s.io/v1
      fieldsType: FieldsV1
      fieldsV1:
      f:status:
      .: {}
      f:creationTime: {}
      f:error:
      .: {}
      f:message: {}
      f:time: {}
      f:readyToUse: {}
      f:restoreSize: {}
      f:snapshotHandle: {}
      manager: csi-snapshotter
      operation: Update
      subresource: status
      time: "2023-05-24T12:02:15Z" apiVersion: snapshot.storage.k8s.io/v1
      fieldsType: FieldsV1
      fieldsV1:
      f:metadata:
      f:labels:
      .: {}
      f:velero.io/backup-name: {}
      manager: velero-plugin-for-csi
      operation: Update
      time: "2023-05-24T12:02:15Z"
      name: snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a
      resourceVersion: "209174"
      uid: 3ae91fce-9078-46ba-9af7-8483d32fd0d5
      spec:
      deletionPolicy: Retain
      driver: ebs.csi.aws.com
      source:
      volumeHandle: vol-039e6374bdf0922bd
      volumeSnapshotClassName: example-snapclass
      volumeSnapshotRef:
      apiVersion: snapshot.storage.k8s.io/v1
      kind: VolumeSnapshot
      name: velero-volume3-57wqb
      namespace: minimal-8csivol
      resourceVersion: "209053"
      uid: 4125d62b-dab7-4d7d-af91-7b5c3dbee52a
      status:
      creationTime: 1684929730484000000
      error:
      message: 'Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a: "snapshot controller failed to update snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-4125d62b-dab7-4d7d-af91-7b5c3dbee52a\": the object has been modified; please apply your changes to the latest version and try again"'
      time: "2023-05-24T12:02:15Z"
      readyToUse: false
      restoreSize: 1073741824
      snapshotHandle: snap-0ffb8994814f826a0 

      Slack discussion:- https://redhat-internal.slack.com/archives/C0144ECKUJ0/p1684931431092739

              spampatt@redhat.com Shubham Pampattiwar
              rhn-support-prajoshi Prasad Joshi
              Prasad Joshi Prasad Joshi
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: