Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-3922 Datamover backups are failing in OCP 4.16 cluster
  3. OADP-4193

[IBM QE-P] Verify Bug OADP-3922 - Datamover backups are failing in OCP 4.16 cluster

XMLWordPrintable

    • Icon: Sub-task Sub-task
    • Resolution: Done
    • Icon: Undefined Undefined
    • OADP 1.4.0
    • None
    • None
    • None
    • 4
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • 0
    • 0.0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown

      Description of problem:

      The datamover backups are failing in OCP 4.16 cluster. This is due to the low SDK version velero is using where Spec.SourceVolumeMode field doesn't exist. So the problem only happens in the environment that uses external snapshotter v7.0 and higher.

      Upstream issue:- 
      https://github.com/vmware-tanzu/velero/issues/7433
       

      Version-Release number of selected component (if applicable):
      OADP 1.3.1
      OCP 4.16

       

      How reproducible:
      Always 

       

      Steps to Reproduce:
      1. Create a DPA with CSI and nodeAgent enabled.
      2. Deploy a stateful application

      3. Trigger datamover backup.

      Actual results:
      Backup is marked as partiallyFailed.

      [prajoshi@localhost OADP]$ velero describe backup -n openshift-adp --details test-backup
      Name:         test-backup
      Namespace:    openshift-adp
      Labels:       velero.io/storage-location=bsl1
      Annotations:  velero.io/resource-timeout=10m0s
                    velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806
                    velero.io/source-cluster-k8s-major-version=1
                    velero.io/source-cluster-k8s-minor-version=29
      
      Phase:  PartiallyFailed (run `velero backup logs test-backup` for more information)
      
      
      Errors:
        Velero:     <none>
        Cluster:    <none>
        Namespaces: <none>
      
      Namespaces:
        Included:  ocp-mysql
        Excluded:  <none>
      
      Resources:
        Included:        *
        Excluded:        <none>
        Cluster-scoped:  auto
      
      Label selector:  <none>
      
      Or label selector:  <none>
      
      Storage Location:  bsl1
      
      Velero-Native Snapshot PVs:  auto
      Snapshot Move Data:          true
      Data Mover:                  velero
      
      TTL:  720h0m0s
      
      CSISnapshotTimeout:    10m0s
      ItemOperationTimeout:  4h0m0s
      
      Hooks:  <none>
      
      Backup Format Version:  1.1.0
      
      Started:    2024-04-17 18:54:46 +0530 IST
      Completed:  2024-04-17 18:55:32 +0530 IST
      
      Expiration:  2024-05-17 18:54:46 +0530 IST
      
      Total items to be backed up:  49
      Items backed up:              49
      
      Backup Item Operations:
        Operation for persistentvolumeclaims ocp-mysql/mysql:
          Backup Item Action Plugin:  velero.io/csi-pvc-backupper
          Operation ID:               du-7053f1ca-7da7-41f6-82ab-61092a2a9398.cec978ba-affc-425a666de
          Items to Update:
                                 datauploads.velero.io openshift-adp/test-backup-r8g2z
          Phase:                 Failed
          Operation Error:       error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-b472b317-88a1-45ce-8ddb-6fd7849898bf: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil
          Progress description:  Failed
          Created:               2024-04-17 18:55:01 +0530 IST
          Started:               2024-04-17 18:55:01 +0530 IST
          Updated:               2024-04-17 18:55:21 +0530 IST
        Operation for persistentvolumeclaims ocp-mysql/mysql-1:
          Backup Item Action Plugin:  velero.io/csi-pvc-backupper
          Operation ID:               du-7053f1ca-7da7-41f6-82ab-61092a2a9398.683dc3fe-8276-4eb8ccd3d
          Items to Update:
                                 datauploads.velero.io openshift-adp/test-backup-jsqhd
          Phase:                 Failed
          Operation Error:       error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-203eec4e-e3b5-4449-aeee-f590433b9865: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil
          Progress description:  Failed
          Created:               2024-04-17 18:55:11 +0530 IST
          Started:               2024-04-17 18:55:11 +0530 IST
          Updated:               2024-04-17 18:55:25 +0530 IST
      Resource List:
        apps/v1/Deployment:
          - ocp-mysql/mysql
        apps/v1/ReplicaSet:
          - ocp-mysql/mysql-7f77d47fd7
        authorization.openshift.io/v1/RoleBinding:
          - ocp-mysql/admin
          - ocp-mysql/system:deployers
          - ocp-mysql/system:image-builders
          - ocp-mysql/system:image-pullers
        discovery.k8s.io/v1/EndpointSlice:
          - ocp-mysql/mysql-9m9n9
        rbac.authorization.k8s.io/v1/RoleBinding:
          - ocp-mysql/admin
          - ocp-mysql/system:deployers
          - ocp-mysql/system:image-builders
          - ocp-mysql/system:image-pullers
        v1/ConfigMap:
          - ocp-mysql/kube-root-ca.crt
          - ocp-mysql/openshift-service-ca.crt
        v1/Endpoints:
          - ocp-mysql/mysql
        v1/Event:
          - ocp-mysql/mysql-1.17c7135af6cfdb73
          - ocp-mysql/mysql-1.17c7135b1b7dbd8c
          - ocp-mysql/mysql-1.17c7135b1b81ddc7
          - ocp-mysql/mysql-1.17c7135c03517b2e
          - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135c45a2554d
          - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135e067330ce
          - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135eebee2801
          - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135f348b2014
          - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135f36167ec6
          - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c713613e8b25aa
          - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c713614506645a
          - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c713614603dbfd
          - ocp-mysql/mysql-7f77d47fd7.17c7135b1a08768a
          - ocp-mysql/mysql.17c7135ad0d5bda7
          - ocp-mysql/mysql.17c7135b17b712d5
          - ocp-mysql/mysql.17c7135b1b618f1e
          - ocp-mysql/mysql.17c7135b1b7582b0
          - ocp-mysql/mysql.17c7135c0a057f9f
        v1/Namespace:
          - ocp-mysql
        v1/PersistentVolume:
          - pvc-683dc3fe-8276-4eba-bd91-99b73c348159
          - pvc-cec978ba-affc-425d-8553-40331c2a6d60
        v1/PersistentVolumeClaim:
          - ocp-mysql/mysql
          - ocp-mysql/mysql-1
        v1/Pod:
          - ocp-mysql/mysql-7f77d47fd7-rwxgd
        v1/Secret:
          - ocp-mysql/builder-dockercfg-rnsvr
          - ocp-mysql/builder-token-rm55q
          - ocp-mysql/default-dockercfg-f6wsl
          - ocp-mysql/default-token-8mgk9
          - ocp-mysql/deployer-dockercfg-tphcx
          - ocp-mysql/deployer-token-dcnx4
          - ocp-mysql/mysql
        v1/Service:
          - ocp-mysql/mysql
        v1/ServiceAccount:
          - ocp-mysql/builder
          - ocp-mysql/default
          - ocp-mysql/deployer
      
      Velero-Native Snapshots: <none included> 

       

      Expected results:

      Backup should be completed successfully. 

      Additional info:

      oc get dataupload test-backup-jsqhd -o yaml
      apiVersion: velero.io/v2alpha1
      kind: DataUpload
      metadata:
        creationTimestamp: "2024-04-17T13:25:11Z"
        generateName: test-backup-
        generation: 3
        labels:
          velero.io/accepted-by: oadp-77901-5b7sk-worker-a-vnlmj
          velero.io/async-operation-id: du-7053f1ca-7da7-41f6-82ab-61092a2a9398.683dc3fe-8276-4eb8ccd3d
          velero.io/backup-name: test-backup
          velero.io/backup-uid: 7053f1ca-7da7-41f6-82ab-61092a2a9398
          velero.io/pvc-uid: 683dc3fe-8276-4eba-bd91-99b73c348159
        name: test-backup-jsqhd
        namespace: openshift-adp
        ownerReferences:
        - apiVersion: velero.io/v1
          controller: true
          kind: Backup
          name: test-backup
          uid: 7053f1ca-7da7-41f6-82ab-61092a2a9398
        resourceVersion: "151759"
        uid: 5e4ee9ed-e481-4915-b53e-fae4c0b8c936
      spec:
        backupStorageLocation: bsl1
        csiSnapshot:
          snapshotClass: ""
          storageClass: standard-csi
          volumeSnapshot: velero-mysql-1-6w8sw
        operationTimeout: 10m0s
        snapshotType: CSI
        sourceNamespace: ocp-mysql
        sourcePVC: mysql-1
      status:
        completionTimestamp: "2024-04-17T13:25:25Z"
        message: 'error to expose snapshot: error to remove protect from volume snapshot
          content: error to update VolumeSnapshotContent snapcontent-203eec4e-e3b5-4449-aeee-f590433b9865:
          admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request:
          Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil'
        phase: Failed
        progress: {}
        startTimestamp: "2024-04-17T13:25:11Z" 

            anayek Aniruddha Nayek
            akarol@redhat.com Aziza Karol
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: