-
Sub-task
-
Resolution: Done
-
Undefined
-
None
-
None
-
None
-
4
-
False
-
-
False
-
ToDo
-
-
-
0
-
0.0
-
Very Likely
-
0
-
None
-
Unset
-
Unknown
Description of problem:
The datamover backups are failing in OCP 4.16 cluster. This is due to the low SDK version velero is using where Spec.SourceVolumeMode field doesn't exist. So the problem only happens in the environment that uses external snapshotter v7.0 and higher.
Upstream issue:-
https://github.com/vmware-tanzu/velero/issues/7433
Version-Release number of selected component (if applicable):
OADP 1.3.1
OCP 4.16
How reproducible:
Always
Steps to Reproduce:
1. Create a DPA with CSI and nodeAgent enabled.
2. Deploy a stateful application
3. Trigger datamover backup.
Actual results:
Backup is marked as partiallyFailed.
[prajoshi@localhost OADP]$ velero describe backup -n openshift-adp --details test-backup Name: test-backup Namespace: openshift-adp Labels: velero.io/storage-location=bsl1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: PartiallyFailed (run `velero backup logs test-backup` for more information) Errors: Velero: <none> Cluster: <none> Namespaces: <none> Namespaces: Included: ocp-mysql Excluded: <none> Resources: Included: * Excluded: <none> Cluster-scoped: auto Label selector: <none> Or label selector: <none> Storage Location: bsl1 Velero-Native Snapshot PVs: auto Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2024-04-17 18:54:46 +0530 IST Completed: 2024-04-17 18:55:32 +0530 IST Expiration: 2024-05-17 18:54:46 +0530 IST Total items to be backed up: 49 Items backed up: 49 Backup Item Operations: Operation for persistentvolumeclaims ocp-mysql/mysql: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-7053f1ca-7da7-41f6-82ab-61092a2a9398.cec978ba-affc-425a666de Items to Update: datauploads.velero.io openshift-adp/test-backup-r8g2z Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-b472b317-88a1-45ce-8ddb-6fd7849898bf: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-17 18:55:01 +0530 IST Started: 2024-04-17 18:55:01 +0530 IST Updated: 2024-04-17 18:55:21 +0530 IST Operation for persistentvolumeclaims ocp-mysql/mysql-1: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-7053f1ca-7da7-41f6-82ab-61092a2a9398.683dc3fe-8276-4eb8ccd3d Items to Update: datauploads.velero.io openshift-adp/test-backup-jsqhd Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-203eec4e-e3b5-4449-aeee-f590433b9865: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-17 18:55:11 +0530 IST Started: 2024-04-17 18:55:11 +0530 IST Updated: 2024-04-17 18:55:25 +0530 IST Resource List: apps/v1/Deployment: - ocp-mysql/mysql apps/v1/ReplicaSet: - ocp-mysql/mysql-7f77d47fd7 authorization.openshift.io/v1/RoleBinding: - ocp-mysql/admin - ocp-mysql/system:deployers - ocp-mysql/system:image-builders - ocp-mysql/system:image-pullers discovery.k8s.io/v1/EndpointSlice: - ocp-mysql/mysql-9m9n9 rbac.authorization.k8s.io/v1/RoleBinding: - ocp-mysql/admin - ocp-mysql/system:deployers - ocp-mysql/system:image-builders - ocp-mysql/system:image-pullers v1/ConfigMap: - ocp-mysql/kube-root-ca.crt - ocp-mysql/openshift-service-ca.crt v1/Endpoints: - ocp-mysql/mysql v1/Event: - ocp-mysql/mysql-1.17c7135af6cfdb73 - ocp-mysql/mysql-1.17c7135b1b7dbd8c - ocp-mysql/mysql-1.17c7135b1b81ddc7 - ocp-mysql/mysql-1.17c7135c03517b2e - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135c45a2554d - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135e067330ce - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135eebee2801 - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135f348b2014 - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135f36167ec6 - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c713613e8b25aa - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c713614506645a - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c713614603dbfd - ocp-mysql/mysql-7f77d47fd7.17c7135b1a08768a - ocp-mysql/mysql.17c7135ad0d5bda7 - ocp-mysql/mysql.17c7135b17b712d5 - ocp-mysql/mysql.17c7135b1b618f1e - ocp-mysql/mysql.17c7135b1b7582b0 - ocp-mysql/mysql.17c7135c0a057f9f v1/Namespace: - ocp-mysql v1/PersistentVolume: - pvc-683dc3fe-8276-4eba-bd91-99b73c348159 - pvc-cec978ba-affc-425d-8553-40331c2a6d60 v1/PersistentVolumeClaim: - ocp-mysql/mysql - ocp-mysql/mysql-1 v1/Pod: - ocp-mysql/mysql-7f77d47fd7-rwxgd v1/Secret: - ocp-mysql/builder-dockercfg-rnsvr - ocp-mysql/builder-token-rm55q - ocp-mysql/default-dockercfg-f6wsl - ocp-mysql/default-token-8mgk9 - ocp-mysql/deployer-dockercfg-tphcx - ocp-mysql/deployer-token-dcnx4 - ocp-mysql/mysql v1/Service: - ocp-mysql/mysql v1/ServiceAccount: - ocp-mysql/builder - ocp-mysql/default - ocp-mysql/deployer Velero-Native Snapshots: <none included>
Expected results:
Backup should be completed successfully.
Additional info:
oc get dataupload test-backup-jsqhd -o yaml apiVersion: velero.io/v2alpha1 kind: DataUpload metadata: creationTimestamp: "2024-04-17T13:25:11Z" generateName: test-backup- generation: 3 labels: velero.io/accepted-by: oadp-77901-5b7sk-worker-a-vnlmj velero.io/async-operation-id: du-7053f1ca-7da7-41f6-82ab-61092a2a9398.683dc3fe-8276-4eb8ccd3d velero.io/backup-name: test-backup velero.io/backup-uid: 7053f1ca-7da7-41f6-82ab-61092a2a9398 velero.io/pvc-uid: 683dc3fe-8276-4eba-bd91-99b73c348159 name: test-backup-jsqhd namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: test-backup uid: 7053f1ca-7da7-41f6-82ab-61092a2a9398 resourceVersion: "151759" uid: 5e4ee9ed-e481-4915-b53e-fae4c0b8c936 spec: backupStorageLocation: bsl1 csiSnapshot: snapshotClass: "" storageClass: standard-csi volumeSnapshot: velero-mysql-1-6w8sw operationTimeout: 10m0s snapshotType: CSI sourceNamespace: ocp-mysql sourcePVC: mysql-1 status: completionTimestamp: "2024-04-17T13:25:25Z" message: 'error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-203eec4e-e3b5-4449-aeee-f590433b9865: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil' phase: Failed progress: {} startTimestamp: "2024-04-17T13:25:11Z"