Loading...

Type: Bug
Resolution: Unresolved
Priority: Blocker
Fix Version/s: OADP 1.4.0
Affects Version/s: OADP 1.3.0, OADP 1.3.1
Component/s: data-mover
Labels:
- triaged

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
QEStatus:
ToDo
Regression:
No
Intelligence Requested:
Market:

Cost of Delay:
0
WSJF:
0
Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

SFDC Cases Links:
SFDC Cases Counter:

Description of problem:

The datamover backups are failing in OCP 4.16 cluster. This is due to the low SDK version velero is using where Spec.SourceVolumeMode field doesn't exist. So the problem only happens in the environment that uses external snapshotter v7.0 and higher.

Upstream issue:-
https://github.com/vmware-tanzu/velero/issues/7433

Version-Release number of selected component (if applicable):
OADP 1.3.1
OCP 4.16

How reproducible:
Always

Steps to Reproduce:
1. Create a DPA with CSI and nodeAgent enabled.
2. Deploy a stateful application

3. Trigger datamover backup.

Actual results:
Backup is marked as partiallyFailed.

[prajoshi@localhost OADP]$ velero describe backup -n openshift-adp --details test-backup
Name:         test-backup
Namespace:    openshift-adp
Labels:       velero.io/storage-location=bsl1
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=29

Phase:  PartiallyFailed (run `velero backup logs test-backup` for more information)


Errors:
  Velero:     <none>
  Cluster:    <none>
  Namespaces: <none>

Namespaces:
  Included:  ocp-mysql
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  bsl1

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          true
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-04-17 18:54:46 +0530 IST
Completed:  2024-04-17 18:55:32 +0530 IST

Expiration:  2024-05-17 18:54:46 +0530 IST

Total items to be backed up:  49
Items backed up:              49

Backup Item Operations:
  Operation for persistentvolumeclaims ocp-mysql/mysql:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-7053f1ca-7da7-41f6-82ab-61092a2a9398.cec978ba-affc-425a666de
    Items to Update:
                           datauploads.velero.io openshift-adp/test-backup-r8g2z
    Phase:                 Failed
    Operation Error:       error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-b472b317-88a1-45ce-8ddb-6fd7849898bf: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil
    Progress description:  Failed
    Created:               2024-04-17 18:55:01 +0530 IST
    Started:               2024-04-17 18:55:01 +0530 IST
    Updated:               2024-04-17 18:55:21 +0530 IST
  Operation for persistentvolumeclaims ocp-mysql/mysql-1:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-7053f1ca-7da7-41f6-82ab-61092a2a9398.683dc3fe-8276-4eb8ccd3d
    Items to Update:
                           datauploads.velero.io openshift-adp/test-backup-jsqhd
    Phase:                 Failed
    Operation Error:       error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-203eec4e-e3b5-4449-aeee-f590433b9865: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil
    Progress description:  Failed
    Created:               2024-04-17 18:55:11 +0530 IST
    Started:               2024-04-17 18:55:11 +0530 IST
    Updated:               2024-04-17 18:55:25 +0530 IST
Resource List:
  apps/v1/Deployment:
    - ocp-mysql/mysql
  apps/v1/ReplicaSet:
    - ocp-mysql/mysql-7f77d47fd7
  authorization.openshift.io/v1/RoleBinding:
    - ocp-mysql/admin
    - ocp-mysql/system:deployers
    - ocp-mysql/system:image-builders
    - ocp-mysql/system:image-pullers
  discovery.k8s.io/v1/EndpointSlice:
    - ocp-mysql/mysql-9m9n9
  rbac.authorization.k8s.io/v1/RoleBinding:
    - ocp-mysql/admin
    - ocp-mysql/system:deployers
    - ocp-mysql/system:image-builders
    - ocp-mysql/system:image-pullers
  v1/ConfigMap:
    - ocp-mysql/kube-root-ca.crt
    - ocp-mysql/openshift-service-ca.crt
  v1/Endpoints:
    - ocp-mysql/mysql
  v1/Event:
    - ocp-mysql/mysql-1.17c7135af6cfdb73
    - ocp-mysql/mysql-1.17c7135b1b7dbd8c
    - ocp-mysql/mysql-1.17c7135b1b81ddc7
    - ocp-mysql/mysql-1.17c7135c03517b2e
    - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135c45a2554d
    - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135e067330ce
    - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135eebee2801
    - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135f348b2014
    - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c7135f36167ec6
    - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c713613e8b25aa
    - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c713614506645a
    - ocp-mysql/mysql-7f77d47fd7-rwxgd.17c713614603dbfd
    - ocp-mysql/mysql-7f77d47fd7.17c7135b1a08768a
    - ocp-mysql/mysql.17c7135ad0d5bda7
    - ocp-mysql/mysql.17c7135b17b712d5
    - ocp-mysql/mysql.17c7135b1b618f1e
    - ocp-mysql/mysql.17c7135b1b7582b0
    - ocp-mysql/mysql.17c7135c0a057f9f
  v1/Namespace:
    - ocp-mysql
  v1/PersistentVolume:
    - pvc-683dc3fe-8276-4eba-bd91-99b73c348159
    - pvc-cec978ba-affc-425d-8553-40331c2a6d60
  v1/PersistentVolumeClaim:
    - ocp-mysql/mysql
    - ocp-mysql/mysql-1
  v1/Pod:
    - ocp-mysql/mysql-7f77d47fd7-rwxgd
  v1/Secret:
    - ocp-mysql/builder-dockercfg-rnsvr
    - ocp-mysql/builder-token-rm55q
    - ocp-mysql/default-dockercfg-f6wsl
    - ocp-mysql/default-token-8mgk9
    - ocp-mysql/deployer-dockercfg-tphcx
    - ocp-mysql/deployer-token-dcnx4
    - ocp-mysql/mysql
  v1/Service:
    - ocp-mysql/mysql
  v1/ServiceAccount:
    - ocp-mysql/builder
    - ocp-mysql/default
    - ocp-mysql/deployer

Velero-Native Snapshots: <none included>

Expected results:

Backup should be completed successfully.

Additional info:

oc get dataupload test-backup-jsqhd -o yaml
apiVersion: velero.io/v2alpha1
kind: DataUpload
metadata:
  creationTimestamp: "2024-04-17T13:25:11Z"
  generateName: test-backup-
  generation: 3
  labels:
    velero.io/accepted-by: oadp-77901-5b7sk-worker-a-vnlmj
    velero.io/async-operation-id: du-7053f1ca-7da7-41f6-82ab-61092a2a9398.683dc3fe-8276-4eb8ccd3d
    velero.io/backup-name: test-backup
    velero.io/backup-uid: 7053f1ca-7da7-41f6-82ab-61092a2a9398
    velero.io/pvc-uid: 683dc3fe-8276-4eba-bd91-99b73c348159
  name: test-backup-jsqhd
  namespace: openshift-adp
  ownerReferences:
  - apiVersion: velero.io/v1
    controller: true
    kind: Backup
    name: test-backup
    uid: 7053f1ca-7da7-41f6-82ab-61092a2a9398
  resourceVersion: "151759"
  uid: 5e4ee9ed-e481-4915-b53e-fae4c0b8c936
spec:
  backupStorageLocation: bsl1
  csiSnapshot:
    snapshotClass: ""
    storageClass: standard-csi
    volumeSnapshot: velero-mysql-1-6w8sw
  operationTimeout: 10m0s
  snapshotType: CSI
  sourceNamespace: ocp-mysql
  sourcePVC: mysql-1
status:
  completionTimestamp: "2024-04-17T13:25:25Z"
  message: 'error to expose snapshot: error to remove protect from volume snapshot
    content: error to update VolumeSnapshotContent snapcontent-203eec4e-e3b5-4449-aeee-f590433b9865:
    admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request:
    Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil'
  phase: Failed
  progress: {}
  startTimestamp: "2024-04-17T13:25:11Z"

links to

https://github.com/vmware-tanzu/velero/pull/7489

Details

Description

Description of problem:

Backup should be completed successfully.

Attachments

Issue Links

Activity

People

Dates