Loading...

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: OADP 1.6.0
Affects Version/s: None
Component/s: volumeGroupSnapshot
Labels:
- oadp_upstream_bug_fix
- triaged

Activity Type:
Quality / Stability / Reliability
Story Points:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
QEStatus:
ToDo
Intelligence Requested:
Market:

Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

Regression:
None

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

PX Impact Score:

Description of problem:

Encountered this issue when I was testing volumeGroupSnapshot feature on velero 1.17. VolumeSnapshotContent resource are left out in the cluster. Backup logs suggest the vs and vsc is removed from the cluster.

$ oc get vsc
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER                               VOLUMESNAPSHOTCLASS                VOLUMESNAPSHOT                                                              VOLUMESNAPSHOTNAMESPACE   AGE
snapcontent-41df6239-4e1c-4193-a7dd-b595fe999aaa   true         2147483648    Delete           openshift-storage.rbd.csi.ceph.com   odf-operator-rbdplugin-snapclass   snapshot-a157830608ee6b2606a0c39bbc7748e757996b399b8c26ef92d7737b9e76d30e   ocp-mysql                 8m58s

Version-Release number of selected component (if applicable):
Velero 1.17
OCP 4.19

How reproducible:

Always

Steps to Reproduce:
1. Install velero 1.17 in cluster
2. Deploy an application consisting more than 1 PVC
3. Add vgs label to deployed pvc.

oc label pvc mysql-data -n ocp-mysql  velero.io/volume-group=test

4. Trigger CSI backup

Actual results: VolumeSnapshotContent resources are left out in the cluster.

Expected results:

Velero should cleanup the volumeSnapshotContent resource from the cluster.

Additional info:

backup yaml

$ oc get backup.velero.io test-backup -n velero -o yaml -w
apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    velero.io/resource-timeout: 10m0s
    velero.io/source-cluster-k8s-gitversion: v1.32.8
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "32"
  creationTimestamp: "2025-09-25T08:26:28Z"
  generation: 7
  labels:
    velero.io/storage-location: default
  name: test-backup
  namespace: velero
  resourceVersion: "196461"
  uid: a17ec267-38e0-4608-a0a0-12d5d05f9b74
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  excludedClusterScopedResources:
  - volumesnapshotcontents.snapshot.storage.k8s.io
  excludedNamespaceScopedResources:
  - volumesnapshots.snapshot.storage.k8s.io
  includedNamespaces:
  - ocp-mysql
  itemOperationTimeout: 4h0m0s
  snapshotMoveData: false
  storageLocation: default
  ttl: 720h0m0s
  volumeGroupSnapshotLabelKey: velero.io/volume-group
status:
  backupItemOperationsAttempted: 2
  backupItemOperationsCompleted: 2
  completionTimestamp: "2025-09-25T08:28:32Z"
  csiVolumeSnapshotsAttempted: 2
  csiVolumeSnapshotsCompleted: 2
  expiration: "2025-10-25T08:26:28Z"
  formatVersion: 1.1.0
  hookStatus: {}
  phase: Completed
  progress:
    itemsBackedUp: 65
    totalItems: 65
  startTimestamp: "2025-09-25T08:26:28Z"
  version: 1
  warnings: 1

backup log suggest snapshot is removed from cluster.

time="2025-09-25T08:28:32Z" level=info msg="Updated 2 items out of an estimated total of 4 (estimate will change throughout the backup finalizer)" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/backup.go:1080" name=snapcontent-a157830608ee6b2606a0c39bbc7748e757996b399b8c26ef92d7737b9e76d30e namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Processing item" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/backup.go:1037" name=snapshot-4f326b2fb55317c9f997f2087a8f8b67cb3dddfce8ef691d26ed0e422ef13f2a namespace=ocp-mysql progress= resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Skipping the exclusion checks for this resource" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/item_backupper.go:115" name=snapshot-4f326b2fb55317c9f997f2087a8f8b67cb3dddfce8ef691d26ed0e422ef13f2a namespace=ocp-mysql resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Backing up item" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/item_backupper.go:187" name=snapshot-4f326b2fb55317c9f997f2087a8f8b67cb3dddfce8ef691d26ed0e422ef13f2a namespace=ocp-mysql resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Executing custom action" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/item_backupper.go:360" name=snapshot-4f326b2fb55317c9f997f2087a8f8b67cb3dddfce8ef691d26ed0e422ef13f2a namespace=ocp-mysql resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Executing VolumeSnapshotBackupItemAction" backup=velero/test-backup cmd=/velero controller=backup-finalizer logSource="pkg/backup/actions/csi/volumesnapshot_action.go:79" pluginName=velero
time="2025-09-25T08:28:32Z" level=info msg="Deleting Volumesnapshot ocp-mysql/snapshot-4f326b2fb55317c9f997f2087a8f8b67cb3dddfce8ef691d26ed0e422ef13f2a" backup=velero/test-backup cmd=/velero controller=backup-finalizer logSource="pkg/util/csi/volume_snapshot.go:550" pluginName=velero
time="2025-09-25T08:28:32Z" level=info msg="Deleted VolumeSnapshot ocp-mysql/snapshot-4f326b2fb55317c9f997f2087a8f8b67cb3dddfce8ef691d26ed0e422ef13f2a and VolumeSnapshotContent snapcontent-4f326b2fb55317c9f997f2087a8f8b67cb3dddfce8ef691d26ed0e422ef13f2a" backup=velero/test-backup cmd=/velero controller=backup-finalizer logSource="pkg/util/csi/volume_snapshot.go:583" pluginName=velero
time="2025-09-25T08:28:32Z" level=info msg="Updated 3 items out of an estimated total of 4 (estimate will change throughout the backup finalizer)" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/backup.go:1080" name=snapshot-4f326b2fb55317c9f997f2087a8f8b67cb3dddfce8ef691d26ed0e422ef13f2a namespace=ocp-mysql progress= resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Processing item" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/backup.go:1037" name=snapshot-a157830608ee6b2606a0c39bbc7748e757996b399b8c26ef92d7737b9e76d30e namespace=ocp-mysql progress= resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Skipping the exclusion checks for this resource" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/item_backupper.go:115" name=snapshot-a157830608ee6b2606a0c39bbc7748e757996b399b8c26ef92d7737b9e76d30e namespace=ocp-mysql resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Backing up item" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/item_backupper.go:187" name=snapshot-a157830608ee6b2606a0c39bbc7748e757996b399b8c26ef92d7737b9e76d30e namespace=ocp-mysql resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Executing custom action" backup=velero/test-backup controller=backup-finalizer logSource="pkg/backup/item_backupper.go:360" name=snapshot-a157830608ee6b2606a0c39bbc7748e757996b399b8c26ef92d7737b9e76d30e namespace=ocp-mysql resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-09-25T08:28:32Z" level=info msg="Executing VolumeSnapshotBackupItemAction" backup=velero/test-backup cmd=/velero controller=backup-finalizer logSource="pkg/backup/actions/csi/volumesnapshot_action.go:79" pluginName=velero
time="2025-09-25T08:28:32Z" level=info msg="Deleting Volumesnapshot ocp-mysql/snapshot-a157830608ee6b2606a0c39bbc7748e757996b399b8c26ef92d7737b9e76d30e" backup=velero/test-backup cmd=/velero controller=backup-finalizer logSource="pkg/util/csi/volume_snapshot.go:550" pluginName=velero
time="2025-09-25T08:28:32Z" level=info msg="Deleted VolumeSnapshot ocp-mysql/snapshot-a157830608ee6b2606a0c39bbc7748e757996b399b8c26ef92d7737b9e76d30e and VolumeSnapshotContent snapcontent-a157830608ee6b2606a0c39bbc7748e757996b399b8c26ef92d7737b9e76d30e" backup=velero/test-backup cmd=/velero controller=backup-finalizer logSource="pkg/util/csi/volume_snapshot.go:583" pluginName=velero

No volumeSnapshot resource found in all namespaces

$ oc get vs -A
No resources found

Details

Description

Description of problem:

Always

Attachments

Easy Agile Planning Poker

Activity

People

Dates