Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-52035

[4.16] Velero backup PartiallyFailed due to PVC' failing to be snapshoted

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Normal Normal
    • CNV v4.16.7
    • CNV v4.16.3, CNV v4.17.1
    • Storage Platform
    • None
    • Storage Core Sprint 262, Storage Core Sprint 263, CNV Storage 266, CNV Storage 267, CNV Storage 268
    • None

      Description of problem:

      When backing up VM or CDI DV its possible to get a PartiallyFailled backup
      In the logs youll be able to see something like that:
      In backup describe --details:
        Velero:    name: /prime-cb3e67a7-3365-491e-9fa5-c3df99192865 message: /Error backing up item error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=kvp-e2e-tests-swlqn, name=velero-prime-cb3e67a7-3365-491e-9fa5-c3df99192865-sf8r7): rpc error: code = Unknown desc = failed to get volumesnapshot kvp-e2e-tests-swlqn/velero-prime-cb3e67a7-3365-491e-9fa5-c3df99192865-sf8r7: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
                   name: /test-vm message: /Error backing up item error: /error executing custom action (groupResource=virtualmachines.kubevirt.io, namespace=kvp-e2e-tests-swlqn, name=test-vm): rpc error: code = Unknown desc = VM would not be restored correctly
      In backup log:
      "Waiting for CSI driver to reconcile volumesnapshot kvp-e2e-tests-swlqn/velero-prime-cb3e67a7-3365-491e-9fa5-c3df99192865-sf8r7. Retrying in 5s" backup=openshift-adp/test-backup-1731249330041542604 cmd=/velero logSource="/remote-source/velero/app/pkg/util/csi/volume_snapshot.go:713" pluginName=velero"

      Version-Release number of selected component (if applicable):

      4.16.3, 4.17.1

      How reproducible:

      Sometimes

      Steps to Reproduce:

      1.Create a VM with storage class that supports snapshots
      2.Create velero backup
      

      Actual results:

      Backup PartiallyFailed

      Expected results:

      Success

      Additional info:

      This is a racefull bug. It happens since in the time of the backup PVC' is being in terminating state, which prevent velero from creating a snapshot for it. 
      We want to PVC' to be excluded from the backup

              skagan@redhat.com Shelly Kagan
              skagan@redhat.com Shelly Kagan
              Dalia Frank Dalia Frank
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: