Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-1143

Backup Using "Restic" Failed when sc set to "cephfs" exit on "Fatal: unable to save snapshot: snapshot is empty"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • OADP 1.2.0
    • OADP 1.2.0
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • QE - Ack
    • oadp-velero-container-1.2.0-36
    • ToDo
    • Yes
    • Important
    • 8
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown

    Description

      Backup using "restic"  failed when the storage class set to cephfs  
      tested on both clouds 10 & 33 with the same adp version:  iib:387699
      running the same test using  ceph-rbd  storage class with restic pass !

      the main error : 

      error: exit status 1 stderr: Fatal: unable to save snapshot: snapshot is empty 

      the error  from the velero log : 

      time="2023-01-02T10:23:22Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/busybox-perf-single-ns-1-pods logSource="/remote-source/velero/app/pk
      g/backup/backup.go:421" name=busybox-perf-single-ns-1-pods-1
      time="2023-01-02T10:23:22Z" level=error msg="Error backing up item" backup=openshift-adp/busybox-perf-single-ns-1-pods error="pod volume backup failed: running backup, std
      err=error running restic backup command restic backup --repo=s3:https://s3-openshift-storage.apps.mtv30.cloud10mpqe.com/oadp-bucket/velero/restic/busybox-perf-single-ns-1-
      pods -password-file=/tmp/credentials/openshift-adp/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic . --tag=pod-uid=033c7ac4-13e7-49e8-a73a
      c2303e5fd84c --tag=pvc-uid=4123b054-b2ac-4cdd-82dc-a199b22ade6f --tag=volume=busybox-perf-single-ns-1-pods-1 --tag=backup=busybox-perf-single-ns-1-pods --tag=backup-uid=21
      81e634-0eed-4bb7-9190-c662b699a93a --tag=ns=busybox-perf-single-ns-1-pods --tag=pod=busybox-perf-single-ns-1-pods-1 --host=velero --json --insecure-tls=true with error: ex
      it status 1 stderr: Fatal: unable to save snapshot: snapshot is empty\n: error running restic backup command restic backup --repo=s3:https://s3-openshift-storage.apps.mtv3
      0.cloud10mpqe.com/oadp-bucket/velero/restic/busybox-perf-single-ns-1-pods --password-file=/tmp/credentials/openshift-adp/velero-repo-credentials-repository-password --cach
      e-dir=/scratch/.cache/restic . -tag=pod-uid=033c7ac4-13e7-49e8-a73a-c2303e5fd84c --tag=pvc-uid=4123b054-b2ac-4cdd-82dc-a199b22ade6f --tag=volume=busybox-perf-single-ns-1
      pods-1 --tag=backup=busybox-perf-single-ns-1-pods --tag=backup-uid=2181e634-0eed-4bb7-9190-c662b699a93a --tag=ns=busybox-perf-single-ns-1-pods --tag=pod=busybox-perf-singl
      e-ns-1-pods-1 --host=velero --json --insecure-tls=true with error: exit status 1 stderr: Fatal: unable to save snapshot: snapshot is empty\n" error.file="/remote-source/ve
      lero/app/pkg/podvolume/backupper.go:223" error.function="github.com/vmware-tanzu/velero/pkg/podvolume.(*backupper).BackupPodVolumes" logSource="/remote-source/velero/app/p
      kg/backup/backup.go:425" name=busybox-perf-single-ns-1-pods-1
      time="2023-01-02T10:23:22Z" level=info msg="Backed up 3 items out of an estimated total of 40 (estimate will change throughout the backup)" backup=openshift-adp/busybox-pe
      rf-single-ns-1-pods logSource="/remote-source/velero/app/pkg/backup/backup.go:388" name=busybox-perf-single-ns-1-pods-1 namespace=busybox-perf-single-ns-1-pods progress= r
      esource=pods 

       

      [root@f01-h14-000-r640 oadp-helpers]# velero backup describe busybox-perf-single-ns-1-pods
      I0102 12:35:42.516664  485523 request.go:601] Waited for 1.192676722s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/ocs.openshift.io/v1?timeout=32s
      Name:         busybox-perf-single-ns-1-pods
      Namespace:    openshift-adp
      Labels:       velero.io/storage-location=example-velero-1
      Annotations:  velero.io/source-cluster-k8s-gitversion=v1.24.0+3882f8f
                    velero.io/source-cluster-k8s-major-version=1
                    velero.io/source-cluster-k8s-minor-version=24Phase:  PartiallyFailed (run `velero backup logs busybox-perf-single-ns-1-pods` for more information)Errors:    1
      Warnings:  0
      

       

      [root@f01-h14-000-r640 oadp-helpers]# oc get pvc
      NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
      pvc-busybox-perf-single-ns-1-pods-1   Bound    pvc-4123b054-b2ac-4cdd-82dc-a199b22ade6f   128Mi      RWO            ocs-storagecluster-cephfs   98m
       

      Attachments

        Activity

          People

            spampatt@redhat.com Shubham Pampattiwar
            tzahia Tzahi Ashkenazi
            Tzahi Ashkenazi Tzahi Ashkenazi
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: