Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-778

Restic backup restore doesn't work with AWS EFS

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Done
    • None
    • OADP 1.1.1
    • Documentation
    • False
    • Hide

      None

      Show
      None
    • False
    • Release Notes
    • Passed
    • No
    • Note: OADP/Velero currently does not support backup/restore of AWS EFS Volumes
    • Known Issue
    • 0
    • 0
    • Very Likely
    • 0
    • Customer Escalated, Customer Facing
    • None
    • Unset
    • Unknown

    Description

      Description of problem:

      Restic restoration fails on AWS EFS storage with the following error:

      time="2022-08-30T20:37:48Z" level=error msg="unable to successfully complete restic restores of pod's volumes" error="pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=s3:s3-us-east-2.amazonaws.com/cs-app-backup-bucket-dev/velero/restic/ckretler-sandbox --password-file=/tmp/credentials/openshift-adp/velero-restic-credentials-repository-password --cache-dir=/scratch/.cache/restic e272f797 --target=. --delete --skip-unchanged, stdout=Skip Unchanged True\nrestoring <Snapshot e272f797 of [/host_pods/5a2244f0-9ce4-428d-8aff-8af799a015cc/volumes/kubernetes.io~csi/pvc-7ac3424c-bc50-4119-a88d-6382f7f8d866/mount] at 2022-08-30 20:35:22.652520881 +0000 UTC by root@velero> to .\n, stderr=ignoring error for /bigfile: Lchown: lchown /host_pods/fc1194b7-f306-4dac-9543-d9c5dcf91c26/volumes/kubernetes.io~csi/pvc-6130721b-2d5f-4897-a2b3-e5890e9d7a78/mount/bigfile: operation not permitted\nignoring error for /file.txt: Lchown: lchown /host_pods/fc1194b7-f306-4dac-9543-d9c5dcf91c26/volumes/kubernetes.io~csi/pvc-6130721b-2d5f-4897-a2b3-e5890e9d7a78/mount/file.txt: operation not permitted\nFatal: There were 2 errors\n\n: exit status 1" logSource="pkg/restore/restore.go:1464" restore=openshift-adp/ckretler-restore
      

       

      Version-Release number of selected component (if applicable):

       OADP operator 1.0.3

      How reproducible:

      All the time.

      Steps to Reproduce:

      • OADP operator installed
      • EFS CSI driver installed
      • Create a SC using EFS
        apiVersion: storage.k8s.io/v1
        kind: StorageClass
        metadata:
          annotations:
            storageclass.kubernetes.io/is-default-class: "true"
          name: efs
        parameters:
          basePath: /pvs
          directoryPerms: "700"
          fileSystemId: fs-0886f245d447f0cf5
          provisioningMode: efs-ap
        provisioner: efs.csi.aws.com
        reclaimPolicy: Delete
        volumeBindingMode: Immediate
        
      • Some workload that is using the EFS SC
      • Run a restic backup
      • Delete the workload
      • Restore the backup

      Actual results:

      Restoration fails with following error:

      time="2022-08-30T20:37:48Z" level=error msg="unable to successfully complete restic restores of pod's volumes" error="pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=s3:s3-us-east-2.amazonaws.com/cs-app-backup-bucket-dev/velero/restic/ckretler-sandbox --password-file=/tmp/credentials/openshift-adp/velero-restic-credentials-repository-password --cache-dir=/scratch/.cache/restic e272f797 --target=. --delete --skip-unchanged, stdout=Skip Unchanged True\nrestoring <Snapshot e272f797 of [/host_pods/5a2244f0-9ce4-428d-8aff-8af799a015cc/volumes/kubernetes.io~csi/pvc-7ac3424c-bc50-4119-a88d-6382f7f8d866/mount] at 2022-08-30 20:35:22.652520881 +0000 UTC by root@velero> to .\n, stderr=ignoring error for /bigfile: Lchown: lchown /host_pods/fc1194b7-f306-4dac-9543-d9c5dcf91c26/volumes/kubernetes.io~csi/pvc-6130721b-2d5f-4897-a2b3-e5890e9d7a78/mount/bigfile: operation not permitted\nignoring error for /file.txt: Lchown: lchown /host_pods/fc1194b7-f306-4dac-9543-d9c5dcf91c26/volumes/kubernetes.io~csi/pvc-6130721b-2d5f-4897-a2b3-e5890e9d7a78/mount/file.txt: operation not permitted\nFatal: There were 2 errors\n\n: exit status 1" logSource="pkg/restore/restore.go:1464" restore=openshift-adp/ckretler-restore
      

      Expected results:

      A successful backup restoration

      Additional info:

      https://github.com/vmware-tanzu/velero/issues/3729

      Attachments

        Activity

          People

            richard.hoch Richard Hoch
            rhn-support-gilins Givaldo Lins
            Maya Peretz Maya Peretz
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: