Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-196

Creating backup on AWS partially fails

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • OADP 1.0.0
    • None
    • None
    • None
    • False
    • False
    • Passed
    • 0
    • 0
    • Untriaged
    • None

      Description of problem: backup always fails on AWS since OADP 0.5.4. Verified on build  155353 and it was not reproducible.

      Severity: Blocker

      Version-Release number of selected component (if applicable):

      http://registry-proxy.engineering.redhat.com/rh-osbs/iib:157683

      How reproducible: Always

      Steps to Reproduce:

      1. Create DPA and secret:
       oc create secret generic cloud-credentials -n velero --from-file cloud=<creds file>
  kind: DataProtectionApplication
  metadata:
    creationTimestamp: "2022-01-12T15:08:59Z"
    generation: 1
    name: example-velero
    namespace: velero
    resourceVersion: "216427"
    uid: 138d69e4-71c2-4697-990f-9be8f268420f
  spec:
    backupLocations:
    - velero:
        config:
          profile: snapshot
          region: us-east-2
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: mayapvelerooadp
          prefix: velero
        provider: aws
    configuration:
      restic:
        enable: true
      velero:
        defaultPlugins:
        - openshift
        - csi
        - aws
    snapshotLocations:
    - velero:
        config:
          profile: snapshot
          region: us-east-2
        provider: aws
 
      1. Create backup of some app and check the result

       

      Actual results: keeps failing with this error:

      time="2022-01-12T14:41:35Z" level=error msg="Error getting volume snapshotter for volume snapshot location" backup=velero/backup9 error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/remote-source/app/velero-plugin-for-aws/volume_snapshotter.go:66" error.function=main.getSession logSource="pkg/backup/item_backupper.go:449" name=pvc-ffc80f0b-f408-44a5-9656-344ca998447b namespace= persistentVolume=pvc-ffc80f0b-f408-44a5-9656-344ca998447b resource=persistentvolumes volumeSnapshotLocation=example-velero-

       

      Here is the console output after installing 15535 and after installing 157683:

      https://privatebin.corp.redhat.com/?031d2d8ee2a121da#BKSRSLnda82FCHED8oAxsyEVWbhbaX19p9cvSyNCdsiD

       

            [OADP-196] Creating backup on AWS partially fails

            Errata Tool added a comment -

            This issue has been addressed in the following products:

            OADP-1.0-RHEL-8

            Via RHBA-2022:0455 https://access.redhat.com/errata/RHBA-2022:0455

            Errata Tool added a comment - This issue has been addressed in the following products: OADP-1 .0-RHEL-8 Via RHBA-2022 :0455 https://access.redhat.com/errata/RHBA-2022:0455

            Errata Tool added a comment -

            This issue has been addressed in the following products:

            OADP-1.0-RHEL-8

            Via RHBA-2022:0455 https://access.redhat.com/errata/RHBA-2022:0455

            Errata Tool added a comment - This issue has been addressed in the following products: OADP-1 .0-RHEL-8 Via RHBA-2022 :0455 https://access.redhat.com/errata/RHBA-2022:0455

            Errata Tool added a comment -

            This issue has been addressed in the following products:

            OADP-1.0-RHEL-8

            Via RHBA-2022:0455 https://access.redhat.com/errata/RHBA-2022:0455

            Errata Tool added a comment - This issue has been addressed in the following products: OADP-1 .0-RHEL-8 Via RHBA-2022 :0455 https://access.redhat.com/errata/RHBA-2022:0455

            Maya Peretz added a comment -

            Verified on 0.5.5 downstream build; fixed (registry-proxy.engineering.redhat.com/rh-osbs/iib:160509)

            http://pastebin.test.redhat.com/1022128

             

            Maya Peretz added a comment - Verified on 0.5.5 downstream build; fixed (registry-proxy.engineering.redhat.com/rh-osbs/iib:160509) http://pastebin.test.redhat.com/1022128  

            Maya Peretz added a comment -

            Verified against upstream 0.5.5, seems like it's fixed:

            (mtc-e2e-venv) [mperetz@mperetz oadp-operator]$ oc get dpa -n velero -o yaml
apiVersion: v1
items:
- apiVersion: oadp.openshift.io/v1alpha1
  kind: DataProtectionApplication
  metadata:
    creationTimestamp: "2022-01-16T14:21:30Z"
    generation: 1
    name: example-velero
    namespace: velero
    resourceVersion: "151955"
    uid: 66dd1674-8ac6-4714-814f-8a94bf66d198
  spec:
    backupLocations:
    - velero:
        config:
          profile: snapshot
          region: us-east-2
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: mayapvelerooadp
          prefix: velero
        provider: aws
    configuration:
      restic:
        enable: true
      velero:
        defaultPlugins:
        - openshift
        - csi
        - aws
    snapshotLocations:
    - velero:
        config:
          profile: snapshot
          region: us-east-2
        provider: aws
  status:
    conditions:
    - lastTransitionTime: "2022-01-16T14:23:03Z"
      message: Reconcile complete
      reason: Complete
      status: "True"
      type: Reconciled
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
(mtc-e2e-venv) [mperetz@mperetz oadp-operator]$ velero backup create backup$((i++)) --include-namespaces=mysql-persistent
Backup request "backup1" submitted successfully.
Run `velero backup describe backup1` or `velero backup logs backup1` for more details.
(mtc-e2e-venv) [mperetz@mperetz velero-examples]$ 
(mtc-e2e-venv) [mperetz@mperetz velero-examples]$ 
(mtc-e2e-venv) [mperetz@mperetz velero-examples]$ velero backup get
NAME      STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
backup0   Completed   0        0          2022-01-16 16:29:27 +0200 IST   29d       example-velero-1   <none>
backup1   Completed   0        0          2022-01-16 16:33:36 +0200 IST   29d       example-velero-1   <none>

            Maya Peretz added a comment - Verified against upstream 0.5.5, seems like it's fixed: (mtc-e2e-venv) [mperetz@mperetz oadp-operator]$ oc get dpa -n velero -o yaml apiVersion: v1 items: - apiVersion: oadp.openshift.io/v1alpha1   kind: DataProtectionApplication   metadata:     creationTimestamp: "2022-01-16T14:21:30Z"     generation: 1     name: example-velero     namespace: velero     resourceVersion: "151955"     uid: 66dd1674-8ac6-4714-814f-8a94bf66d198   spec:     backupLocations:     - velero:         config:           profile: snapshot           region: us-east-2         credential:           key: cloud           name: cloud-credentials         default: true         objectStorage:           bucket: mayapvelerooadp           prefix: velero         provider: aws     configuration:       restic:         enable: true       velero:         defaultPlugins:         - openshift         - csi         - aws     snapshotLocations:     - velero:         config:           profile: snapshot           region: us-east-2         provider: aws   status:     conditions:     - lastTransitionTime: "2022-01-16T14:23:03Z"       message: Reconcile complete       reason: Complete       status: "True"       type: Reconciled kind: List metadata:   resourceVersion: ""   selfLink: "" (mtc-e2e-venv) [mperetz@mperetz oadp-operator]$ velero backup create backup$((i++)) --include-namespaces=mysql-persistent Backup request "backup1" submitted successfully. Run `velero backup describe backup1` or `velero backup logs backup1` for more details. (mtc-e2e-venv) [mperetz@mperetz velero-examples]$  (mtc-e2e-venv) [mperetz@mperetz velero-examples]$  (mtc-e2e-venv) [mperetz@mperetz velero-examples]$ velero backup get NAME      STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR backup0   Completed   0        0          2022-01-16 16:29:27 +0200 IST   29d       example-velero-1   <none> backup1   Completed   0        0          2022-01-16 16:33:36 +0200 IST   29d       example-velero-1   <none>

            Dylan Murray added a comment -

            https://github.com/openshift/oadp-operator/pull/530

            Dylan Murray added a comment - https://github.com/openshift/oadp-operator/pull/530

              rhn-engineering-dymurray Dylan Murray
              mperetz@redhat.com Maya Peretz
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: