Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-2409

Possible pod volume backup failure if velero is installed in multiple namespaces

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • oadp-operator-bundle-container-1.2.2-16
    • ToDo
    • 0
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • No

      Description of problem:

      Downstream tracker for https://github.com/vmware-tanzu/velero/issues/6519

      Internal discussion https://redhat-internal.slack.com/archives/C0144ECKUJ0/p1689659147989289

      The test case affected is:
      https://polarion.engineering.redhat.com/polarion/#/project/OADP/workitem?id=OADP-256
      "Configure DPA with multiple BSL having same provider"
      sseago 

      How to reproduce:
      1. Deploy 2 OADP instances, each in its own namespace.
      2. Run in parallel the following 2 test cases using Restic:
      https://polarion.engineering.redhat.com/polarion/#/project/OADP/workitem?id=OADP-256
      https://polarion.engineering.redhat.com/polarion/#/project/OADP/workitem?id=OADP-98

      Expected results:
      2 test cases should pass.

      Actual results :
      Test case OADP_98 passed, test case OADP_256 fail.
      Test case OADP_256 fail on backup step.

      Error example from AWS cloud cluster (from QE automation logs):

      [FAILED] backup phase is: PartiallyFailed; expected: Completed
        validation errors: []
        velero failure logs: [velero container contains "level=error" in line#127: time="2023-07-20T15:58:44Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/2)" controller=backup-storage-location logSource="/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:194"
         velero container contains "level=error" in line#938: time="2023-07-20T15:59:35Z" level=error msg="Error backing up item" backup=openshift-adp/mysql-42e6adbb-2716-11ee-be6f-5405db5be9ea error="pod volume backup failed: error creating uploader: failed to connect repository: error running command=restic snapshots --repo=s3:s3-us-east-2.amazonaws.com/oadpbucket219252/velero-e2e-42758ed4-2716-11ee-be6f-5405db5be9ea/restic/test-oadp-256-1 --password-file=/tmp/credentials/openshift-adp-2/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic --latest=1, stdout=, stderr=Fatal: unable to open config file: Stat: Access Denied.\nIs there a repository at the following location?\ns3:s3-us-east-2.amazonaws.com/oadpbucket219252/velero-e2e-42758ed4-2716-11ee-be6f-5405db5be9ea/restic/test-oadp-256-1\n: exit status 1" error.file="/remote-source/velero/app/pkg/podvolume/backupper.go:250" error.function="github.com/vmware-tanzu/velero/pkg/podvolume.(*backupper).BackupPodVolumes" logSource="/remote-source/velero/app/pkg/backup/backup.go:435" name=mysql-6b9879d57c-4fmz7
        ]
        Expected
            <v1.BackupPhase>: PartiallyFailed
        to equal
            <v1.BackupPhase>: Completed
       

       

       

              sseago Scott Seago
              sbahar Shahaf Bahar
              Scott Seago, Shahaf Bahar
              Shahaf Bahar Shahaf Bahar
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: