Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-2409

Possible pod volume backup failure if velero is installed in multiple namespaces

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • oadp-operator-bundle-container-1.2.2-16
    • ToDo
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • No

      Description of problem:

      Downstream tracker for https://github.com/vmware-tanzu/velero/issues/6519

      Internal discussion https://redhat-internal.slack.com/archives/C0144ECKUJ0/p1689659147989289

      The test case affected is:
      https://polarion.engineering.redhat.com/polarion/#/project/OADP/workitem?id=OADP-256
      "Configure DPA with multiple BSL having same provider"
      sseago 

      How to reproduce:
      1. Deploy 2 OADP instances, each in its own namespace.
      2. Run in parallel the following 2 test cases using Restic:
      https://polarion.engineering.redhat.com/polarion/#/project/OADP/workitem?id=OADP-256
      https://polarion.engineering.redhat.com/polarion/#/project/OADP/workitem?id=OADP-98

      Expected results:
      2 test cases should pass.

      Actual results :
      Test case OADP_98 passed, test case OADP_256 fail.
      Test case OADP_256 fail on backup step.

      Error example from AWS cloud cluster (from QE automation logs):

      [FAILED] backup phase is: PartiallyFailed; expected: Completed
  validation errors: []
  velero failure logs: [velero container contains "level=error" in line#127: time="2023-07-20T15:58:44Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/2)" controller=backup-storage-location logSource="/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:194"
   velero container contains "level=error" in line#938: time="2023-07-20T15:59:35Z" level=error msg="Error backing up item" backup=openshift-adp/mysql-42e6adbb-2716-11ee-be6f-5405db5be9ea error="pod volume backup failed: error creating uploader: failed to connect repository: error running command=restic snapshots --repo=s3:s3-us-east-2.amazonaws.com/oadpbucket219252/velero-e2e-42758ed4-2716-11ee-be6f-5405db5be9ea/restic/test-oadp-256-1 --password-file=/tmp/credentials/openshift-adp-2/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic --latest=1, stdout=, stderr=Fatal: unable to open config file: Stat: Access Denied.\nIs there a repository at the following location?\ns3:s3-us-east-2.amazonaws.com/oadpbucket219252/velero-e2e-42758ed4-2716-11ee-be6f-5405db5be9ea/restic/test-oadp-256-1\n: exit status 1" error.file="/remote-source/velero/app/pkg/podvolume/backupper.go:250" error.function="github.com/vmware-tanzu/velero/pkg/podvolume.(*backupper).BackupPodVolumes" logSource="/remote-source/velero/app/pkg/backup/backup.go:435" name=mysql-6b9879d57c-4fmz7
  ]
  Expected
      <v1.BackupPhase>: PartiallyFailed
  to equal
      <v1.BackupPhase>: Completed

       

       

          1.
          		 [RedHat QE] Verify Bug OADP-2409 - Possible pod volume backup failure if velero is installed in multiple namespaces Sub-task Release Pending Undefined Shahaf Bahar
          2.
          		 [IBM QE-P] Verify Bug OADP-2409 - Possible pod volume backup failure if velero is installed in multiple namespaces Sub-task Release Pending Undefined Sonia Garudi
          3.
          		 [IBM QE-Z] Verify Bug OADP-2409 - Possible pod volume backup failure if velero is installed in multiple namespaces Sub-task Release Pending Undefined SHIVA SAI K (Inactive)

            [OADP-2409] Possible pod volume backup failure if velero is installed in multiple namespaces

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (OpenShift API for Data Protection (OADP) 1.2.2 security and bug fix update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHBA-2023:5754

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (OpenShift API for Data Protection (OADP) 1.2.2 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:5754

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in merge request !215 of rhmt / OADP Midstream on branch OADP-1.2-rhel-8_upstream_4f29a6674978da524f1af96f9532d47d:

            Updated 2 upstream sources

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in merge request !215 of rhmt / OADP Midstream on branch OADP-1.2-rhel-8_ upstream _4f29a6674978da524f1af96f9532d47d : Updated 2 upstream sources

            Scott Seago added a comment -

            Pr for OADP 1.2: https://github.com/openshift/velero/pull/284

            Scott Seago added a comment - Pr for OADP 1.2: https://github.com/openshift/velero/pull/284

              sseago Scott Seago
              sbahar Shahaf Bahar
              Scott Seago, Shahaf Bahar
              Shahaf Bahar Shahaf Bahar
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: