Loading...

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: OADP 1.6.0
Affects Version/s: OADP 1.4.4
Component/s: velero
Labels:
- oadp_dpa
- triaged

Activity Type:
Quality / Stability / Reliability
Story Points:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
QEStatus:
ToDo
Intelligence Requested:
Market:

WSJF:
0
Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

Regression:
None

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem: slack

Backup failing if "region" is not provided for AWS bucket in DPA, as the BackupRepository complains for region. But DPA reconciles successfully.

And BSL remains available
If I add snapshotLocations as well to this DPA and provide region in snapshotLocations, it works fine. and backuprepository also connects.
But not without that.

Version-Release number of selected component (if applicable):

Midstream "main"

How reproducible:

Always

Steps to Reproduce:
1. Deploy any stateful application
2. Setup DPA as shown below, and perform backup with snapshotmoveData: true

Actual results:

Backup PArtially fails with Error in BackupRepository

Expected results:

Backup should work

Additional info:

oc get dpa -o yaml
apiVersion: v1
items:
- apiVersion: oadp.openshift.io/v1alpha1
  kind: DataProtectionApplication
  metadata:
    creationTimestamp: "2025-03-10T05:14:24Z"
    generation: 2
    name: ts-dpa
    namespace: openshift-adp
    resourceVersion: "38676"
    uid: f55cebc5-84ec-45c5-8e36-91341524d94a
  spec:
    backupLocations:
    - velero:
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: oadp113331tvmz7
          prefix: velero
        provider: aws
    configuration:
      nodeAgent:
        enable: true
        uploaderType: kopia
      velero:
        defaultPlugins:
        - csi
        - aws
        - openshift
  status:
    conditions:
    - lastTransitionTime: "2025-03-10T05:14:25Z"
      message: Reconcile complete
      reason: Complete
      status: "True"
      type: Reconciled
kind: List
metadata:
  resourceVersion: ""

backup repository:

oc get backuprepository -o yaml
apiVersion: v1
items:
- apiVersion: velero.io/v1
  kind: BackupRepository
  metadata:
    creationTimestamp: "2025-03-10T05:21:30Z"
    generateName: mysql-ts-dpa-1-kopia-
    generation: 2
    labels:
      velero.io/repository-type: kopia
      velero.io/storage-location: ts-dpa-1
      velero.io/volume-namespace: mysql
    name: mysql-ts-dpa-1-kopia-nkbvd
    namespace: openshift-adp
    resourceVersion: "39558"
    uid: 4de30be8-833b-4242-a76f-3692232bd7ee
  spec:
    backupStorageLocation: ts-dpa-1
    maintenanceFrequency: 1h0m0s
    repositoryType: kopia
    resticIdentifier: ""
    volumeNamespace: mysql
  status:
    message: 'error to get identifier for repo mysql-ts-dpa-1-kopia-nkbvd: failed
      to detect the region via bucket: oadp113331tvmz7: operation error S3: HeadBucket,
      get identity: get credentials: failed to refresh cached credentials, failed
      to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds:
      GetMetadata, exceeded maximum number of attempts, 3, request send failed, Get
      "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp
      169.254.169.254:80: connect: connection refused'
    phase: NotReady
kind: List
metadata:
  resourceVersion: ""

Backup:

velero describe backup b2 -n openshift-adp --details
Name:         b2
Namespace:    openshift-adp
Labels:       velero.io/storage-location=ts-dpa-1
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.32.2
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=32

Phase:  PartiallyFailed (run `velero backup logs b2` for more information)


Errors:
  Velero:     <none>
  Cluster:    <none>
  Namespaces: <none>

Namespaces:
  Included:  mysql
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  ts-dpa-1

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          true
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  10m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2025-03-10 10:50:02 +0530 IST
Completed:  2025-03-10 10:52:47 +0530 IST

Expiration:  2025-04-09 10:50:01 +0530 IST

Total items to be backed up:  52
Items backed up:              52

Backup Item Operations:
  Operation for persistentvolumeclaims mysql/mysql-data:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-76349720-e71c-44bb-a7de-1b4cfd697fa3.1bd3ee13-8999-42913324b
    Items to Update:
                           datauploads.velero.io openshift-adp/b2-pwqq9
    Phase:                 Failed
    Operation Error:       data path backup failed: Failed to run data path service for DataUpload b2-pwqq9: error to initialize data path: error to ensure backup repository ts-dpa-1-mysql-kopia: failed to wait BackupRepository, errored early: backup repository is not ready: error to get identifier for repo mysql-ts-dpa-1-kopia-nkbvd: failed to detect the region via bucket: oadp113331tvmz7: operation error S3: HeadBucket, get identity: get credentials: failed to refresh cached credentials, failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: connection refused
    Progress description:  Failed
    Created:               2025-03-10 10:50:08 +0530 IST
    Started:               2025-03-10 10:51:30 +0530 IST
    Updated:               2025-03-10 10:52:38 +0530 IST
  Operation for persistentvolumeclaims mysql/mysql-data1:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-76349720-e71c-44bb-a7de-1b4cfd697fa3.1586673a-bf08-4cabe6390
    Items to Update:
                           datauploads.velero.io openshift-adp/b2-thtqc
    Phase:                 Failed
    Operation Error:       data path backup failed: Failed to run data path service for DataUpload b2-thtqc: error to initialize data path: error to ensure backup repository ts-dpa-1-mysql-kopia: failed to wait BackupRepository, errored early: backup repository is not ready: error to get identifier for repo mysql-ts-dpa-1-kopia-nkbvd: failed to detect the region via bucket: oadp113331tvmz7: operation error S3: HeadBucket, get identity: get credentials: failed to refresh cached credentials, failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: connection refused
    Progress description:  Failed
    Created:               2025-03-10 10:50:14 +0530 IST
    Started:               2025-03-10 10:51:35 +0530 IST
    Updated:               2025-03-10 10:52:39 +0530 IST
Resource List:
  apps/v1/Deployment:
    - mysql/mysql
  apps/v1/ReplicaSet:
    - mysql/mysql-64c9d6466
  authorization.openshift.io/v1/RoleBinding:
    - mysql/admin
    - mysql/system:deployers
    - mysql/system:image-builders
    - mysql/system:image-pullers
  discovery.k8s.io/v1/EndpointSlice:
    - mysql/mysql-j6v4l
  rbac.authorization.k8s.io/v1/RoleBinding:
    - mysql/admin
    - mysql/system:deployers
    - mysql/system:image-builders
    - mysql/system:image-pullers
  v1/ConfigMap:
    - mysql/kube-root-ca.crt
    - mysql/openshift-service-ca.crt
  v1/Endpoints:
    - mysql/mysql
  v1/Event:
    - mysql/mysql-64c9d6466-bzmg6.182b5928dd7ba443
    - mysql/mysql-64c9d6466-bzmg6.182b59296cc43caa
    - mysql/mysql-64c9d6466-bzmg6.182b5929b3b37038
    - mysql/mysql-64c9d6466-bzmg6.182b592b68b806b5
    - mysql/mysql-64c9d6466-bzmg6.182b592b6a2782a0
    - mysql/mysql-64c9d6466-bzmg6.182b592d3d0f75d0
    - mysql/mysql-64c9d6466-bzmg6.182b592d405628fe
    - mysql/mysql-64c9d6466-bzmg6.182b592d40c25ed2
    - mysql/mysql-64c9d6466.182b5928289f1252
    - mysql/mysql-data.182b5927cec28643
    - mysql/mysql-data.182b592829d489af
    - mysql/mysql-data.182b592829fa5ec8
    - mysql/mysql-data.182b5928a870343c
    - mysql/mysql-data1.182b5927f210ab45
    - mysql/mysql-data1.182b59282aa618c2
    - mysql/mysql-data1.182b59282aac957a
    - mysql/mysql-data1.182b5928a89a58e2
    - mysql/mysql.182b59282799b275
    - mysql/velero-mysql-data-6br2j.182b5941d553ac8b
    - mysql/velero-mysql-data-6br2j.182b59420e1a6f76
    - mysql/velero-mysql-data-6br2j.182b595189e2b5ff
    - mysql/velero-mysql-data1-62cd7.182b594309095ea9
    - mysql/velero-mysql-data1-62cd7.182b594345887d96
    - mysql/velero-mysql-data1-62cd7.182b5952bf580ea4
  v1/Namespace:
    - mysql
  v1/PersistentVolume:
    - pvc-1586673a-bf08-4ca4-a3a0-114e2db7db31
    - pvc-1bd3ee13-8999-4291-9825-edc84c513130
  v1/PersistentVolumeClaim:
    - mysql/mysql-data
    - mysql/mysql-data1
  v1/Pod:
    - mysql/mysql-64c9d6466-bzmg6
  v1/Secret:
    - mysql/builder-dockercfg-z5xcm
    - mysql/default-dockercfg-gcdv8
    - mysql/deployer-dockercfg-nlpm4
    - mysql/mysql
  v1/Service:
    - mysql/mysql
  v1/ServiceAccount:
    - mysql/builder
    - mysql/default
    - mysql/deployer

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots:
    mysql/mysql-data:
      Data Movement:
        Operation ID: du-76349720-e71c-44bb-a7de-1b4cfd697fa3.1bd3ee13-8999-42913324b
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 0
        Result: failed
    mysql/mysql-data1:
      Data Movement:
        Operation ID: du-76349720-e71c-44bb-a7de-1b4cfd697fa3.1586673a-bf08-4cabe6390
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 0
        Result: failed

  Pod Volume Backups: <none included>

HooksAttempted:  0
HooksFailed:     0

oc get bsl -o yaml
apiVersion: v1
items:
- apiVersion: velero.io/v1
  kind: BackupStorageLocation
  metadata:
    creationTimestamp: "2025-03-10T15:13:31Z"
    generation: 3
    labels:
      app.kubernetes.io/component: bsl
      app.kubernetes.io/instance: ts-dpa-1
      app.kubernetes.io/managed-by: oadp-operator
      app.kubernetes.io/name: oadp-operator-velero
      openshift.io/oadp: "True"
      openshift.io/oadp-registry: "True"
    name: ts-dpa-1
    namespace: openshift-adp
    ownerReferences:
    - apiVersion: oadp.openshift.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: DataProtectionApplication
      name: ts-dpa
      uid: 74a07f8c-2bc7-4849-9d49-7f2c3d38509f
    resourceVersion: "189684"
    uid: 24e72446-ee7c-47ea-a0ab-b19f264d178a
  spec:
    credential:
      key: cloud
      name: cloud-credentials
    default: true
    objectStorage:
      bucket: oadp113331tvmz7
      prefix: velero
    provider: aws
  status:
    lastSyncedTime: "2025-03-10T15:13:36Z"
    lastValidationTime: "2025-03-10T15:13:36Z"
    phase: Available
kind: List
metadata:
  resourceVersion: ""

is cloned by

OADP-6065 OADP-1.5 Backup Partially Fails with AWS bucket with " region not found ", in absence of VSL spec.

New

is related to

OADP-6065 OADP-1.5 Backup Partially Fails with AWS bucket with " region not found ", in absence of VSL spec.

New

links to

openshift/oadp-operator#1740: OADP-5777: Add automatic S3 bucket region detection and document AWS HeadBucket API behavior

Details

Description

Description of problem: slack

Backup failing if "region" is not provided for AWS bucket in DPA, as the BackupRepository complains for region. But DPA reconciles successfully.

Midstream "main"

Always

Actual results:

Backup PArtially fails with Error in BackupRepository

Backup should work

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates