Loading...

Type: Bug
Resolution: Done-Errata
Priority: Major
Fix Version/s: OADP 1.5.1
Affects Version/s: OADP 1.5.0
Component/s: None
Labels:

Activity Type:
Quality / Stability / Reliability
Story Points:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Fixed in Build:
oadp-operator-bundle-container-1.5.1-32
QEStatus:
ToDo
Intelligence Requested:
Market:

Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

Regression:
None

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem:

Ran full regression with hypershift plugin enabled. Test cases where includedNamespace spec is nil are failing partially. Same case passes without hypershift plugin.

Version-Release number of selected component (if applicable):
OADP 1.5.0

How reproducible:
Always

Steps to Reproduce:
1. Created DPA with hypershift plugin enabled

$ oc get dpa ts-dpa -o yaml
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  creationTimestamp: "2025-06-09T05:54:00Z"
  generation: 4
  name: ts-dpa
  namespace: openshift-adp
  resourceVersion: "52608"
  uid: c07739e3-ab86-4ac0-ba9e-abd464187157
spec:
  backupLocations:
  - velero:
      credential:
        key: cloud
        name: cloud-credentials-gcp
      default: true
      objectStorage:
        bucket: oadp121511v8z6s
        prefix: velero
      provider: gcp
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      defaultPlugins:
      - gcp
      - openshift
      - hypershift
      disableFsBackup: false
  logFormat: text
status:
  conditions:
  - lastTransitionTime: "2025-06-09T05:54:00Z"
    message: Reconcile complete
    reason: Complete
    status: "True"
    type: Reconciled

2. Deployed application

oc get pod -n ocp-mysql
NAME                    READY   STATUS    RESTARTS   AGE
mysql-64c9d6466-gqp2t   1/1     Running   0          38m

3. Triggered backup with labelSelector. (Make sure includedNamespace field is nil)

$ oc get backup test-backup1 -o yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    velero.io/resource-timeout: 10m0s
    velero.io/source-cluster-k8s-gitversion: v1.32.3
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "32"
  creationTimestamp: "2025-06-09T06:07:23Z"
  generation: 7
  labels:
    velero.io/storage-location: ts-dpa-1
  name: test-backup1
  namespace: openshift-adp
  resourceVersion: "40925"
  uid: 9d1e5d28-13d0-48a7-9a7b-4fa468446d0f
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: true
  itemOperationTimeout: 4h0m0s
  labelSelector:
    matchLabels:
      app: mysql
  snapshotMoveData: false
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
status:
  completionTimestamp: "2025-06-09T06:07:48Z"
  expiration: "2025-07-09T06:07:23Z"
  formatVersion: 1.1.0
  hookStatus: {}
  phase: Completed
  progress:
    itemsBackedUp: 24
    totalItems: 24
  startTimestamp: "2025-06-09T06:07:24Z"
  version: 1

4. Removed app namespace and triggered restore

$ oc get restore -o yaml test-restore1 
apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: "2025-06-09T06:51:18Z"
  finalizers:
  - restores.velero.io/external-resources-finalizer
  generation: 7
  name: test-restore1
  namespace: openshift-adp
  resourceVersion: "52875"
  uid: c061e9a9-aa70-441f-909b-9f3565fb98a7
spec:
  backupName: test-backup1
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 4h0m0s
status:
  completionTimestamp: "2025-06-09T06:51:21Z"
  errors: 10
  hookStatus: {}
  phase: PartiallyFailed
  progress:
    itemsRestored: 24
    totalItems: 24
  startTimestamp: "2025-06-09T06:51:18Z"

Actual results:

Restore is partially failed with below error.

$ oc logs velero-86c7b6c44c-nhqv8 | grep error
Defaulted container "velero" out of: velero, velero-plugin-for-gcp (init), openshift-velero-plugin (init), hypershift-oadp-plugin (init)
time="2025-06-09T06:50:46Z" level=warning msg="Failed to set default backup storage location at server start" backupStorageLocation=default error="backupstoragelocations.velero.io \"default\" not found" logSource="/remote-source/velero/app/pkg/cmd/server/server.go:385"
time="2025-06-09T06:51:20Z" level=error msg="Namespace ocp-mysql, resource restore error: error preparing persistentvolumeclaims/ocp-mysql/mysql-data: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1
time="2025-06-09T06:51:20Z" level=error msg="Namespace ocp-mysql, resource restore error: error preparing persistentvolumeclaims/ocp-mysql/mysql-data1: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1
time="2025-06-09T06:51:20Z" level=error msg="Namespace ocp-mysql, resource restore error: error preparing secrets/ocp-mysql/mysql: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1
time="2025-06-09T06:51:20Z" level=error msg="Namespace ocp-mysql, resource restore error: error preparing pods/ocp-mysql/mysql-64c9d6466-gj5fg: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1
time="2025-06-09T06:51:20Z" level=error msg="Namespace ocp-mysql, resource restore error: error preparing deployments.apps/ocp-mysql/mysql: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1
time="2025-06-09T06:51:20Z" level=error msg="Namespace mysql, resource restore error: error preparing persistentvolumeclaims/mysql/mysql-data: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1
time="2025-06-09T06:51:20Z" level=error msg="Namespace mysql, resource restore error: error preparing persistentvolumeclaims/mysql/mysql-data1: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1
time="2025-06-09T06:51:20Z" level=error msg="Namespace mysql, resource restore error: error preparing secrets/mysql/mysql: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1
time="2025-06-09T06:51:20Z" level=error msg="Namespace mysql, resource restore error: error preparing pods/mysql/mysql-64c9d6466-7mz59: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1
time="2025-06-09T06:51:20Z" level=error msg="Namespace mysql, resource restore error: error preparing deployments.apps/mysql/mysql: rpc error: code = Unknown desc = backup or included namespaces is nil" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:603" restore=openshift-adp/test-restore1