Loading...

Type: Bug
Resolution: Done-Errata
Priority: Blocker
Fix Version/s: OADP 1.4.2
Affects Version/s: OADP 1.4.2
Component/s: velero
Labels:

Story Points:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Fixed in Build:
oadp-operator-bundle-container-1.4.2-20
QEStatus:
ToDo
Intelligence Requested:
Market:

Cost of Delay:
0
WSJF:
0.000
Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

Regression:
None

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

When we perform a datamover backup of any workload (lets say, in namespace W) using OADP installed in namespace A (lets say), but restore the workload from an OADP installed in the namespace B (lets say), The restore fails.

Version-Release number of selected component (if applicable):

OADP 1.4.2

How reproducible:

Always

Steps to Reproduce:
1. Install OADP in namespace openshift-adp
2. Install OADP in namespace oadp2
3. Deploy workload in namespace mysql-test
4. Perform backup using OADP in openshift-adp
5. Perform restore using OADP in oadp2

Actual results:

Restore results in Partially Failed.

Expected results:

Restore should be successfull.

Additional info:

The Application to Backup / Restore

$ oc get all -n mysql-test
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                         READY   STATUS    RESTARTS   AGE
pod/mysql-5f7547b97f-xxtz6   1/1     Running   0          3m36s

NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/mysql   ClusterIP   172.30.39.13   <none>        3306/TCP   3m39s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql   1/1     1            1           3m37s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-5f7547b97f   1         1         1       3m38s

OADP installed in openshift-adp with DPA created.

$ oc get pods -n openshift-adp
NAME                                                READY   STATUS    RESTARTS   AGE
node-agent-4qp9r                                    1/1     Running   0          12s
node-agent-dlgmz                                    1/1     Running   0          12s
node-agent-vk829                                    1/1     Running   0          12s
openshift-adp-controller-manager-5f9c4dbdf4-f6dj2   1/1     Running   0          3h37m
velero-5dcbc86f75-7sgjq                             1/1     Running   0          12s

DPA :

apiVersion: v1
items:
- apiVersion: oadp.openshift.io/v1alpha1
  kind: DataProtectionApplication
  metadata:
    creationTimestamp: "2025-01-09T09:58:05Z"
    generation: 1
    name: ts-dpa
    namespace: openshift-adp
    resourceVersion: "135714"
    uid: 38b93cc2-78ec-4666-931e-630098dd60a7
  spec:
    backupLocations:
    - velero:
        credential:
          key: cloud
          name: cloud-credentials-gcp
        default: true
        objectStorage:
          bucket: oadp107141nr57r
          prefix: velero
        provider: gcp
    configuration:
      nodeAgent:
        enable: true
        uploaderType: kopia
      velero:
        defaultPlugins:
        - csi
        - gcp
        - openshift
  status:
    conditions:
    - lastTransitionTime: "2025-01-09T09:58:05Z"
      message: Reconcile complete
      reason: Complete
      status: "True"
      type: Reconciled
kind: List
metadata:
  resourceVersion: ""

Backup CR

apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    velero.io/resource-timeout: 10m0s
    velero.io/source-cluster-k8s-gitversion: v1.30.7
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "30"
  creationTimestamp: "2025-01-09T10:16:55Z"
  generation: 8
  labels:
    velero.io/storage-location: ts-dpa-1
  name: backup1
  namespace: openshift-adp
  resourceVersion: "144209"
  uid: e760c716-7851-4cc3-9316-75fc3a6288b4
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  includedNamespaces:
  - mysql-test
  itemOperationTimeout: 0h3m0s
  snapshotMoveData: true
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
status:
  backupItemOperationsAttempted: 2
  backupItemOperationsCompleted: 2
  completionTimestamp: "2025-01-09T10:18:50Z"
  expiration: "2025-02-08T10:16:55Z"
  formatVersion: 1.1.0
  hookStatus: {}
  phase: Completed
  progress:
    itemsBackedUp: 45
    totalItems: 45
  startTimestamp: "2025-01-09T10:16:56Z"
  version: 1

Restore CR in oadp2 namespace:

$ oc get restore restore1 -o yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: "2025-01-09T10:25:30Z"
  finalizers:
  - restores.velero.io/external-resources-finalizer
  generation: 7
  name: restore1
  namespace: oadp2
  resourceVersion: "146609"
  uid: 3ce81300-46c1-4806-902e-79fc6318a3f0
spec:
  backupName: backup1
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 1h0m0s
status:
  completionTimestamp: "2025-01-09T10:25:33Z"
  errors: 2
  hookStatus: {}
  phase: PartiallyFailed
  progress:
    itemsRestored: 30
    totalItems: 30
  startTimestamp: "2025-01-09T10:25:30Z"
  warnings: 6

Restore CR (With Namespace Mapping) in oadp2:

apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: "2025-01-09T10:28:01Z"
  finalizers:
  - restores.velero.io/external-resources-finalizer
  generation: 7
  name: restore2
  namespace: oadp2
  resourceVersion: "147521"
  uid: c12a9323-6e99-44d7-b59e-7cb01065b35a
spec:
  backupName: backup1
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 1h0m0s
  namespaceMapping:
    openshift-adp: oadp2
status:
  completionTimestamp: "2025-01-09T10:28:04Z"
  errors: 2
  hookStatus: {}
  phase: PartiallyFailed
  progress:
    itemsRestored: 30
    totalItems: 30
  startTimestamp: "2025-01-09T10:28:01Z"
  warnings: 6

Describe restore output:

 velero describe restore restore1 -n oadp2 --details -n oadp2
Name:         restore1
Namespace:    oadp2
Labels:       <none>
Annotations:  <none>

Phase:                       PartiallyFailed (run 'velero restore logs restore1' for more information)
Total items to be restored:  30
Items restored:              30

Started:    2025-01-09 15:55:30 +0530 IST
Completed:  2025-01-09 15:55:33 +0530 IST

Warnings:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    mysql-test:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                 could not restore, ConfigMap "openshift-service-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                 could not restore, RoleBinding "admin" already exists. Warning: the in-cluster version is different than the backed-up version
                 could not restore, RoleBinding "system:deployers" already exists. Warning: the in-cluster version is different than the backed-up version
                 could not restore, RoleBinding "system:image-builders" already exists. Warning: the in-cluster version is different than the backed-up version
                 could not restore, RoleBinding "system:image-pullers" already exists. Warning: the in-cluster version is different than the backed-up version

Errors:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    mysql-test:  error preparing persistentvolumeclaims/mysql-test/mysql-data: rpc error: code = Unknown desc = fail get DataUploadResult for restore: restore1: no DataUpload result cm found with labels velero.io/pvc-namespace-name=mysql-test.mysql-data,velero.io/restore-uid=3ce81300-46c1-4806-902e-79fc6318a3f0,velero.io/resource-usage=DataUpload
                 error preparing persistentvolumeclaims/mysql-test/mysql-data1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: restore1: no DataUpload result cm found with labels velero.io/pvc-namespace-name=mysql-test.mysql-data1,velero.io/restore-uid=3ce81300-46c1-4806-902e-79fc6318a3f0,velero.io/resource-usage=DataUpload

Backup:  backup1

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

is depended on by

ACM-16218 Virtualization - Package VM backup and restore policy with ACM

Review

links to

Datamover restore fails if restore cluster is using velero installed in a different namespace than velero on the backup cluster

https://github.com/vmware-tanzu/velero/pull/8621

https://github.com/vmware-tanzu/velero/pull/8622

openshift/velero#358: OADP-5460 Datamover new ns oadp

RHSA-2024:138985 OpenShift API for Data Protection (OADP) 1.4.2 security and bug fix update

mentioned in: Page Loading...; Page Loading...; Page Loading...

mentioned on

Merge request - Updated US source to: 53575df Always create DataUpload configmap in restore namespace (#358)

(1 links to, 3 mentioned in, 1 mentioned on)

There are no Sub-Tasks for this issue.

Details

Description

Description of problem:

When we perform a datamover backup of any workload (lets say, in namespace W) using OADP installed in namespace A (lets say), but restore the workload from an OADP installed in the namespace B (lets say), The restore fails.

OADP 1.4.2

Always

Actual results:

Restore results in Partially Failed.

Restore should be successfull.

Attachments

Issue Links

Easy Agile Planning Poker

Sub-Tasks

Activity

People

Dates