[OADP-1732] [IBM QE-Z] Verify Bug OADP-1640 - CSI backup/restore to FS backup/restore fails - Red Hat Issue Tracker

Type: Sub-task
Resolution: Done
Priority: Undefined
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
None

Story Points:
4
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
QEStatus:
ToDo
Intelligence Requested:
Market:

WSJF:
0
Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

https://access.redhat.com/support/cases/#/case/03480591/

Completed backup/restore with CSI snapshotting enabled
Complete new FS backup of previously restored volumes from CSI snapshot
Restore from FS backup
Restore fails as PVR never completes, and PVC remains pending with event:

failed to provision volume with StorageClass "gp2-csi": error getting handle for DataSource Type VolumeSnapshot by Name velero-mssql-pvc-v787x: error getting snapshot velero-mssql-pvc-v787x from api server: volumesnapshots.snapshot.storage.k8s.io "velero-mssql-pvc-v787x" not found

PVC dataSourceRef is reset here: https://github.com/vmware-tanzu/velero-plugin-for-csi/blob/main/internal/restore/pvc_action.go#L71-L81 which causes the error.

SHIVA SAI K (Inactive) added a comment - 2023/05/02 6:34 AM - edited

Verified with ~~OADP-1~~.2 with build 55

1. Deploy stateful app

# oc get po -n maria
NAME                   READY   STATUS    RESTARTS   AGE
mysql-c96c5db4-sh9bt   1/1     Running   0          83s

2. Perform Backup/Restore using CSI

# oc get dpa ts-dpa -o yaml
 spec:
  backupLocations:
  - velero:
      config:
        profile: default
        region: us-east-1
      credential:
        key: cloud
        name: cloud-credentials
      default: true
      objectStorage:
        bucket: acmeairoadp
        prefix: velero-dm-may-1-maria-restic
      provider: aws
  configuration:
    restic:
      enable: true
    velero:
      defaultPlugins:
      - openshift
      - aws
status:
  conditions:
  - lastTransitionTime: "2023-05-02T05:16:28Z"
    message: Reconcile complete
    reason: Complete
    status: "True"
    type: Reconciled

# oc get backup backup1 -oyaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"velero.io/v1","kind":"Backup","metadata":{"annotations":{},"labels":{"velero.io/storage-location":"default"},"name":"backup1","namespace":"openshift-adp"},"spec":{"hooks":{},"includedNamespaces":["maria"],"storageLocation":"ts-dpa-1","ttl":"720h0m0s"}}
    velero.io/source-cluster-k8s-gitversion: v1.25.2+cd98eda
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "25"
  creationTimestamp: "2023-05-02T05:17:22Z"
  generation: 4
  labels:
    velero.io/storage-location: ts-dpa-1
  name: backup1
  namespace: openshift-adp
  resourceVersion: "1481079"
  uid: 0345ae53-00e2-402d-a20c-54b58f7d03a3
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  hooks: {}
  includedNamespaces:
  - maria
  itemOperationTimeout: 1h0m0s
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
status:
  completionTimestamp: "2023-05-02T05:18:11Z"
  csiVolumeSnapshotsAttempted: 1
  csiVolumeSnapshotsCompleted: 1
  expiration: "2023-06-01T05:17:23Z"
  formatVersion: 1.1.0
  phase: Completed
  startTimestamp: "2023-05-02T05:17:29Z"
  version: 1

# oc get restore restore1 -oyaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"velero.io/v1","kind":"Restore","metadata":{"annotations":{},"name":"restore1","namespace":"openshift-adp"},"spec":{"backupName":"backup1","excludedResources":["nodes","events","events.events.k8s.io","backups.velero.io","restores.velero.io","resticrepositories.velero.io"],"restorePVs":true}}
  creationTimestamp: "2023-05-02T05:19:07Z"
  generation: 9
  name: restore1
  namespace: openshift-adp
  resourceVersion: "1483262"
  uid: a431f87f-9837-4298-912a-1475fc314a8a
spec:
  backupName: backup1
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 1h0m0s
  restorePVs: true
status:
  completionTimestamp: "2023-05-02T05:19:49Z"
  phase: Completed
  progress:
    itemsRestored: 34
    totalItems: 34
  startTimestamp: "2023-05-02T05:19:07Z"
  warnings: 13

# oc get po -n maria
NAME                   READY   STATUS    RESTARTS   AGE
mysql-c96c5db4-sh9bt   1/1     Running   0          36m 

# oc rsh -nmaria mysql-c96c5db4-sh9bt
sh-4.4$ mysql -uroot MYSQL_DATABASE -e "show tables";
+--------------------------+
| Tables_in_MYSQL_DATABASE |
+--------------------------+
| authors                  |
| posts                    |
+--------------------------+
sh-4.4$ exit

3. Perform Restic Backup on the same app namespace

# oc get dpa ts-dpa -oyaml
spec:
  backupLocations:
  - velero:
      config:
        profile: default
        region: us-east-1
      credential:
        key: cloud
        name: cloud-credentials
      default: true
      objectStorage:
        bucket: acmeairoadp
        prefix: velero-dm-may-1-maria-restic
      provider: aws
  configuration:
    restic:
      enable: true
    velero:
      defaultPlugins:
      - openshift
      - aws
status:
  conditions:
  - lastTransitionTime: "2023-05-02T05:16:28Z"
    message: Reconcile complete
    reason: Complete
    status: "True"
    type: Reconciled

# oc get backup backup-restic -o yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"velero.io/v1","kind":"Backup","metadata":{"annotations":{},"labels":{"velero.io/storage-location":"default"},"name":"backup-restic","namespace":"openshift-adp"},"spec":{"defaultVolumesToFsBackup":true,"hooks":{},"includedNamespaces":["maria"],"storageLocation":"ts-dpa-1","ttl":"720h0m0s"}}
    velero.io/source-cluster-k8s-gitversion: v1.25.2+cd98eda
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "25"
  creationTimestamp: "2023-05-02T06:05:26Z"
  generation: 4
  labels:
    velero.io/storage-location: ts-dpa-1
  name: backup-restic
  namespace: openshift-adp
  resourceVersion: "1542642"
  uid: 028348a1-ad14-4be4-b876-ca7c3de56d98
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: true
  hooks: {}
  includedNamespaces:
  - maria
  itemOperationTimeout: 1h0m0s
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
status:
  completionTimestamp: "2023-05-02T06:07:43Z"
  expiration: "2023-06-01T06:05:26Z"
  formatVersion: 1.1.0
  phase: Completed
  startTimestamp: "2023-05-02T06:05:26Z"
  version: 1

# oc get podvolumebackup -A
NAMESPACE       NAME                  STATUS      CREATED   NAMESPACE   POD                    VOLUME       REPOSITORY ID                                                                         UPLOADER TYPE   STORAGE LOCATION   AGE
openshift-adp   backup-restic-9qd7x   Completed   16m       maria       mysql-c96c5db4-sh9bt   mysql-data   s3:s3-us-east-1.amazonaws.com/acmeairoadp/velero-dm-may-1-maria-restic/restic/maria   restic          ts-dpa-1           16m

# oc get restore restore-restic -o yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"velero.io/v1","kind":"Restore","metadata":{"annotations":{},"name":"restore-restic","namespace":"openshift-adp"},"spec":{"backupName":"backup-restic","excludedResources":["nodes","events","events.events.k8s.io","backups.velero.io","restores.velero.io","resticrepositories.velero.io"]}}
  creationTimestamp: "2023-05-02T06:09:30Z"
  generation: 8
  name: restore-restic
  namespace: openshift-adp
  resourceVersion: "1548708"
  uid: 2170eda6-a97f-4cf9-85e4-0f0fb0b7dade
spec:
  backupName: backup-restic
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 1h0m0s
status:
  completionTimestamp: "2023-05-02T06:12:35Z"
  phase: Completed
  progress:
    itemsRestored: 36
    totalItems: 36
  startTimestamp: "2023-05-02T06:09:30Z"
  warnings: 13

# oc get podvolumerestore -A
NAMESPACE       NAME                   NAMESPACE   POD                    UPLOADER TYPE   VOLUME       STATUS      TOTALBYTES   BYTESDONE   AGE
openshift-adp   restore-restic-9pmbl   maria       mysql-c96c5db4-sh9bt   restic          mysql-data   Completed   107855553    107855553   12m

# oc get po -n maria
NAME                   READY   STATUS    RESTARTS   AGE
mysql-c96c5db4-sh9bt   1/1     Running   0          2m57s

]# oc get pvc -n maria mysql -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    backup.velero.io/must-include-additional-items: "true"
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    velero.io/backup-name: backup1
    velero.io/volume-snapshot-name: velero-mysql-z9bcz
    volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
  creationTimestamp: "2023-05-02T06:09:57Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: mysql
    backup.velero.io/must-include-additional-items: "true"
    testlabel: selectors
    testlabel2: foo
    velero.io/backup-name: backup-restic
    velero.io/restore-name: restore-restic
    velero.io/volume-snapshot-name: velero-mysql-z9bcz
  name: mysql
  namespace: maria
  resourceVersion: "1545457"
  uid: 7c30fb3f-458a-49c6-b108-77c57652aff0
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: ocs-storagecluster-cephfs
  volumeMode: Filesystem
  volumeName: pvc-7c30fb3f-458a-49c6-b108-77c57652aff0
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  phase: Bound

# oc rsh -nmaria mysql-c96c5db4-sh9bt
Defaulted container "mysql" out of: mysql, restore-wait (init)
sh-4.4$ mysql -uroot MYSQL_DATABASE -e "show tables";
+--------------------------+
| Tables_in_MYSQL_DATABASE |
+--------------------------+
| authors                  |
| posts                    |
+--------------------------+
sh-4.4$ exit
exit
# oc get vs -A
NAMESPACE   NAME                 READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT             RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT   CREATIONTIME   AGE
maria       velero-mysql-qgg44   false                    velero-velero-mysql-qgg44-slvkm                 csi-vsc                                          2m4s

SHIVA SAI K (Inactive) added a comment - 2023/05/02 6:34 AM - edited Verified with OADP-1 .2 with build 55 1. Deploy stateful app # oc get po -n maria NAME READY STATUS RESTARTS AGE mysql-c96c5db4-sh9bt 1/1 Running 0 83s 2. Perform Backup/Restore using CSI # oc get dpa ts-dpa -o yaml spec: backupLocations: - velero: config: profile: default region: us-east-1 credential: key: cloud name: cloud-credentials default : true objectStorage: bucket: acmeairoadp prefix: velero-dm-may-1-maria-restic provider: aws configuration: restic: enable: true velero: defaultPlugins: - openshift - aws status: conditions: - lastTransitionTime: "2023-05-02T05:16:28Z" message: Reconcile complete reason: Complete status: "True" type: Reconciled # oc get backup backup1 -oyaml apiVersion: velero.io/v1 kind: Backup metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { "apiVersion" : "velero.io/v1" , "kind" : "Backup" , "metadata" :{ "annotations" :{}, "labels" :{ "velero.io/storage-location" : " default " }, "name" : "backup1" , "namespace" : "openshift-adp" }, "spec" :{ "hooks" :{}, "includedNamespaces" :[ "maria" ], "storageLocation" : "ts-dpa-1" , "ttl" : "720h0m0s" }} velero.io/source-cluster-k8s-gitversion: v1.25.2+cd98eda velero.io/source-cluster-k8s-major-version: "1" velero.io/source-cluster-k8s-minor-version: "25" creationTimestamp: "2023-05-02T05:17:22Z" generation: 4 labels: velero.io/storage-location: ts-dpa-1 name: backup1 namespace: openshift-adp resourceVersion: "1481079" uid: 0345ae53-00e2-402d-a20c-54b58f7d03a3 spec: csiSnapshotTimeout: 10m0s defaultVolumesToFsBackup: false hooks: {} includedNamespaces: - maria itemOperationTimeout: 1h0m0s storageLocation: ts-dpa-1 ttl: 720h0m0s status: completionTimestamp: "2023-05-02T05:18:11Z" csiVolumeSnapshotsAttempted: 1 csiVolumeSnapshotsCompleted: 1 expiration: "2023-06-01T05:17:23Z" formatVersion: 1.1.0 phase: Completed startTimestamp: "2023-05-02T05:17:29Z" version: 1 # oc get restore restore1 -oyaml apiVersion: velero.io/v1 kind: Restore metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { "apiVersion" : "velero.io/v1" , "kind" : "Restore" , "metadata" :{ "annotations" :{}, "name" : "restore1" , "namespace" : "openshift-adp" }, "spec" :{ "backupName" : "backup1" , "excludedResources" :[ "nodes" , "events" , "events.events.k8s.io" , "backups.velero.io" , "restores.velero.io" , "resticrepositories.velero.io" ], "restorePVs" : true }} creationTimestamp: "2023-05-02T05:19:07Z" generation: 9 name: restore1 namespace: openshift-adp resourceVersion: "1483262" uid: a431f87f-9837-4298-912a-1475fc314a8a spec: backupName: backup1 excludedResources: - nodes - events - events.events.k8s.io - backups.velero.io - restores.velero.io - resticrepositories.velero.io - csinodes.storage.k8s.io - volumeattachments.storage.k8s.io - backuprepositories.velero.io itemOperationTimeout: 1h0m0s restorePVs: true status: completionTimestamp: "2023-05-02T05:19:49Z" phase: Completed progress: itemsRestored: 34 totalItems: 34 startTimestamp: "2023-05-02T05:19:07Z" warnings: 13 # oc get po -n maria NAME READY STATUS RESTARTS AGE mysql-c96c5db4-sh9bt 1/1 Running 0 36m # oc rsh -nmaria mysql-c96c5db4-sh9bt sh-4.4$ mysql -uroot MYSQL_DATABASE -e "show tables" ; +--------------------------+ | Tables_in_MYSQL_DATABASE | +--------------------------+ | authors | | posts | +--------------------------+ sh-4.4$ exit 3. Perform Restic Backup on the same app namespace # oc get dpa ts-dpa -oyaml spec: backupLocations: - velero: config: profile: default region: us-east-1 credential: key: cloud name: cloud-credentials default : true objectStorage: bucket: acmeairoadp prefix: velero-dm-may-1-maria-restic provider: aws configuration: restic: enable: true velero: defaultPlugins: - openshift - aws status: conditions: - lastTransitionTime: "2023-05-02T05:16:28Z" message: Reconcile complete reason: Complete status: "True" type: Reconciled # oc get backup backup-restic -o yaml apiVersion: velero.io/v1 kind: Backup metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { "apiVersion" : "velero.io/v1" , "kind" : "Backup" , "metadata" :{ "annotations" :{}, "labels" :{ "velero.io/storage-location" : " default " }, "name" : "backup-restic" , "namespace" : "openshift-adp" }, "spec" :{ "defaultVolumesToFsBackup" : true , "hooks" :{}, "includedNamespaces" :[ "maria" ], "storageLocation" : "ts-dpa-1" , "ttl" : "720h0m0s" }} velero.io/source-cluster-k8s-gitversion: v1.25.2+cd98eda velero.io/source-cluster-k8s-major-version: "1" velero.io/source-cluster-k8s-minor-version: "25" creationTimestamp: "2023-05-02T06:05:26Z" generation: 4 labels: velero.io/storage-location: ts-dpa-1 name: backup-restic namespace: openshift-adp resourceVersion: "1542642" uid: 028348a1-ad14-4be4-b876-ca7c3de56d98 spec: csiSnapshotTimeout: 10m0s defaultVolumesToFsBackup: true hooks: {} includedNamespaces: - maria itemOperationTimeout: 1h0m0s storageLocation: ts-dpa-1 ttl: 720h0m0s status: completionTimestamp: "2023-05-02T06:07:43Z" expiration: "2023-06-01T06:05:26Z" formatVersion: 1.1.0 phase: Completed startTimestamp: "2023-05-02T06:05:26Z" version: 1 # oc get podvolumebackup -A NAMESPACE NAME STATUS CREATED NAMESPACE POD VOLUME REPOSITORY ID UPLOADER TYPE STORAGE LOCATION AGE openshift-adp backup-restic-9qd7x Completed 16m maria mysql-c96c5db4-sh9bt mysql-data s3:s3-us-east-1.amazonaws.com/acmeairoadp/velero-dm-may-1-maria-restic/restic/maria restic ts-dpa-1 16m # oc get restore restore-restic -o yaml apiVersion: velero.io/v1 kind: Restore metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { "apiVersion" : "velero.io/v1" , "kind" : "Restore" , "metadata" :{ "annotations" :{}, "name" : "restore-restic" , "namespace" : "openshift-adp" }, "spec" :{ "backupName" : "backup-restic" , "excludedResources" :[ "nodes" , "events" , "events.events.k8s.io" , "backups.velero.io" , "restores.velero.io" , "resticrepositories.velero.io" ]}} creationTimestamp: "2023-05-02T06:09:30Z" generation: 8 name: restore-restic namespace: openshift-adp resourceVersion: "1548708" uid: 2170eda6-a97f-4cf9-85e4-0f0fb0b7dade spec: backupName: backup-restic excludedResources: - nodes - events - events.events.k8s.io - backups.velero.io - restores.velero.io - resticrepositories.velero.io - csinodes.storage.k8s.io - volumeattachments.storage.k8s.io - backuprepositories.velero.io itemOperationTimeout: 1h0m0s status: completionTimestamp: "2023-05-02T06:12:35Z" phase: Completed progress: itemsRestored: 36 totalItems: 36 startTimestamp: "2023-05-02T06:09:30Z" warnings: 13 # oc get podvolumerestore -A NAMESPACE NAME NAMESPACE POD UPLOADER TYPE VOLUME STATUS TOTALBYTES BYTESDONE AGE openshift-adp restore-restic-9pmbl maria mysql-c96c5db4-sh9bt restic mysql-data Completed 107855553 107855553 12m # oc get po -n maria NAME READY STATUS RESTARTS AGE mysql-c96c5db4-sh9bt 1/1 Running 0 2m57s ]# oc get pvc -n maria mysql -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: backup.velero.io/must-include-additional-items: " true " pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" velero.io/backup-name: backup1 velero.io/volume-snapshot-name: velero-mysql-z9bcz volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com volume.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com creationTimestamp: "2023-05-02T06:09:57Z" finalizers: - kubernetes.io/pvc-protection labels: app: mysql backup.velero.io/must-include-additional-items: " true " testlabel: selectors testlabel2: foo velero.io/backup-name: backup-restic velero.io/restore-name: restore-restic velero.io/volume-snapshot-name: velero-mysql-z9bcz name: mysql namespace: maria resourceVersion: "1545457" uid: 7c30fb3f-458a-49c6-b108-77c57652aff0 spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: ocs-storagecluster-cephfs volumeMode: Filesystem volumeName: pvc-7c30fb3f-458a-49c6-b108-77c57652aff0 status: accessModes: - ReadWriteOnce capacity: storage: 2Gi phase: Bound # oc rsh -nmaria mysql-c96c5db4-sh9bt Defaulted container "mysql" out of: mysql, restore-wait (init) sh-4.4$ mysql -uroot MYSQL_DATABASE -e "show tables" ; +--------------------------+ | Tables_in_MYSQL_DATABASE | +--------------------------+ | authors | | posts | +--------------------------+ sh-4.4$ exit exit # oc get vs -A NAMESPACE NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE maria velero-mysql-qgg44 false velero-velero-mysql-qgg44-slvkm csi-vsc 2m4s

Assignee:: SHIVA SAI K (Inactive)

Reporter:: Maya Peretz

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2023/04/25 10:06 AM

Updated:: 2025/03/30 3:21 PM

Resolved:: 2023/05/02 6:59 AM

Details

Description

Attachments

Easy Agile Planning Poker

Activity

[OADP-1732] [IBM QE-Z] Verify Bug OADP-1640 - CSI backup/restore to FS backup/restore fails

Collapse comment: SHIVA SAI K (Inactive) added a comment - 2023/05/02 6:34 AM, Edited by SHIVA SAI K - 2023/05/02 6:58 AM

Expand comment: SHIVA SAI K (Inactive) added a comment - 2023/05/02 6:34 AM, Edited by SHIVA SAI K - 2023/05/02 6:58 AM

People

Dates