Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-1870 PodVolumeBackup fails on completed pod volumes in OADP-1.2
  3. OADP-4163

[RedHat QE] Verify Bug OADP-1870 - PodVolumeBackup fails on completed pod volumes in OADP-1.2

XMLWordPrintable

    • Icon: Sub-task Sub-task
    • Resolution: Done
    • Icon: Undefined Undefined
    • OADP 1.2.5
    • None
    • QE-Task
    • None
    • 2
    • False
    • Hide

      None

      Show
      None
    • False
    • Passed
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown

      Description of problem:

      When there is a completed pod that mounted volumes in a namespace covered by restic (podvolumebackup) velero backup, backup will not complete successfully.

       

      Occurs when defaultVolumesToFsBackup is `true`

       

      OADP-1075 contain info on documentation based workaround which applies when OADP-1870 is not yet resolved.

      Version-Release number of selected component (if applicable):

       

      How reproducible:

       

      Steps to Reproduce:
      1. Create a pod that runs to completion and has volumeMounts
      2. Create velero restic backup

      Actual results:

       PodVolumeBackup fails

      Expected results:

       Backup complete successfully

      Additional info:

            [OADP-4163] [RedHat QE] Verify Bug OADP-1870 - PodVolumeBackup fails on completed pod volumes in OADP-1.2

            Prasad Joshi added a comment - - edited

            Verified with OADP 1.2.5 from stage build. 

            $ oc get csv
NAME                   DISPLAY         VERSION   REPLACES               PHASE
oadp-operator.v1.2.5   OADP Operator   1.2.5     oadp-operator.v1.2.4   Succeeded

            1. Deployed application consisting a pod with status completed and has a pvc attached to it. 

            $ oc get pod -n test
NAME                        READY   STATUS      RESTARTS   AGE
data-generation-job-4pgqr   0/1     Completed   0          13m
            $ oc get pvc -n test NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE pvc-for-data   Bound    pvc-435456e0-9874-4f7b-a6bb-e01ca2db4264   1Gi        RWO            standard-csi   16m

            2. Created DPA with restic enabled. 

            $ oc get dpa -o yaml ts-dpa
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  creationTimestamp: "2024-05-28T06:26:05Z"
  generation: 2
  name: ts-dpa
  namespace: openshift-adp
  resourceVersion: "43654"
  uid: d82807b8-ac3a-4606-a3e2-e854cc69be9a
spec:
  backupLocations:
  - velero:
      config:
        insecureSkipTLSVerify: "true"
        profile: default
        region: minio
        s3ForcePathStyle: "true"
        s3Url: http://minio-minio.apps.ci-ln-g48s9jk-72292.origin-ci-int-gce.dev.rhcloud.com
      default: true
      objectStorage:
        bucket: miniobucket
        prefix: velero
      provider: aws
  configuration:
    restic:
      enable: true
    velero:
      defaultPlugins:
      - openshift
      - aws
status:
  conditions:
  - lastTransitionTime: "2024-05-28T06:29:12Z"
    message: Reconcile complete
    reason: Complete
    status: "True"
    type: Reconciled

            3. Backed up test namespace via Restic. 

            $ oc get backup -o yaml test-backup
apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    velero.io/source-cluster-k8s-gitversion: v1.27.13+048520e
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "27"
  creationTimestamp: "2024-05-28T06:32:00Z"
  generation: 6
  labels:
    velero.io/storage-location: ts-dpa-1
  name: test-backup
  namespace: openshift-adp
  resourceVersion: "44869"
  uid: b129d0d2-2871-4047-ba16-689a9ed3015e
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: true
  includedNamespaces:
  - test
  itemOperationTimeout: 1h0m0s
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
status:
  completionTimestamp: "2024-05-28T06:32:05Z"
  expiration: "2024-06-27T06:32:00Z"
  formatVersion: 1.1.0
  phase: Completed
  progress:
    itemsBackedUp: 37
    totalItems: 37
  startTimestamp: "2024-05-28T06:32:00Z"
  version: 1
  warnings: 1 
            $ velero describe backup test-backup -n openshift-adp --details  Namespaces:
    test:   resource: /pods name: /data-generation-job-4pgqr error: /backup for volume volume is skipped: pod is not in the expected status, name=data-generation-job-4pgqr, namespace=test, phase=Succeeded: pod is not running

            Verified that the backup completed successfully. Moving this to verified status. 

            Prasad Joshi added a comment - - edited Verified with OADP 1.2.5 from stage build.  $ oc get csv NAME                   DISPLAY         VERSION   REPLACES               PHASE oadp- operator .v1.2.5   OADP Operator   1.2.5     oadp- operator .v1.2.4   Succeeded 1. Deployed application consisting a pod with status completed and has a pvc attached to it.  $ oc get pod -n test NAME                        READY   STATUS      RESTARTS   AGE data-generation-job-4pgqr   0/1     Completed   0          13m $ oc get pvc -n test NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE pvc- for -data   Bound    pvc-435456e0-9874-4f7b-a6bb-e01ca2db4264   1Gi        RWO            standard-csi   16m 2. Created DPA with restic enabled.  $ oc get dpa -o yaml ts-dpa apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata:   creationTimestamp: "2024-05-28T06:26:05Z"   generation: 2   name: ts-dpa   namespace: openshift-adp   resourceVersion: "43654"   uid: d82807b8-ac3a-4606-a3e2-e854cc69be9a spec:   backupLocations:   - velero:       config:         insecureSkipTLSVerify: " true "         profile: default         region: minio         s3ForcePathStyle: " true "         s3Url: http: //minio-minio.apps.ci-ln-g48s9jk-72292.origin-ci- int -gce.dev.rhcloud.com       default : true       objectStorage:         bucket: miniobucket         prefix: velero       provider: aws   configuration:     restic:       enable: true     velero:       defaultPlugins:       - openshift       - aws status:   conditions:   - lastTransitionTime: "2024-05-28T06:29:12Z"     message: Reconcile complete     reason: Complete     status: "True"     type: Reconciled 3. Backed up test namespace via Restic.  $ oc get backup -o yaml test-backup apiVersion: velero.io/v1 kind: Backup metadata:   annotations:     velero.io/source-cluster-k8s-gitversion: v1.27.13+048520e     velero.io/source-cluster-k8s-major-version: "1"     velero.io/source-cluster-k8s-minor-version: "27"   creationTimestamp: "2024-05-28T06:32:00Z"   generation: 6   labels:     velero.io/storage-location: ts-dpa-1   name: test-backup   namespace: openshift-adp   resourceVersion: "44869"   uid: b129d0d2-2871-4047-ba16-689a9ed3015e spec:   csiSnapshotTimeout: 10m0s   defaultVolumesToFsBackup: true   includedNamespaces:   - test   itemOperationTimeout: 1h0m0s   storageLocation: ts-dpa-1   ttl: 720h0m0s status:   completionTimestamp: "2024-05-28T06:32:05Z"   expiration: "2024-06-27T06:32:00Z"   formatVersion: 1.1.0   phase: Completed   progress:     itemsBackedUp: 37     totalItems: 37   startTimestamp: "2024-05-28T06:32:00Z"   version: 1   warnings: 1 $ velero describe backup test-backup -n openshift-adp --details  Namespaces:     test:   resource: /pods name: /data-generation-job-4pgqr error: /backup for volume volume is skipped: pod is not in the expected status, name=data-generation-job-4pgqr, namespace=test, phase=Succeeded: pod is not running Verified that the backup completed successfully. Moving this to verified status. 

              rhn-support-prajoshi Prasad Joshi
              akarol@redhat.com Aziza Karol
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: