
ceph-rbd StorageClass found: ocs-storagecluster-ceph-rbd
Using the StorageClass for openshift-storage: ocs-storagecluster-ceph-rbd
[34mGenerated settings file under /output_files/api-rdr-sgar-416-c4ee-ibm-com:6443/settings.json[39m
volumesnapshotclass.snapshot.storage.k8s.io/ocs-storagecluster-cephfsplugin-snapclass annotated
volumesnapshotclass.snapshot.storage.k8s.io/ocs-storagecluster-rbdplugin-snapclass annotated
2024/06/18 03:11:04 Setting up clients
2024/06/18 03:11:04 Getting default StorageClass...
2024/06/18 03:11:04 Checking default storage class count
Run the command:
oc get sc
2024/06/18 03:11:04 Got default StorageClass ocs-storagecluster-ceph-rbd
2024/06/18 03:11:04 oc get sc
NAME                                    PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
localblock                              kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  11d
ocs-storagecluster-ceph-rbd (default)   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   11d
ocs-storagecluster-ceph-rgw             openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  11d
ocs-storagecluster-cephfs               openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   11d
openshift-storage.noobaa.io             openshift-storage.noobaa.io/obc         Delete          Immediate              false                  11d

2024/06/18 03:11:04 Using velero prefix: velero-e2e-ed0dc423-2d41-11ef-af3e-fa163ea1e99d
2024/06/18 03:11:04 Checking default storage class count
Running Suite: OADP E2E Suite - /root/testoadp/oadp-e2e-qe/e2e
==============================================================
Random Seed: [1m1718694653[0m

Will run [1m1[0m of [1m193[0m specs
[38;5;243m------------------------------[0m
[0m[1m[SynchronizedBeforeSuite] [0m
[38;5;243m/root/testoadp/oadp-e2e-qe/e2e/e2e_suite_test.go:76[0m
  2024/06/18 03:11:04 Error reading VSL credentials file: open /tmp/test-settings/aws_vsl_creds: no such file or directory
[38;5;10m[SynchronizedBeforeSuite] PASSED [0.037 seconds][0m
[38;5;243m------------------------------[0m
[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m
[38;5;243m------------------------------[0m
[0mbackup restore tests [0m[1m[tc-id:OADP-465] [datamover] Kill velero pod during backup in phase: WaitingForPluginOperations[0m
[38;5;243m/root/testoadp/oadp-e2e-qe/e2e/app_backup/datamover_restore_from_specific_backup.go:153[0m
  2024/06/18 03:11:04 Delete all downloadrequest
No download requests are found
  [1mSTEP:[0m Create DPA CR [38;5;243m@ 06/18/24 03:11:04.563[0m
  2024/06/18 03:11:04 {
    "metadata": {
      "name": "ts-dpa",
      "namespace": "openshift-adp",
      "uid": "88ce7951-9f59-43a4-8994-bb31cf623d39",
      "resourceVersion": "11730135",
      "generation": 1,
      "creationTimestamp": "2024-06-18T07:11:04Z",
      "managedFields": [
        {
          "manager": "e2e.test",
          "operation": "Update",
          "apiVersion": "oadp.openshift.io/v1alpha1",
          "time": "2024-06-18T07:11:04Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:spec": {
              ".": {},
              "f:backupLocations": {},
              "f:configuration": {
                ".": {},
                "f:nodeAgent": {
                  ".": {},
                  "f:enable": {},
                  "f:podConfig": {
                    ".": {},
                    "f:resourceAllocations": {}
                  },
                  "f:uploaderType": {}
                },
                "f:velero": {
                  ".": {},
                  "f:defaultPlugins": {}
                }
              },
              "f:podDnsConfig": {},
              "f:snapshotLocations": {}
            }
          }
        }
      ]
    },
    "spec": {
      "backupLocations": [
        {
          "velero": {
            "provider": "aws",
            "config": {
              "region": "us-east-1"
            },
            "credential": {
              "name": "cloud-credentials",
              "key": "cloud"
            },
            "objectStorage": {
              "bucket": "newocpbucket",
              "prefix": "velero-e2e-ed109ef4-2d41-11ef-af3e-fa163ea1e99d"
            },
            "default": true
          }
        }
      ],
      "snapshotLocations": [],
      "podDnsConfig": {},
      "configuration": {
        "velero": {
          "defaultPlugins": [
            "openshift",
            "aws",
            "csi"
          ]
        },
        "nodeAgent": {
          "enable": true,
          "podConfig": {
            "resourceAllocations": {}
          },
          "uploaderType": "kopia"
        }
      },
      "features": null
    },
    "status": {}
  }
Delete all the backups that remained in the phase InProgress
Deleting backup CRs in progress
Deletion of backup CRs in progress completed
Delete all the restores that remained in the phase InProgress
Deleting restore CRs in progress
Deletion of restore CRs in progress completed
  [1mSTEP:[0m Verify DPA CR setup [38;5;243m@ 06/18/24 03:11:04.592[0m
  2024/06/18 03:11:04 Waiting for velero pod to be running
  2024/06/18 03:11:09 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete'
  [1mSTEP:[0m Prepare backup resources, depending on the volumes backup type [38;5;243m@ 06/18/24 03:11:09.615[0m
Run the command:
oc get ns openshift-storage &> /dev/null && echo true || echo false
  2024/06/18 03:11:09 The 'openshift-storage' namespace exists
  2024/06/18 03:11:09 Checking default storage class count
  2024/06/18 03:11:09 Using the CSI driver: openshift-storage.rbd.csi.ceph.com
  2024/06/18 03:11:09 Snapclass 'example-snapclass' doesn't exist, creating
  2024/06/18 03:11:09 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd'
  2024/06/18 03:11:09 Checking default storage class count
Skipping creation of StorageClass
The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd
  2024/06/18 03:11:09 Checking for correct number of running NodeAgent pods...
  [1mSTEP:[0m Installing application for case mysql [38;5;243m@ 06/18/24 03:11:09.793[0m
[1;35m[WARNING]: provided hosts list is empty, only localhost is available. Note that[0m
[1;35mthe implicit localhost does not match 'all'[0m
[1;35m[WARNING]: Found variable using reserved name: namespace[0m

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [include_vars] ************************************************************
[0;32mok: [localhost][0m

TASK [Remove all the contents from the file] ***********************************
[0;33mchanged: [localhost][0m

TASK [Get cluster endpoint] ****************************************************
[0;33mchanged: [localhost][0m

TASK [Get current admin token] *************************************************
[0;33mchanged: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

TASK [Extract kubernetes minor version from cluster_info] **********************
[0;32mok: [localhost][0m

TASK [Map kubernetes minor to ocp releases] ************************************
[0;32mok: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

PLAY [Execute Task] ************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-465] ***
[0;32mok: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] ***
[0;33mchanged: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] ***
[0;33mchanged: [localhost][0m
[1;30mFAILED - RETRYING: [localhost]: Check pod status (30 retries left).[0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] ***
[0;32mok: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] ***
[0;33mchanged: [localhost][0m
[1;30mFAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left).[0m
[1;30mFAILED - RETRYING: [localhost]: Wait until service ready for connections (29 retries left).[0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] ***
[0;33mchanged: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] ***
[0;33mchanged: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] ***
[0;33mchanged: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] ***
[0;33mchanged: [localhost][0m
Pausing for 30 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] ***
[0;32mok: [localhost][0m

PLAY RECAP *********************************************************************
[0;33mlocalhost[0m                  : [0;32mok=20  [0m [0;33mchanged=10  [0m unreachable=0    failed=0    [0;36mskipped=6   [0m rescued=0    ignored=0   

  2024/06/18 03:12:07 
  2024-06-18 03:11:11,817 p=1106928 u=root n=ansible | TASK [Remove all the contents from the file] ***********************************
  2024-06-18 03:11:11,817 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:12,117 p=1106928 u=root n=ansible | TASK [Get cluster endpoint] ****************************************************
  2024-06-18 03:11:12,117 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:12,421 p=1106928 u=root n=ansible | TASK [Get current admin token] *************************************************
  2024-06-18 03:11:12,421 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:12,440 p=1106928 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:11:12,440 p=1106928 u=root n=ansible | ok: [localhost]
  2024-06-18 03:11:12,723 p=1106928 u=root n=ansible | TASK [Extract kubernetes minor version from cluster_info] **********************
  2024-06-18 03:11:12,723 p=1106928 u=root n=ansible | ok: [localhost]
  2024-06-18 03:11:12,761 p=1106928 u=root n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************
  2024-06-18 03:11:12,761 p=1106928 u=root n=ansible | ok: [localhost]
  2024-06-18 03:11:12,791 p=1106928 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:11:12,791 p=1106928 u=root n=ansible | ok: [localhost]
  2024-06-18 03:11:12,818 p=1106928 u=root n=ansible | PLAY [Execute Task] ************************************************************
  2024-06-18 03:11:13,618 p=1106928 u=root n=ansible | TASK [Gathering Facts] *********************************************************
  2024-06-18 03:11:13,619 p=1106928 u=root n=ansible | ok: [localhost]
  2024-06-18 03:11:14,533 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-465] ***
  2024-06-18 03:11:14,534 p=1106928 u=root n=ansible | ok: [localhost]
  2024-06-18 03:11:14,994 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] ***
  2024-06-18 03:11:14,994 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:16,002 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] ***
  2024-06-18 03:11:16,002 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:22,363 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] ***
  2024-06-18 03:11:22,363 p=1106928 u=root n=ansible | ok: [localhost]
  2024-06-18 03:11:22,857 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] ***
  2024-06-18 03:11:22,857 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:33,883 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] ***
  2024-06-18 03:11:33,883 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:35,102 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] ***
  2024-06-18 03:11:35,103 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:36,271 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] ***
  2024-06-18 03:11:36,271 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:36,995 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] ***
  2024-06-18 03:11:36,995 p=1106928 u=root n=ansible | changed: [localhost]
  2024-06-18 03:11:37,026 p=1106928 u=root n=ansible | Pausing for 30 seconds
  2024-06-18 03:11:37,027 p=1106928 u=root n=ansible | (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
  2024-06-18 03:12:07,028 p=1106928 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] ***
  2024-06-18 03:12:07,028 p=1106928 u=root n=ansible | ok: [localhost]
  2024-06-18 03:12:07,173 p=1106928 u=root n=ansible | PLAY RECAP *********************************************************************
  2024-06-18 03:12:07,173 p=1106928 u=root n=ansible | localhost                  : ok=20   changed=10   unreachable=0    failed=0    skipped=6    rescued=0    ignored=0   
  [1mSTEP:[0m Verify Application deployment [38;5;243m@ 06/18/24 03:12:07.208[0m
[1;35m[WARNING]: provided hosts list is empty, only localhost is available. Note that[0m
[1;35mthe implicit localhost does not match 'all'[0m
[1;35m[WARNING]: Found variable using reserved name: namespace[0m

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [include_vars] ************************************************************
[0;32mok: [localhost][0m

TASK [Remove all the contents from the file] ***********************************
[0;33mchanged: [localhost][0m

TASK [Get cluster endpoint] ****************************************************
[0;33mchanged: [localhost][0m

TASK [Get current admin token] *************************************************
[0;33mchanged: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

TASK [Extract kubernetes minor version from cluster_info] **********************
[0;32mok: [localhost][0m

TASK [Map kubernetes minor to ocp releases] ************************************
[0;32mok: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

PLAY [Execute Task] ************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] ***
[0;32mok: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] ***
[0;33mchanged: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] ***
[0;33mchanged: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] ***
[0;33mchanged: [localhost][0m

PLAY RECAP *********************************************************************
[0;33mlocalhost[0m                  : [0;32mok=14  [0m [0;33mchanged=6   [0m unreachable=0    failed=0    [0;36mskipped=12  [0m rescued=0    ignored=0   

  2024/06/18 03:12:13 
  2024-06-18 03:12:09,220 p=1107507 u=root n=ansible | TASK [Remove all the contents from the file] ***********************************
  2024-06-18 03:12:09,220 p=1107507 u=root n=ansible | changed: [localhost]
  2024-06-18 03:12:09,524 p=1107507 u=root n=ansible | TASK [Get cluster endpoint] ****************************************************
  2024-06-18 03:12:09,524 p=1107507 u=root n=ansible | changed: [localhost]
  2024-06-18 03:12:09,824 p=1107507 u=root n=ansible | TASK [Get current admin token] *************************************************
  2024-06-18 03:12:09,824 p=1107507 u=root n=ansible | changed: [localhost]
  2024-06-18 03:12:09,844 p=1107507 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:12:09,844 p=1107507 u=root n=ansible | ok: [localhost]
  2024-06-18 03:12:10,135 p=1107507 u=root n=ansible | TASK [Extract kubernetes minor version from cluster_info] **********************
  2024-06-18 03:12:10,135 p=1107507 u=root n=ansible | ok: [localhost]
  2024-06-18 03:12:10,173 p=1107507 u=root n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************
  2024-06-18 03:12:10,173 p=1107507 u=root n=ansible | ok: [localhost]
  2024-06-18 03:12:10,203 p=1107507 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:12:10,203 p=1107507 u=root n=ansible | ok: [localhost]
  2024-06-18 03:12:10,229 p=1107507 u=root n=ansible | PLAY [Execute Task] ************************************************************
  2024-06-18 03:12:11,076 p=1107507 u=root n=ansible | TASK [Gathering Facts] *********************************************************
  2024-06-18 03:12:11,076 p=1107507 u=root n=ansible | ok: [localhost]
  2024-06-18 03:12:12,335 p=1107507 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] ***
  2024-06-18 03:12:12,335 p=1107507 u=root n=ansible | ok: [localhost]
  2024-06-18 03:12:12,695 p=1107507 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] ***
  2024-06-18 03:12:12,695 p=1107507 u=root n=ansible | changed: [localhost]
  2024-06-18 03:12:13,122 p=1107507 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] ***
  2024-06-18 03:12:13,122 p=1107507 u=root n=ansible | changed: [localhost]
  2024-06-18 03:12:13,825 p=1107507 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] ***
  2024-06-18 03:12:13,825 p=1107507 u=root n=ansible | changed: [localhost]
  2024-06-18 03:12:13,857 p=1107507 u=root n=ansible | PLAY RECAP *********************************************************************
  2024-06-18 03:12:13,857 p=1107507 u=root n=ansible | localhost                  : ok=14   changed=6    unreachable=0    failed=0    skipped=12   rescued=0    ignored=0   
  2024/06/18 03:12:13 Getting the velero pod before deleting it
  [1mSTEP:[0m Creating backup mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d [38;5;243m@ 06/18/24 03:12:13.898[0m
  2024/06/18 03:12:13 Wait until backup mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d is in WaitingForPluginOperations Phase
  2024/06/18 03:12:13 
  2024/06/18 03:12:16 InProgress
  2024/06/18 03:12:19 InProgress
  2024/06/18 03:12:22 InProgress
  2024/06/18 03:12:25 InProgress
  2024/06/18 03:12:28 WaitingForPluginOperations
  2024/06/18 03:12:28 Deleting the velero pod just after backup is in WaitingForPluginOperations phase
  2024/06/18 03:12:28 Check if new velero pod is deployed after deleting old pod
  2024/06/18 03:12:28 Checking Pod name velero-76dd8f8d77-psxdg
  2024/06/18 03:12:28 The Pod name velero-76dd8f8d77-psxdg still exists
  2024/06/18 03:12:31 Checking Pod name velero-76dd8f8d77-srrmt
  2024/06/18 03:12:31 The Pod name velero-76dd8f8d77-psxdg no longer exists
  2024/06/18 03:12:31 pod: velero-76dd8f8d77-srrmt is not yet running with status: {Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-06-18 03:12:31 -0400 EDT  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-06-18 03:12:29 -0400 EDT ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-06-18 03:12:29 -0400 EDT ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-06-18 03:12:29 -0400 EDT  }]    10.20.188.37 10.129.3.49 [{10.129.3.49}] 2024-06-18 03:12:29 -0400 EDT [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-06-18 03:12:30 -0400 EDT,FinishedAt:2024-06-18 03:12:30 -0400 EDT,ContainerID:cri-o://ffcebd4cfda7eee11e7d6786e23d44ab9544268abc526f7abdbc721517e348a0,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:d89cc887cc9a7832a7f35a4a7dc55e83f1f050391db5fbd2bf76cdb6f24a8de3 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:c79c34b7e2aeaa7b0e00101e724271d325e2097a8e3ef0c075a4ee4238819afd cri-o://ffcebd4cfda7eee11e7d6786e23d44ab9544268abc526f7abdbc721517e348a0 0xc0014e6339} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-06-18 03:12:31 -0400 EDT,FinishedAt:2024-06-18 03:12:31 -0400 EDT,ContainerID:cri-o://50ac52e073b0a1b79ed3ffd41d049c0bd3ae37a263b8076e834cfaa89b0de9e3,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:f3bea8dbe43e99a4d44ab8ca13a257f19fb0da6f25238cd41268cddd04d9e9fa registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:72dd6b3c3c335cd22c51f2e38e422ea9394b71ed328d377da2eeb76332cca952 cri-o://50ac52e073b0a1b79ed3ffd41d049c0bd3ae37a263b8076e834cfaa89b0de9e3 0xc0014e6349}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:19a083030bdfdf56de52d4a9de4c95d4216dcfe863a395304ef5681d9c720ebe   0xc0014e635f}] Burstable []}
  2024/06/18 03:12:37 Wait until backup mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d is done
  backup phase: WaitingForPluginOperations
  DataUpload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-h8krk phase: InProgress
DataUpload Name: mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-h8krk and status: InProgress   2024/06/18 03:12:37 {
    "kind": "DataUpload",
    "apiVersion": "velero.io/v2alpha1",
    "metadata": {
      "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-h8krk",
      "generateName": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-",
      "namespace": "openshift-adp",
      "uid": "c2e76b41-4afb-49a2-9c45-64a684ff9b79",
      "resourceVersion": "11731500",
      "generation": 4,
      "creationTimestamp": "2024-06-18T07:12:20Z",
      "labels": {
        "velero.io/accepted-by": "worker-1",
        "velero.io/async-operation-id": "du-a3fba251-00bd-4e59-bb6f-b324009dd818.84265234-fb2b-46644a04d",
        "velero.io/backup-name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
        "velero.io/backup-uid": "a3fba251-00bd-4e59-bb6f-b324009dd818",
        "velero.io/pvc-uid": "84265234-fb2b-4664-bf3e-e7fbd63f19fa"
      },
      "ownerReferences": [
        {
          "apiVersion": "velero.io/v1",
          "kind": "Backup",
          "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
          "uid": "a3fba251-00bd-4e59-bb6f-b324009dd818",
          "controller": true
        }
      ],
      "finalizers": [
        "velero.io/data-upload-download-finalizer"
      ],
      "managedFields": [
        {
          "manager": "velero",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:12:20Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:generateName": {},
              "f:labels": {
                ".": {},
                "f:velero.io/async-operation-id": {},
                "f:velero.io/backup-name": {},
                "f:velero.io/backup-uid": {},
                "f:velero.io/pvc-uid": {}
              },
              "f:ownerReferences": {
                ".": {},
                "k:{\"uid\":\"a3fba251-00bd-4e59-bb6f-b324009dd818\"}": {}
              }
            },
            "f:spec": {
              ".": {},
              "f:backupStorageLocation": {},
              "f:csiSnapshot": {
                ".": {},
                "f:snapshotClass": {},
                "f:storageClass": {},
                "f:volumeSnapshot": {}
              },
              "f:operationTimeout": {},
              "f:snapshotType": {},
              "f:sourceNamespace": {},
              "f:sourcePVC": {}
            },
            "f:status": {
              ".": {},
              "f:progress": {}
            }
          }
        },
        {
          "manager": "node-agent-server",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:12:33Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:finalizers": {
                ".": {},
                "v:\"velero.io/data-upload-download-finalizer\"": {}
              },
              "f:labels": {
                "f:velero.io/accepted-by": {}
              }
            },
            "f:status": {
              "f:node": {},
              "f:phase": {},
              "f:startTimestamp": {}
            }
          }
        }
      ]
    },
    "spec": {
      "snapshotType": "CSI",
      "csiSnapshot": {
        "volumeSnapshot": "velero-mysql-data-vl2tw",
        "storageClass": "ocs-storagecluster-ceph-rbd",
        "snapshotClass": "example-snapclass"
      },
      "sourcePVC": "mysql-data",
      "backupStorageLocation": "ts-dpa-1",
      "sourceNamespace": "test-oadp-465",
      "operationTimeout": "10m0s"
    },
    "status": {
      "phase": "InProgress",
      "startTimestamp": "2024-06-18T07:12:33Z",
      "progress": {},
      "node": "worker-2"
    }
  }
  DataUpload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-m7chv phase: InProgress
DataUpload Name: mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-m7chv and status: InProgress   2024/06/18 03:12:37 {
    "kind": "DataUpload",
    "apiVersion": "velero.io/v2alpha1",
    "metadata": {
      "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-m7chv",
      "generateName": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-",
      "namespace": "openshift-adp",
      "uid": "6d027251-f5bd-4c99-8552-78091d71f202",
      "resourceVersion": "11731517",
      "generation": 4,
      "creationTimestamp": "2024-06-18T07:12:25Z",
      "labels": {
        "velero.io/accepted-by": "worker-1",
        "velero.io/async-operation-id": "du-a3fba251-00bd-4e59-bb6f-b324009dd818.9baae976-c193-4879b6de5",
        "velero.io/backup-name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
        "velero.io/backup-uid": "a3fba251-00bd-4e59-bb6f-b324009dd818",
        "velero.io/pvc-uid": "9baae976-c193-4873-aa3c-a57bd53afa16"
      },
      "ownerReferences": [
        {
          "apiVersion": "velero.io/v1",
          "kind": "Backup",
          "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
          "uid": "a3fba251-00bd-4e59-bb6f-b324009dd818",
          "controller": true
        }
      ],
      "finalizers": [
        "velero.io/data-upload-download-finalizer"
      ],
      "managedFields": [
        {
          "manager": "velero",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:12:25Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:generateName": {},
              "f:labels": {
                ".": {},
                "f:velero.io/async-operation-id": {},
                "f:velero.io/backup-name": {},
                "f:velero.io/backup-uid": {},
                "f:velero.io/pvc-uid": {}
              },
              "f:ownerReferences": {
                ".": {},
                "k:{\"uid\":\"a3fba251-00bd-4e59-bb6f-b324009dd818\"}": {}
              }
            },
            "f:spec": {
              ".": {},
              "f:backupStorageLocation": {},
              "f:csiSnapshot": {
                ".": {},
                "f:snapshotClass": {},
                "f:storageClass": {},
                "f:volumeSnapshot": {}
              },
              "f:operationTimeout": {},
              "f:snapshotType": {},
              "f:sourceNamespace": {},
              "f:sourcePVC": {}
            },
            "f:status": {
              ".": {},
              "f:progress": {}
            }
          }
        },
        {
          "manager": "node-agent-server",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:12:34Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:finalizers": {
                ".": {},
                "v:\"velero.io/data-upload-download-finalizer\"": {}
              },
              "f:labels": {
                "f:velero.io/accepted-by": {}
              }
            },
            "f:status": {
              "f:node": {},
              "f:phase": {},
              "f:startTimestamp": {}
            }
          }
        }
      ]
    },
    "spec": {
      "snapshotType": "CSI",
      "csiSnapshot": {
        "volumeSnapshot": "velero-mysql-data1-ktfzj",
        "storageClass": "ocs-storagecluster-ceph-rbd",
        "snapshotClass": "example-snapclass"
      },
      "sourcePVC": "mysql-data1",
      "backupStorageLocation": "ts-dpa-1",
      "sourceNamespace": "test-oadp-465",
      "operationTimeout": "10m0s"
    },
    "status": {
      "phase": "InProgress",
      "startTimestamp": "2024-06-18T07:12:34Z",
      "progress": {},
      "node": "worker-0"
    }
  }
  backup phase: Completed
  [1mSTEP:[0m Verify backup mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d has completed successfully [38;5;243m@ 06/18/24 03:12:57.055[0m
  2024/06/18 03:12:57 Backup for case mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d succeeded
  [1mSTEP:[0m Delete the appplication resources mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d [38;5;243m@ 06/18/24 03:12:57.059[0m
  [1mSTEP:[0m Cleanup Application for case mysql [38;5;243m@ 06/18/24 03:12:57.06[0m
[1;35m[WARNING]: provided hosts list is empty, only localhost is available. Note that[0m
[1;35mthe implicit localhost does not match 'all'[0m
[1;35m[WARNING]: Found variable using reserved name: namespace[0m

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [include_vars] ************************************************************
[0;32mok: [localhost][0m

TASK [Remove all the contents from the file] ***********************************
[0;33mchanged: [localhost][0m

TASK [Get cluster endpoint] ****************************************************
[0;33mchanged: [localhost][0m

TASK [Get current admin token] *************************************************
[0;33mchanged: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

TASK [Extract kubernetes minor version from cluster_info] **********************
[0;32mok: [localhost][0m

TASK [Map kubernetes minor to ocp releases] ************************************
[0;32mok: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

PLAY [Execute Task] ************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-465] ***
[0;33mchanged: [localhost][0m

PLAY RECAP *********************************************************************
[0;33mlocalhost[0m                  : [0;32mok=11  [0m [0;33mchanged=4   [0m unreachable=0    failed=0    [0;36mskipped=15  [0m rescued=0    ignored=0   

  2024/06/18 03:13:17 
  2024-06-18 03:12:59,094 p=1107881 u=root n=ansible | TASK [Remove all the contents from the file] ***********************************
  2024-06-18 03:12:59,094 p=1107881 u=root n=ansible | changed: [localhost]
  2024-06-18 03:12:59,397 p=1107881 u=root n=ansible | TASK [Get cluster endpoint] ****************************************************
  2024-06-18 03:12:59,398 p=1107881 u=root n=ansible | changed: [localhost]
  2024-06-18 03:12:59,698 p=1107881 u=root n=ansible | TASK [Get current admin token] *************************************************
  2024-06-18 03:12:59,698 p=1107881 u=root n=ansible | changed: [localhost]
  2024-06-18 03:12:59,718 p=1107881 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:12:59,718 p=1107881 u=root n=ansible | ok: [localhost]
  2024-06-18 03:13:00,002 p=1107881 u=root n=ansible | TASK [Extract kubernetes minor version from cluster_info] **********************
  2024-06-18 03:13:00,002 p=1107881 u=root n=ansible | ok: [localhost]
  2024-06-18 03:13:00,040 p=1107881 u=root n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************
  2024-06-18 03:13:00,040 p=1107881 u=root n=ansible | ok: [localhost]
  2024-06-18 03:13:00,071 p=1107881 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:13:00,071 p=1107881 u=root n=ansible | ok: [localhost]
  2024-06-18 03:13:00,097 p=1107881 u=root n=ansible | PLAY [Execute Task] ************************************************************
  2024-06-18 03:13:00,929 p=1107881 u=root n=ansible | TASK [Gathering Facts] *********************************************************
  2024-06-18 03:13:00,930 p=1107881 u=root n=ansible | ok: [localhost]
  2024-06-18 03:13:16,850 p=1107881 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-465] ***
  2024-06-18 03:13:16,850 p=1107881 u=root n=ansible | changed: [localhost]
  2024-06-18 03:13:17,312 p=1107881 u=root n=ansible | PLAY RECAP *********************************************************************
  2024-06-18 03:13:17,312 p=1107881 u=root n=ansible | localhost                  : ok=11   changed=4    unreachable=0    failed=0    skipped=15   rescued=0    ignored=0   
  [1mSTEP:[0m Create restore mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d from backup mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d [38;5;243m@ 06/18/24 03:13:17.347[0m
  2024/06/18 03:13:17 Wait until restore mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d is done
  restore phase: WaitingForPluginOperations
  DataDownload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-9q279 phase: InProgress
DataDownload Name: mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-9q279 and status: InProgress   2024/06/18 03:13:37 {
    "kind": "DataDownload",
    "apiVersion": "velero.io/v2alpha1",
    "metadata": {
      "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-9q279",
      "generateName": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-",
      "namespace": "openshift-adp",
      "uid": "9e3131e1-fbaa-414a-b654-2bef69833324",
      "resourceVersion": "11732495",
      "generation": 5,
      "creationTimestamp": "2024-06-18T07:13:21Z",
      "labels": {
        "velero.io/accepted-by": "worker-2",
        "velero.io/async-operation-id": "dd-e1828940-7f1e-4d4d-9d07-4a7acc646841.84265234-fb2b-466ee62d2",
        "velero.io/restore-name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
        "velero.io/restore-uid": "e1828940-7f1e-4d4d-9d07-4a7acc646841"
      },
      "ownerReferences": [
        {
          "apiVersion": "velero.io/v1",
          "kind": "Restore",
          "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
          "uid": "e1828940-7f1e-4d4d-9d07-4a7acc646841",
          "controller": true
        }
      ],
      "finalizers": [
        "velero.io/data-upload-download-finalizer"
      ],
      "managedFields": [
        {
          "manager": "velero",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:13:21Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:generateName": {},
              "f:labels": {
                ".": {},
                "f:velero.io/async-operation-id": {},
                "f:velero.io/restore-name": {},
                "f:velero.io/restore-uid": {}
              },
              "f:ownerReferences": {
                ".": {},
                "k:{\"uid\":\"e1828940-7f1e-4d4d-9d07-4a7acc646841\"}": {}
              }
            },
            "f:spec": {
              ".": {},
              "f:backupStorageLocation": {},
              "f:operationTimeout": {},
              "f:snapshotID": {},
              "f:sourceNamespace": {},
              "f:targetVolume": {
                ".": {},
                "f:namespace": {},
                "f:pv": {},
                "f:pvc": {}
              }
            },
            "f:status": {
              ".": {},
              "f:progress": {}
            }
          }
        },
        {
          "manager": "node-agent-server",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:13:36Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:finalizers": {
                ".": {},
                "v:\"velero.io/data-upload-download-finalizer\"": {}
              },
              "f:labels": {
                "f:velero.io/accepted-by": {}
              }
            },
            "f:status": {
              "f:node": {},
              "f:phase": {},
              "f:progress": {
                "f:bytesDone": {},
                "f:totalBytes": {}
              },
              "f:startTimestamp": {}
            }
          }
        }
      ]
    },
    "spec": {
      "targetVolume": {
        "pvc": "mysql-data",
        "pv": "",
        "namespace": "test-oadp-465"
      },
      "backupStorageLocation": "ts-dpa-1",
      "snapshotID": "030c61be76c4317cd49a7008e755495b",
      "sourceNamespace": "test-oadp-465",
      "operationTimeout": "10m0s"
    },
    "status": {
      "phase": "InProgress",
      "startTimestamp": "2024-06-18T07:13:34Z",
      "progress": {
        "totalBytes": 107854713,
        "bytesDone": 107854713
      },
      "node": "worker-2"
    }
  }
  DataDownload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46 phase: Prepared
DataDownload Name: mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46 and status: Prepared   2024/06/18 03:13:37 {
    "kind": "DataDownload",
    "apiVersion": "velero.io/v2alpha1",
    "metadata": {
      "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46",
      "generateName": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-",
      "namespace": "openshift-adp",
      "uid": "636e70ff-3e06-41e3-ae92-ccae3980058e",
      "resourceVersion": "11732488",
      "generation": 3,
      "creationTimestamp": "2024-06-18T07:13:21Z",
      "labels": {
        "velero.io/accepted-by": "worker-0",
        "velero.io/async-operation-id": "dd-e1828940-7f1e-4d4d-9d07-4a7acc646841.9baae976-c193-487d6fcf8",
        "velero.io/restore-name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
        "velero.io/restore-uid": "e1828940-7f1e-4d4d-9d07-4a7acc646841"
      },
      "ownerReferences": [
        {
          "apiVersion": "velero.io/v1",
          "kind": "Restore",
          "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
          "uid": "e1828940-7f1e-4d4d-9d07-4a7acc646841",
          "controller": true
        }
      ],
      "finalizers": [
        "velero.io/data-upload-download-finalizer"
      ],
      "managedFields": [
        {
          "manager": "velero",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:13:21Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:generateName": {},
              "f:labels": {
                ".": {},
                "f:velero.io/async-operation-id": {},
                "f:velero.io/restore-name": {},
                "f:velero.io/restore-uid": {}
              },
              "f:ownerReferences": {
                ".": {},
                "k:{\"uid\":\"e1828940-7f1e-4d4d-9d07-4a7acc646841\"}": {}
              }
            },
            "f:spec": {
              ".": {},
              "f:backupStorageLocation": {},
              "f:operationTimeout": {},
              "f:snapshotID": {},
              "f:sourceNamespace": {},
              "f:targetVolume": {
                ".": {},
                "f:namespace": {},
                "f:pv": {},
                "f:pvc": {}
              }
            },
            "f:status": {
              ".": {},
              "f:progress": {}
            }
          }
        },
        {
          "manager": "node-agent-server",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:13:35Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:finalizers": {
                ".": {},
                "v:\"velero.io/data-upload-download-finalizer\"": {}
              },
              "f:labels": {
                "f:velero.io/accepted-by": {}
              }
            },
            "f:status": {
              "f:node": {},
              "f:phase": {}
            }
          }
        }
      ]
    },
    "spec": {
      "targetVolume": {
        "pvc": "mysql-data1",
        "pv": "",
        "namespace": "test-oadp-465"
      },
      "backupStorageLocation": "ts-dpa-1",
      "snapshotID": "ca401223d3cb69e8712880a08b84f4ee",
      "sourceNamespace": "test-oadp-465",
      "operationTimeout": "10m0s"
    },
    "status": {
      "phase": "Prepared",
      "progress": {},
      "node": "worker-2"
    }
  }
  restore phase: WaitingForPluginOperations
  DataDownload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-9q279 phase: Completed
  DataDownload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46 phase: InProgress
DataDownload Name: mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46 and status: InProgress   2024/06/18 03:13:57 {
    "kind": "DataDownload",
    "apiVersion": "velero.io/v2alpha1",
    "metadata": {
      "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46",
      "generateName": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-",
      "namespace": "openshift-adp",
      "uid": "636e70ff-3e06-41e3-ae92-ccae3980058e",
      "resourceVersion": "11732624",
      "generation": 4,
      "creationTimestamp": "2024-06-18T07:13:21Z",
      "labels": {
        "velero.io/accepted-by": "worker-0",
        "velero.io/async-operation-id": "dd-e1828940-7f1e-4d4d-9d07-4a7acc646841.9baae976-c193-487d6fcf8",
        "velero.io/restore-name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
        "velero.io/restore-uid": "e1828940-7f1e-4d4d-9d07-4a7acc646841"
      },
      "ownerReferences": [
        {
          "apiVersion": "velero.io/v1",
          "kind": "Restore",
          "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
          "uid": "e1828940-7f1e-4d4d-9d07-4a7acc646841",
          "controller": true
        }
      ],
      "finalizers": [
        "velero.io/data-upload-download-finalizer"
      ],
      "managedFields": [
        {
          "manager": "velero",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:13:21Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:generateName": {},
              "f:labels": {
                ".": {},
                "f:velero.io/async-operation-id": {},
                "f:velero.io/restore-name": {},
                "f:velero.io/restore-uid": {}
              },
              "f:ownerReferences": {
                ".": {},
                "k:{\"uid\":\"e1828940-7f1e-4d4d-9d07-4a7acc646841\"}": {}
              }
            },
            "f:spec": {
              ".": {},
              "f:backupStorageLocation": {},
              "f:operationTimeout": {},
              "f:snapshotID": {},
              "f:sourceNamespace": {},
              "f:targetVolume": {
                ".": {},
                "f:namespace": {},
                "f:pv": {},
                "f:pvc": {}
              }
            },
            "f:status": {
              ".": {},
              "f:progress": {}
            }
          }
        },
        {
          "manager": "node-agent-server",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:13:45Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:finalizers": {
                ".": {},
                "v:\"velero.io/data-upload-download-finalizer\"": {}
              },
              "f:labels": {
                "f:velero.io/accepted-by": {}
              }
            },
            "f:status": {
              "f:node": {},
              "f:phase": {},
              "f:startTimestamp": {}
            }
          }
        }
      ]
    },
    "spec": {
      "targetVolume": {
        "pvc": "mysql-data1",
        "pv": "",
        "namespace": "test-oadp-465"
      },
      "backupStorageLocation": "ts-dpa-1",
      "snapshotID": "ca401223d3cb69e8712880a08b84f4ee",
      "sourceNamespace": "test-oadp-465",
      "operationTimeout": "10m0s"
    },
    "status": {
      "phase": "InProgress",
      "startTimestamp": "2024-06-18T07:13:45Z",
      "progress": {},
      "node": "worker-2"
    }
  }
  restore phase: WaitingForPluginOperations
  DataDownload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-9q279 phase: Completed
  DataDownload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46 phase: InProgress
DataDownload Name: mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46 and status: InProgress   2024/06/18 03:14:17 {
    "kind": "DataDownload",
    "apiVersion": "velero.io/v2alpha1",
    "metadata": {
      "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46",
      "generateName": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-",
      "namespace": "openshift-adp",
      "uid": "636e70ff-3e06-41e3-ae92-ccae3980058e",
      "resourceVersion": "11732966",
      "generation": 5,
      "creationTimestamp": "2024-06-18T07:13:21Z",
      "labels": {
        "velero.io/accepted-by": "worker-0",
        "velero.io/async-operation-id": "dd-e1828940-7f1e-4d4d-9d07-4a7acc646841.9baae976-c193-487d6fcf8",
        "velero.io/restore-name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
        "velero.io/restore-uid": "e1828940-7f1e-4d4d-9d07-4a7acc646841"
      },
      "ownerReferences": [
        {
          "apiVersion": "velero.io/v1",
          "kind": "Restore",
          "name": "mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d",
          "uid": "e1828940-7f1e-4d4d-9d07-4a7acc646841",
          "controller": true
        }
      ],
      "finalizers": [
        "velero.io/data-upload-download-finalizer"
      ],
      "managedFields": [
        {
          "manager": "velero",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:13:21Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:generateName": {},
              "f:labels": {
                ".": {},
                "f:velero.io/async-operation-id": {},
                "f:velero.io/restore-name": {},
                "f:velero.io/restore-uid": {}
              },
              "f:ownerReferences": {
                ".": {},
                "k:{\"uid\":\"e1828940-7f1e-4d4d-9d07-4a7acc646841\"}": {}
              }
            },
            "f:spec": {
              ".": {},
              "f:backupStorageLocation": {},
              "f:operationTimeout": {},
              "f:snapshotID": {},
              "f:sourceNamespace": {},
              "f:targetVolume": {
                ".": {},
                "f:namespace": {},
                "f:pv": {},
                "f:pvc": {}
              }
            },
            "f:status": {
              ".": {},
              "f:progress": {}
            }
          }
        },
        {
          "manager": "node-agent-server",
          "operation": "Update",
          "apiVersion": "velero.io/v2alpha1",
          "time": "2024-06-18T07:14:14Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:finalizers": {
                ".": {},
                "v:\"velero.io/data-upload-download-finalizer\"": {}
              },
              "f:labels": {
                "f:velero.io/accepted-by": {}
              }
            },
            "f:status": {
              "f:node": {},
              "f:phase": {},
              "f:progress": {
                "f:bytesDone": {},
                "f:totalBytes": {}
              },
              "f:startTimestamp": {}
            }
          }
        }
      ]
    },
    "spec": {
      "targetVolume": {
        "pvc": "mysql-data1",
        "pv": "",
        "namespace": "test-oadp-465"
      },
      "backupStorageLocation": "ts-dpa-1",
      "snapshotID": "ca401223d3cb69e8712880a08b84f4ee",
      "sourceNamespace": "test-oadp-465",
      "operationTimeout": "10m0s"
    },
    "status": {
      "phase": "InProgress",
      "startTimestamp": "2024-06-18T07:13:45Z",
      "progress": {
        "totalBytes": 104857640,
        "bytesDone": 104857640
      },
      "node": "worker-2"
    }
  }
  restore phase: Finalizing
  DataDownload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-9q279 phase: Completed
  DataDownload mysql-ed16fc6e-2d41-11ef-af3e-fa163ea1e99d-t5d46 phase: Completed
  restore phase: Completed
  [1mSTEP:[0m Validate the application after restore [38;5;243m@ 06/18/24 03:14:57.584[0m
  [1mSTEP:[0m Verify Application deployment for case mysql [38;5;243m@ 06/18/24 03:14:57.584[0m
[1;35m[WARNING]: provided hosts list is empty, only localhost is available. Note that[0m
[1;35mthe implicit localhost does not match 'all'[0m
[1;35m[WARNING]: Found variable using reserved name: namespace[0m

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [include_vars] ************************************************************
[0;32mok: [localhost][0m

TASK [Remove all the contents from the file] ***********************************
[0;33mchanged: [localhost][0m

TASK [Get cluster endpoint] ****************************************************
[0;33mchanged: [localhost][0m

TASK [Get current admin token] *************************************************
[0;33mchanged: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

TASK [Extract kubernetes minor version from cluster_info] **********************
[0;32mok: [localhost][0m

TASK [Map kubernetes minor to ocp releases] ************************************
[0;32mok: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

PLAY [Execute Task] ************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] ***
[0;32mok: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] ***
[0;33mchanged: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] ***
[0;33mchanged: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] ***
[0;33mchanged: [localhost][0m

PLAY RECAP *********************************************************************
[0;33mlocalhost[0m                  : [0;32mok=14  [0m [0;33mchanged=6   [0m unreachable=0    failed=0    [0;36mskipped=12  [0m rescued=0    ignored=0   

  2024/06/18 03:15:10 
  2024-06-18 03:14:59,623 p=1108152 u=root n=ansible | TASK [Remove all the contents from the file] ***********************************
  2024-06-18 03:14:59,624 p=1108152 u=root n=ansible | changed: [localhost]
  2024-06-18 03:14:59,931 p=1108152 u=root n=ansible | TASK [Get cluster endpoint] ****************************************************
  2024-06-18 03:14:59,931 p=1108152 u=root n=ansible | changed: [localhost]
  2024-06-18 03:15:00,240 p=1108152 u=root n=ansible | TASK [Get current admin token] *************************************************
  2024-06-18 03:15:00,240 p=1108152 u=root n=ansible | changed: [localhost]
  2024-06-18 03:15:00,260 p=1108152 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:15:00,260 p=1108152 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:00,547 p=1108152 u=root n=ansible | TASK [Extract kubernetes minor version from cluster_info] **********************
  2024-06-18 03:15:00,547 p=1108152 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:00,587 p=1108152 u=root n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************
  2024-06-18 03:15:00,587 p=1108152 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:00,617 p=1108152 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:15:00,618 p=1108152 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:00,644 p=1108152 u=root n=ansible | PLAY [Execute Task] ************************************************************
  2024-06-18 03:15:01,514 p=1108152 u=root n=ansible | TASK [Gathering Facts] *********************************************************
  2024-06-18 03:15:01,514 p=1108152 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:02,812 p=1108152 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] ***
  2024-06-18 03:15:02,812 p=1108152 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:03,179 p=1108152 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] ***
  2024-06-18 03:15:03,180 p=1108152 u=root n=ansible | changed: [localhost]
  2024-06-18 03:15:03,602 p=1108152 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] ***
  2024-06-18 03:15:03,602 p=1108152 u=root n=ansible | changed: [localhost]
  2024-06-18 03:15:10,732 p=1108152 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] ***
  2024-06-18 03:15:10,732 p=1108152 u=root n=ansible | changed: [localhost]
  2024-06-18 03:15:10,764 p=1108152 u=root n=ansible | PLAY RECAP *********************************************************************
  2024-06-18 03:15:10,764 p=1108152 u=root n=ansible | localhost                  : ok=14   changed=6    unreachable=0    failed=0    skipped=12   rescued=0    ignored=0   
  2024/06/18 03:15:10 Using Must-gather image: brew.registry.redhat.io/rh-osbs/oadp-oadp-mustgather-rhel9:1.4.0-5
  2024/06/18 03:15:10 Cleaning resources
  2024/06/18 03:15:10 Delete secret cloud-credentials
  2024/06/18 03:15:10 Cleaning app
[1;35m[WARNING]: provided hosts list is empty, only localhost is available. Note that[0m
[1;35mthe implicit localhost does not match 'all'[0m
[1;35m[WARNING]: Found variable using reserved name: namespace[0m

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [include_vars] ************************************************************
[0;32mok: [localhost][0m

TASK [Remove all the contents from the file] ***********************************
[0;33mchanged: [localhost][0m

TASK [Get cluster endpoint] ****************************************************
[0;33mchanged: [localhost][0m

TASK [Get current admin token] *************************************************
[0;33mchanged: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

TASK [Extract kubernetes minor version from cluster_info] **********************
[0;32mok: [localhost][0m

TASK [Map kubernetes minor to ocp releases] ************************************
[0;32mok: [localhost][0m

TASK [set_fact] ****************************************************************
[0;32mok: [localhost][0m

PLAY [Execute Task] ************************************************************

TASK [Gathering Facts] *********************************************************
[0;32mok: [localhost][0m

TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-465] ***
[0;33mchanged: [localhost][0m

PLAY RECAP *********************************************************************
[0;33mlocalhost[0m                  : [0;32mok=11  [0m [0;33mchanged=4   [0m unreachable=0    failed=0    [0;36mskipped=15  [0m rescued=0    ignored=0   

  2024/06/18 03:15:31 
  2024-06-18 03:15:12,870 p=1108520 u=root n=ansible | TASK [Remove all the contents from the file] ***********************************
  2024-06-18 03:15:12,870 p=1108520 u=root n=ansible | changed: [localhost]
  2024-06-18 03:15:13,172 p=1108520 u=root n=ansible | TASK [Get cluster endpoint] ****************************************************
  2024-06-18 03:15:13,172 p=1108520 u=root n=ansible | changed: [localhost]
  2024-06-18 03:15:13,472 p=1108520 u=root n=ansible | TASK [Get current admin token] *************************************************
  2024-06-18 03:15:13,472 p=1108520 u=root n=ansible | changed: [localhost]
  2024-06-18 03:15:13,491 p=1108520 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:15:13,491 p=1108520 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:13,777 p=1108520 u=root n=ansible | TASK [Extract kubernetes minor version from cluster_info] **********************
  2024-06-18 03:15:13,777 p=1108520 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:13,816 p=1108520 u=root n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************
  2024-06-18 03:15:13,816 p=1108520 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:13,846 p=1108520 u=root n=ansible | TASK [set_fact] ****************************************************************
  2024-06-18 03:15:13,847 p=1108520 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:13,873 p=1108520 u=root n=ansible | PLAY [Execute Task] ************************************************************
  2024-06-18 03:15:14,690 p=1108520 u=root n=ansible | TASK [Gathering Facts] *********************************************************
  2024-06-18 03:15:14,691 p=1108520 u=root n=ansible | ok: [localhost]
  2024-06-18 03:15:30,616 p=1108520 u=root n=ansible | TASK [/root/testoadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-465] ***
  2024-06-18 03:15:30,616 p=1108520 u=root n=ansible | changed: [localhost]
  2024-06-18 03:15:31,080 p=1108520 u=root n=ansible | PLAY RECAP *********************************************************************
  2024-06-18 03:15:31,080 p=1108520 u=root n=ansible | localhost                  : ok=11   changed=4    unreachable=0    failed=0    skipped=15   rescued=0    ignored=0   
  2024/06/18 03:15:31 Cleaning setup resources for the backup
  2024/06/18 03:15:31 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd'
  2024/06/18 03:15:31 Checking default storage class count
Skipping creation of StorageClass
The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd
  2024/06/18 03:15:31 Deleting VolumeSnapshotClass 'example-snapclass'
[38;5;10m• [266.604 seconds][0m
[38;5;243m------------------------------[0m
[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m
[38;5;243m------------------------------[0m
[0m[1m[SynchronizedAfterSuite] [0m
[38;5;243m/root/testoadp/oadp-e2e-qe/e2e/e2e_suite_test.go:175[0m
  2024/06/18 03:15:31 Deleting Velero CR
[38;5;10m[SynchronizedAfterSuite] PASSED [0.010 seconds][0m
[38;5;243m------------------------------[0m
[0m[1m[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report[0m
[38;5;243mautogenerated by Ginkgo[0m
[38;5;10m[ReportAfterSuite] PASSED [0.010 seconds][0m
[38;5;243m------------------------------[0m

[38;5;10m[1mRan 1 of 193 Specs in 266.657 seconds[0m
[38;5;10m[1mSUCCESS![0m -- [38;5;10m[1m1 Passed[0m | [38;5;9m[1m0 Failed[0m | [38;5;11m[1m0 Pending[0m | [38;5;14m[1m192 Skipped[0m
PASS

Ginkgo ran 1 suite in 4m37.815380207s
Test Suite Passed