ceph-rbd StorageClass found: ocs-storagecluster-ceph-rbd Using the StorageClass for openshift-storage: ocs-storagecluster-ceph-rbd Generated settings file under /output_files/api-rdr-ani-oadp-15-ibm-com:6443/settings.json volumesnapshotclass.snapshot.storage.k8s.io/ocs-storagecluster-cephfsplugin-snapclass annotated volumesnapshotclass.snapshot.storage.k8s.io/ocs-storagecluster-rbdplugin-snapclass annotated 2024/05/30 07:09:32 Setting up clients 2024/05/30 07:09:32 Getting default StorageClass... 2024/05/30 07:09:32 Checking default storage class count Run the command: oc get sc 2024/05/30 07:09:32 Got default StorageClass ocs-storagecluster-ceph-rbd 2024/05/30 07:09:32 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE localblock kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 2d5h ocs-storagecluster-ceph-rbd (default) openshift-storage.rbd.csi.ceph.com Delete Immediate true 2d3h ocs-storagecluster-ceph-rgw openshift-storage.ceph.rook.io/bucket Delete Immediate false 2d3h ocs-storagecluster-cephfs openshift-storage.cephfs.csi.ceph.com Delete Immediate true 2d3h openshift-storage.noobaa.io openshift-storage.noobaa.io/obc Delete Immediate false 2d3h 2024/05/30 07:09:32 Using velero prefix: velero-e2e-1765bb87-1e75-11ef-9641-fa163e318b23 Running Suite: OADP E2E Suite - /root/oadp/oadp-e2e-qe/e2e ========================================================== Random Seed: 1717067360 Will run 1 of 147 specs ------------------------------ [SynchronizedBeforeSuite]  /root/oadp/oadp-e2e-qe/e2e/e2e_suite_test.go:73 2024/05/30 07:09:32 Update resource allocation for the manager pod in the OADP namespace: openshift-adp 2024/05/30 07:09:32 Get the OADP ClusterServiceVersion in the namespace: openshift-adp 2024/05/30 07:09:32 Successfully got the OADP ClusterServiceVersion name: oadp-operator.v1.2.5 2024/05/30 07:09:32 Successfully updated the OADP ClusterServiceVersion name: oadp-operator.v1.2.5 [SynchronizedBeforeSuite] PASSED [0.172 seconds] ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ Backup Restore tests PVC mounted to a Completed pod [tc-id:OADP-404] PVC bound to Completed Pod with Restic /root/oadp/oadp-e2e-qe/e2e/app_backup/skip_pvc_not_running_pod.go:58 STEP: Create a Backup @ 05/30/24 07:09:32.622 2024/05/30 07:09:32 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 05/30/24 07:09:32.626 Updating resource allocations for Restic because running tests in parallel Updating resource allocations for Velero because running tests in parallel 2024/05/30 07:09:32 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "a095fdcb-7ecc-4144-a3ed-612dea1d88b5", "resourceVersion": "1448865", "generation": 1, "creationTimestamp": "2024-05-30T11:09:32Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-05-30T11:09:32Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:restic": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": { ".": {}, "f:requests": { ".": {}, "f:cpu": {}, "f:memory": {} } } } }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": { ".": {}, "f:requests": { ".": {}, "f:cpu": {}, "f:memory": {} } } } } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-1" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "newocpbucket", "prefix": "velero-e2e-177e4a7f-1e75-11ef-9641-fa163e318b23" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ], "podConfig": { "resourceAllocations": { "requests": { "cpu": "100m", "memory": "64Mi" } } } }, "restic": { "enable": true, "podConfig": { "resourceAllocations": { "requests": { "cpu": "100m", "memory": "64Mi" } } } } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 05/30/24 07:09:32.722 2024/05/30 07:09:32 Waiting for velero pod to be running 2024/05/30 07:09:37 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 05/30/24 07:09:37.742 2024/05/30 07:09:37 Checking for correct number of running Restic pods... STEP: Installing application for case app @ 05/30/24 07:09:37.756 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Check namespace test-oadp-404] *** ok: [localhost] TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Create namespace test-oadp-404] *** changed: [localhost] TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Create a pod mounted to a pvc with data] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=13  changed=5  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/05/30 07:09:44 2024-05-30 07:09:39,784 p=15077 u=root n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-05-30 07:09:39,784 p=15077 u=root n=ansible | changed: [localhost] 2024-05-30 07:09:40,120 p=15077 u=root n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-05-30 07:09:40,120 p=15077 u=root n=ansible | changed: [localhost] 2024-05-30 07:09:40,455 p=15077 u=root n=ansible | TASK [Get current admin token] ************************************************* 2024-05-30 07:09:40,455 p=15077 u=root n=ansible | changed: [localhost] 2024-05-30 07:09:40,478 p=15077 u=root n=ansible | TASK [set_fact] **************************************************************** 2024-05-30 07:09:40,478 p=15077 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:40,780 p=15077 u=root n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-05-30 07:09:40,780 p=15077 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:40,822 p=15077 u=root n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-05-30 07:09:40,822 p=15077 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:40,856 p=15077 u=root n=ansible | TASK [set_fact] **************************************************************** 2024-05-30 07:09:40,856 p=15077 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:40,888 p=15077 u=root n=ansible | PLAY [Execute Task] ************************************************************ 2024-05-30 07:09:41,603 p=15077 u=root n=ansible | TASK [Gathering Facts] ********************************************************* 2024-05-30 07:09:41,603 p=15077 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:42,624 p=15077 u=root n=ansible | TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Check namespace test-oadp-404] *** 2024-05-30 07:09:42,625 p=15077 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:43,106 p=15077 u=root n=ansible | TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Create namespace test-oadp-404] *** 2024-05-30 07:09:43,107 p=15077 u=root n=ansible | changed: [localhost] 2024-05-30 07:09:44,193 p=15077 u=root n=ansible | TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Create a pod mounted to a pvc with data] *** 2024-05-30 07:09:44,194 p=15077 u=root n=ansible | changed: [localhost] 2024-05-30 07:09:44,403 p=15077 u=root n=ansible | PLAY RECAP ********************************************************************* 2024-05-30 07:09:44,403 p=15077 u=root n=ansible | localhost : ok=13 changed=5 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 05/30/24 07:09:44.444 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Check if job exists] *** ok: [localhost] TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Assert that the job exists] *** ok: [localhost] => {  "changed": false,  "msg": "Job data-generation-job exists" } FAILED - RETRYING: [localhost]: Wait for job to complete successfully (20 retries left). FAILED - RETRYING: [localhost]: Wait for job to complete successfully (19 retries left). TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Wait for job to complete successfully] *** ok: [localhost] TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Wait for pod created by job to exist and complete] *** ok: [localhost] TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Assert that the pod exists and has completed] *** ok: [localhost] => {  "changed": false,  "msg": "Pod for job data-generation-job exists and has completed" } PLAY RECAP ********************************************************************* localhost : ok=15  changed=3  unreachable=0 failed=0 skipped=4  rescued=0 ignored=0 2024/05/30 07:10:02 2024-05-30 07:09:46,511 p=15381 u=root n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-05-30 07:09:46,511 p=15381 u=root n=ansible | changed: [localhost] 2024-05-30 07:09:46,850 p=15381 u=root n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-05-30 07:09:46,850 p=15381 u=root n=ansible | changed: [localhost] 2024-05-30 07:09:47,189 p=15381 u=root n=ansible | TASK [Get current admin token] ************************************************* 2024-05-30 07:09:47,189 p=15381 u=root n=ansible | changed: [localhost] 2024-05-30 07:09:47,212 p=15381 u=root n=ansible | TASK [set_fact] **************************************************************** 2024-05-30 07:09:47,212 p=15381 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:47,527 p=15381 u=root n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-05-30 07:09:47,527 p=15381 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:47,571 p=15381 u=root n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-05-30 07:09:47,572 p=15381 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:47,608 p=15381 u=root n=ansible | TASK [set_fact] **************************************************************** 2024-05-30 07:09:47,609 p=15381 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:47,641 p=15381 u=root n=ansible | PLAY [Execute Task] ************************************************************ 2024-05-30 07:09:48,359 p=15381 u=root n=ansible | TASK [Gathering Facts] ********************************************************* 2024-05-30 07:09:48,360 p=15381 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:49,538 p=15381 u=root n=ansible | TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Check if job exists] *** 2024-05-30 07:09:49,538 p=15381 u=root n=ansible | ok: [localhost] 2024-05-30 07:09:49,578 p=15381 u=root n=ansible | TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Assert that the job exists] *** 2024-05-30 07:09:49,578 p=15381 u=root n=ansible | ok: [localhost] => { "changed": false, "msg": "Job data-generation-job exists" } 2024-05-30 07:10:01,824 p=15381 u=root n=ansible | TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Wait for job to complete successfully] *** 2024-05-30 07:10:01,825 p=15381 u=root n=ansible | ok: [localhost] 2024-05-30 07:10:02,569 p=15381 u=root n=ansible | TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Wait for pod created by job to exist and complete] *** 2024-05-30 07:10:02,569 p=15381 u=root n=ansible | ok: [localhost] 2024-05-30 07:10:02,618 p=15381 u=root n=ansible | TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Assert that the pod exists and has completed] *** 2024-05-30 07:10:02,619 p=15381 u=root n=ansible | ok: [localhost] => { "changed": false, "msg": "Pod for job data-generation-job exists and has completed" } 2024-05-30 07:10:02,661 p=15381 u=root n=ansible | PLAY RECAP ********************************************************************* 2024-05-30 07:10:02,661 p=15381 u=root n=ansible | localhost : ok=15 changed=3 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0 STEP: Creating backup app-1782afa9-1e75-11ef-9641-fa163e318b23 @ 05/30/24 07:10:02.71 2024/05/30 07:10:02 Wait until backup app-1782afa9-1e75-11ef-9641-fa163e318b23 is completed backup phase: Completed STEP: Verify backup app-1782afa9-1e75-11ef-9641-fa163e318b23 has completed successfully @ 05/30/24 07:10:22.73 2024/05/30 07:10:22 Backup for case app succeeded 2024/05/30 07:10:23 Exec stderr: "" 2024/05/30 07:10:23 Using Must-gather image: brew.registry.redhat.io/rh-osbs/oadp-oadp-mustgather-rhel8:1.2.5-2 2024/05/30 07:10:23 Cleaning resources 2024/05/30 07:10:23 Delete secret cloud-credentials 2024/05/30 07:10:23 Delete DPA CR 2024/05/30 07:10:23 Verify Velero pods are terminated 2024/05/30 07:10:28 Cleaning app [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Remove namespace test-oadp-404] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=8  rescued=0 ignored=0 2024/05/30 07:10:48 2024-05-30 07:10:30,281 p=15697 u=root n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-05-30 07:10:30,281 p=15697 u=root n=ansible | changed: [localhost] 2024-05-30 07:10:30,640 p=15697 u=root n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-05-30 07:10:30,641 p=15697 u=root n=ansible | changed: [localhost] 2024-05-30 07:10:31,016 p=15697 u=root n=ansible | TASK [Get current admin token] ************************************************* 2024-05-30 07:10:31,016 p=15697 u=root n=ansible | changed: [localhost] 2024-05-30 07:10:31,041 p=15697 u=root n=ansible | TASK [set_fact] **************************************************************** 2024-05-30 07:10:31,041 p=15697 u=root n=ansible | ok: [localhost] 2024-05-30 07:10:31,347 p=15697 u=root n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-05-30 07:10:31,347 p=15697 u=root n=ansible | ok: [localhost] 2024-05-30 07:10:31,390 p=15697 u=root n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-05-30 07:10:31,390 p=15697 u=root n=ansible | ok: [localhost] 2024-05-30 07:10:31,424 p=15697 u=root n=ansible | TASK [set_fact] **************************************************************** 2024-05-30 07:10:31,424 p=15697 u=root n=ansible | ok: [localhost] 2024-05-30 07:10:31,456 p=15697 u=root n=ansible | PLAY [Execute Task] ************************************************************ 2024-05-30 07:10:32,184 p=15697 u=root n=ansible | TASK [Gathering Facts] ********************************************************* 2024-05-30 07:10:32,184 p=15697 u=root n=ansible | ok: [localhost] 2024-05-30 07:10:48,264 p=15697 u=root n=ansible | TASK [/root/oadp/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-pvc-with-data-attached-to-a-completed-pod : Remove namespace test-oadp-404] *** 2024-05-30 07:10:48,264 p=15697 u=root n=ansible | changed: [localhost] 2024-05-30 07:10:48,591 p=15697 u=root n=ansible | PLAY RECAP ********************************************************************* 2024-05-30 07:10:48,592 p=15697 u=root n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0 2024/05/30 07:10:48 Cleaning setup resources for the backup • [76.032 seconds] ------------------------------ SSS ------------------------------ [SynchronizedAfterSuite]  /root/oadp/oadp-e2e-qe/e2e/e2e_suite_test.go:162 2024/05/30 07:10:48 Deleting Velero CR [SynchronizedAfterSuite] PASSED [0.005 seconds] ------------------------------ [DeferCleanup (Suite)]  /root/oadp/oadp-e2e-qe/utils/subscription/subscription.go:216 2024/05/30 07:10:48 Successfully got the OADP ClusterServiceVersion name: oadp-operator.v1.2.5 2024/05/30 07:10:48 Successfully updated the OADP ClusterServiceVersion name: oadp-operator.v1.2.5 [DeferCleanup (Suite)] PASSED [0.126 seconds] ------------------------------ [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report autogenerated by Ginkgo [ReportAfterSuite] PASSED [0.012 seconds] ------------------------------ Ran 1 of 147 Specs in 76.341 seconds SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 146 Skipped PASS Ginkgo ran 1 suite in 1m28.615031461s Test Suite Passed