Ginkgo detected a version mismatch between the Ginkgo CLI and the version of Ginkgo imported by your packages: Ginkgo CLI Version: 2.23.4 Mismatched package versions found: 2.22.0 used by e2e Ginkgo will continue to attempt to run but you may see errors (including flag parsing errors) and should either update your go.mod or your version of the Ginkgo CLI to match. To install the matching version of the CLI run go install github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file. Alternatively you can use go run github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file to invoke the matching version of the Ginkgo CLI. If you are attempting to test multiple packages that each have a different version of the Ginkgo library with a single Ginkgo CLI that is currently unsupported.  2025/07/06 15:57:40 Setting up clients 2025/07/06 15:57:40 Getting default StorageClass... 2025/07/06 15:57:40 Checking default storage class count Run the command: oc get sc 2025/07/06 15:57:41 Got default StorageClass ocs-storagecluster-ceph-rbd 2025/07/06 15:57:42 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE csi-manila-ceph manila.csi.openstack.org Delete Immediate true 4d8h csi-manila-netapp manila.csi.openstack.org Delete Immediate true 4d8h hostpath-csi-basic kubevirt.io.hostpath-provisioner Delete WaitForFirstConsumer false 4d7h hostpath-csi-pvc-block kubevirt.io.hostpath-provisioner Delete WaitForFirstConsumer false 4d7h local-block-hpp kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 4d7h local-block-ocs kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 4d7h ocs-storagecluster-ceph-rbd (default) openshift-storage.rbd.csi.ceph.com Delete Immediate true 4d7h ocs-storagecluster-ceph-rbd-virtualization openshift-storage.rbd.csi.ceph.com Delete Immediate true 4d7h ocs-storagecluster-ceph-rgw openshift-storage.ceph.rook.io/bucket Delete Immediate false 3h31m ocs-storagecluster-cephfs openshift-storage.cephfs.csi.ceph.com Delete Immediate true 4d7h standard-csi cinder.csi.openstack.org Delete WaitForFirstConsumer true 4d8h 2025/07/06 15:57:42 Using velero prefix: velero-e2e-cdc64acc-5a68-11f0-a2c6-c69ad4ed902f 2025/07/06 15:57:42 Checking default storage class count Running Suite: OADP E2E Suite - /home/amos/git/d/oadp-e2e-qe/e2e ================================================================ Random Seed: 1751806658 Will run 2 of 223 specs ------------------------------ [SynchronizedBeforeSuite]  /home/amos/git/d/oadp-e2e-qe/e2e/e2e_suite_test.go:84 > Enter [SynchronizedBeforeSuite] TOP-LEVEL @ 07/06/25 15:57:42.956 < Exit [SynchronizedBeforeSuite] TOP-LEVEL @ 07/06/25 15:57:42.956 (0s) > Enter [SynchronizedBeforeSuite] TOP-LEVEL @ 07/06/25 15:57:42.956 2025/07/06 15:57:43 Reading the VSL credentials file: /home/amos/helpers/minio/cred < Exit [SynchronizedBeforeSuite] TOP-LEVEL @ 07/06/25 15:57:43.82 (864ms) [SynchronizedBeforeSuite] PASSED [0.864 seconds] ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ Backup restore tests Remove Backup [tc-id:OADP-229] Verify volumesnapshotcontent are removed /home/amos/git/d/oadp-e2e-qe/e2e/app_backup/csi-backup.go:51 > Enter [BeforeEach] Backup restore tests @ 07/06/25 15:57:43.821 < Exit [BeforeEach] Backup restore tests @ 07/06/25 15:57:44.021 (200ms) > Enter [JustBeforeEach] TOP-LEVEL @ 07/06/25 15:57:44.021 < Exit [JustBeforeEach] TOP-LEVEL @ 07/06/25 15:57:44.021 (0s) > Enter [It] [tc-id:OADP-229] Verify volumesnapshotcontent are removed @ 07/06/25 15:57:44.021 STEP: Create a Backup @ 07/06/25 15:57:44.021 2025/07/06 15:57:44 Skipping the Dpa creation part STEP: Prepare backup resources, depending on the volumes backup type @ 07/06/25 15:57:44.022 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2025/07/06 15:57:45 The 'openshift-storage' namespace exists 2025/07/06 15:57:45 Checking default storage class count 2025/07/06 15:57:45 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2025/07/06 15:57:46 Snapclass 'example-snapclass' already exists, skip creating 2025/07/06 15:57:46 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2025/07/06 15:57:46 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd STEP: Installing application for case backup1 @ 07/06/25 15:57:46.89 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-229] *** ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left). TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] Pausing for 30 seconds TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=20  changed=10  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2025/07/06 15:59:00 2025-07-06 15:57:49,606 p=121246 u=amos n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-07-06 15:57:49,606 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:57:49,855 p=121246 u=amos n=ansible INFO| TASK [Get cluster endpoint] **************************************************** 2025-07-06 15:57:49,855 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:57:50,096 p=121246 u=amos n=ansible INFO| TASK [Get current admin token] ************************************************* 2025-07-06 15:57:50,096 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:57:50,109 p=121246 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 15:57:50,109 p=121246 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:57:50,428 p=121246 u=amos n=ansible INFO| TASK [Extract kubernetes minor version from cluster_info] ********************** 2025-07-06 15:57:50,428 p=121246 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:57:50,451 p=121246 u=amos n=ansible INFO| TASK [Map kubernetes minor to ocp releases] ************************************ 2025-07-06 15:57:50,452 p=121246 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:57:50,465 p=121246 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 15:57:50,465 p=121246 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:57:50,467 p=121246 u=amos n=ansible INFO| PLAY [Execute Task] ************************************************************ 2025-07-06 15:57:51,101 p=121246 u=amos n=ansible INFO| TASK [Gathering Facts] ********************************************************* 2025-07-06 15:57:51,101 p=121246 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:57:52,293 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-229] *** 2025-07-06 15:57:52,293 p=121246 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:57:53,275 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2025-07-06 15:57:53,275 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:57:56,373 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2025-07-06 15:57:56,373 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:57:57,514 p=121246 u=amos n=ansible INFO| FAILED - RETRYING: [localhost]: Check pod status (30 retries left). 2025-07-06 15:58:03,518 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2025-07-06 15:58:03,518 p=121246 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:58:08,936 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2025-07-06 15:58:08,937 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:58:10,360 p=121246 u=amos n=ansible INFO| FAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left). 2025-07-06 15:58:16,779 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2025-07-06 15:58:16,779 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:58:21,135 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2025-07-06 15:58:21,136 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:58:27,019 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2025-07-06 15:58:27,019 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:58:30,796 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2025-07-06 15:58:30,796 p=121246 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:58:30,810 p=121246 u=amos n=ansible INFO| Pausing for 30 seconds 2025-07-06 15:59:00,812 p=121246 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** 2025-07-06 15:59:00,812 p=121246 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:59:00,903 p=121246 u=amos n=ansible INFO| PLAY RECAP ********************************************************************* 2025-07-06 15:59:00,903 p=121246 u=amos n=ansible INFO| localhost : ok=20 changed=10 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Verify Application deployment @ 07/06/25 15:59:00.943 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2025/07/06 15:59:13 2025-07-06 15:59:02,705 p=121771 u=amos n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-07-06 15:59:02,705 p=121771 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:59:02,927 p=121771 u=amos n=ansible INFO| TASK [Get cluster endpoint] **************************************************** 2025-07-06 15:59:02,927 p=121771 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:59:03,186 p=121771 u=amos n=ansible INFO| TASK [Get current admin token] ************************************************* 2025-07-06 15:59:03,187 p=121771 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:59:03,199 p=121771 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 15:59:03,199 p=121771 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:59:03,527 p=121771 u=amos n=ansible INFO| TASK [Extract kubernetes minor version from cluster_info] ********************** 2025-07-06 15:59:03,527 p=121771 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:59:03,553 p=121771 u=amos n=ansible INFO| TASK [Map kubernetes minor to ocp releases] ************************************ 2025-07-06 15:59:03,553 p=121771 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:59:03,570 p=121771 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 15:59:03,570 p=121771 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:59:03,572 p=121771 u=amos n=ansible INFO| PLAY [Execute Task] ************************************************************ 2025-07-06 15:59:04,238 p=121771 u=amos n=ansible INFO| TASK [Gathering Facts] ********************************************************* 2025-07-06 15:59:04,239 p=121771 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:59:06,480 p=121771 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2025-07-06 15:59:06,481 p=121771 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 15:59:08,089 p=121771 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2025-07-06 15:59:08,089 p=121771 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:59:10,883 p=121771 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2025-07-06 15:59:10,883 p=121771 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:59:13,631 p=121771 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2025-07-06 15:59:13,631 p=121771 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 15:59:13,636 p=121771 u=amos n=ansible INFO| PLAY RECAP ********************************************************************* 2025-07-06 15:59:13,636 p=121771 u=amos n=ansible INFO| localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 2025/07/06 15:59:13 {{ } { } [{{ } {mysql-data test-oadp-229 7aab630d-717c-4562-9169-010b8495775a 4844732 0 2025-07-06 15:57:55 +0300 IDT map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes reclaimspace.csiaddons.openshift.io/cronjob:mysql-data-1751806675 reclaimspace.csiaddons.openshift.io/schedule:@weekly volume.beta.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com volume.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2025-07-06 15:57:55 +0300 IDT FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {csi-addons-manager Update v1 2025-07-06 15:57:55 +0300 IDT FieldsV1 {"f:metadata":{"f:annotations":{"f:reclaimspace.csiaddons.openshift.io/cronjob":{},"f:reclaimspace.csiaddons.openshift.io/schedule":{}}}} } {kube-controller-manager Update v1 2025-07-06 15:57:55 +0300 IDT FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2025-07-06 15:57:55 +0300 IDT FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-7aab630d-717c-4562-9169-010b8495775a 0xc000119e70 0xc000119ea0 nil nil } {Bound [ReadWriteOnce] map[storage:{{2147483648 0} {} 2Gi BinarySI}] [] map[] map[] nil}} {{ } {mysql-data1 test-oadp-229 90d06953-c72f-49b3-b81e-9faeb73b773c 4844749 0 2025-07-06 15:57:55 +0300 IDT map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes reclaimspace.csiaddons.openshift.io/cronjob:mysql-data1-1751806676 reclaimspace.csiaddons.openshift.io/schedule:@weekly volume.beta.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com volume.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2025-07-06 15:57:55 +0300 IDT FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2025-07-06 15:57:55 +0300 IDT FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2025-07-06 15:57:55 +0300 IDT FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status} {csi-addons-manager Update v1 2025-07-06 15:57:56 +0300 IDT FieldsV1 {"f:metadata":{"f:annotations":{"f:reclaimspace.csiaddons.openshift.io/cronjob":{},"f:reclaimspace.csiaddons.openshift.io/schedule":{}}}} }]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-90d06953-c72f-49b3-b81e-9faeb73b773c 0xc000af66d0 0xc000af66f0 nil nil } {Bound [ReadWriteOnce] map[storage:{{2147483648 0} {} 2Gi BinarySI}] [] map[] map[] nil}}]} STEP: Creating backup backup1-cebf2fdb-5a68-11f0-a2c6-c69ad4ed902f @ 07/06/25 15:59:13.847 2025/07/06 15:59:14 Wait until backup backup1-cebf2fdb-5a68-11f0-a2c6-c69ad4ed902f is completed backup phase: InProgress backup phase: Completed 2025/07/06 15:59:34 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2025/07/06 15:59:42 Run velero describe on the backup 2025/07/06 15:59:42 [./velero describe backup backup1-cebf2fdb-5a68-11f0-a2c6-c69ad4ed902f -n openshift-adp --details --insecure-skip-tls-verify] 2025/07/06 15:59:43 Exec stderr: "" 2025/07/06 15:59:43 Name: backup1-cebf2fdb-5a68-11f0-a2c6-c69ad4ed902f Namespace: openshift-adp Labels: velero.io/storage-location=dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.32.5 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=32 Phase: Completed Namespaces: Included: test-oadp-229 Excluded: Resources: Included: * Excluded: volumesnapshots.snapshot.storage.k8s.io, volumesnapshotcontents.snapshot.storage.k8s.io Cluster-scoped: auto Label selector: Or label selector: Storage Location: dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2025-07-06 12:59:14 +0000 UTC Completed: 2025-07-06 12:59:26 +0000 UTC Expiration: 2025-08-05 12:59:14 +0000 UTC Total items to be backed up: 51 Items backed up: 51 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io test-oadp-229/velero-mysql-data-b4gqt: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: test-oadp-229/velero-mysql-data-b4gqt/2025-07-06T12:59:20Z Items to Update: volumesnapshots.snapshot.storage.k8s.io test-oadp-229/velero-mysql-data-b4gqt volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-7b0e3631-321f-468c-aceb-e2f37c5deb3f Phase: Completed Created: 2025-07-06 12:59:20 +0000 UTC Started: 2025-07-06 12:59:20 +0000 UTC Updated: 2025-07-06 12:59:25 +0000 UTC Operation for volumesnapshots.snapshot.storage.k8s.io test-oadp-229/velero-mysql-data1-86p65: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: test-oadp-229/velero-mysql-data1-86p65/2025-07-06T12:59:25Z Items to Update: volumesnapshots.snapshot.storage.k8s.io test-oadp-229/velero-mysql-data1-86p65 volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-a83b0e0e-e979-4b43-902d-ecd94da3243a Phase: Completed Created: 2025-07-06 12:59:25 +0000 UTC Started: 2025-07-06 12:59:25 +0000 UTC Updated: 2025-07-06 12:59:25 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - reclaimspacecronjobs.csiaddons.openshift.io apps/v1/Deployment: - test-oadp-229/mysql apps/v1/ReplicaSet: - test-oadp-229/mysql-64c9d6466 authorization.openshift.io/v1/RoleBinding: - test-oadp-229/admin - test-oadp-229/system:deployers - test-oadp-229/system:image-builders - test-oadp-229/system:image-pullers csiaddons.openshift.io/v1alpha1/ReclaimSpaceCronJob: - test-oadp-229/mysql-data-1751806675 - test-oadp-229/mysql-data1-1751806676 discovery.k8s.io/v1/EndpointSlice: - test-oadp-229/mysql-qqjhl rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-229/admin - test-oadp-229/system:deployers - test-oadp-229/system:image-builders - test-oadp-229/system:image-pullers snapshot.storage.k8s.io/v1/VolumeSnapshot: - test-oadp-229/velero-mysql-data-b4gqt - test-oadp-229/velero-mysql-data1-86p65 snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-7b0e3631-321f-468c-aceb-e2f37c5deb3f - snapcontent-a83b0e0e-e979-4b43-902d-ecd94da3243a v1/ConfigMap: - test-oadp-229/kube-root-ca.crt - test-oadp-229/openshift-service-ca.crt v1/Endpoints: - test-oadp-229/mysql v1/Event: - test-oadp-229/mysql-64c9d6466-qw64c.184faae3fb2204af - test-oadp-229/mysql-64c9d6466-qw64c.184faae41fa5c879 - test-oadp-229/mysql-64c9d6466-qw64c.184faae42045eeb7 - test-oadp-229/mysql-64c9d6466-qw64c.184faae4692130a6 - test-oadp-229/mysql-64c9d6466-qw64c.184faae46a85c4ad - test-oadp-229/mysql-64c9d6466-qw64c.184faae46efbfa88 - test-oadp-229/mysql-64c9d6466-qw64c.184faae46f848ffd - test-oadp-229/mysql-64c9d6466.184faae3fa8cd387 - test-oadp-229/mysql-data.184faae3c9192256 - test-oadp-229/mysql-data.184faae3c91f2e70 - test-oadp-229/mysql-data.184faae3d4de3723 - test-oadp-229/mysql-data1.184faae3e0d7c74e - test-oadp-229/mysql-data1.184faae3e0ec72c4 - test-oadp-229/mysql-data1.184faae3ecfb09a8 - test-oadp-229/mysql.184faae3f9ef7500 v1/Namespace: - test-oadp-229 v1/PersistentVolume: - pvc-7aab630d-717c-4562-9169-010b8495775a - pvc-90d06953-c72f-49b3-b81e-9faeb73b773c v1/PersistentVolumeClaim: - test-oadp-229/mysql-data - test-oadp-229/mysql-data1 v1/Pod: - test-oadp-229/mysql-64c9d6466-qw64c v1/Secret: - test-oadp-229/builder-dockercfg-kvrfj - test-oadp-229/default-dockercfg-mg54m - test-oadp-229/deployer-dockercfg-kb6m2 - test-oadp-229/mysql v1/Service: - test-oadp-229/mysql v1/ServiceAccount: - test-oadp-229/builder - test-oadp-229/default - test-oadp-229/deployer Backup Volumes: Velero-Native Snapshots: CSI Snapshots: test-oadp-229/mysql-data: Snapshot: Operation ID: test-oadp-229/velero-mysql-data-b4gqt/2025-07-06T12:59:20Z Snapshot Content Name: snapcontent-7b0e3631-321f-468c-aceb-e2f37c5deb3f Storage Snapshot ID: 0001-0011-openshift-storage-0000000000000002-d714d27e-2dcc-4083-88c0-355b9a41c765 Snapshot Size (bytes): 2147483648 CSI Driver: openshift-storage.rbd.csi.ceph.com Result: succeeded test-oadp-229/mysql-data1: Snapshot: Operation ID: test-oadp-229/velero-mysql-data1-86p65/2025-07-06T12:59:25Z Snapshot Content Name: snapcontent-a83b0e0e-e979-4b43-902d-ecd94da3243a Storage Snapshot ID: 0001-0011-openshift-storage-0000000000000002-538f3712-86ad-4399-a617-f6be390348ec Snapshot Size (bytes): 2147483648 CSI Driver: openshift-storage.rbd.csi.ceph.com Result: succeeded Pod Volume Backups: HooksAttempted: 0 HooksFailed: 0 STEP: Verify backup backup1-cebf2fdb-5a68-11f0-a2c6-c69ad4ed902f has completed successfully @ 07/06/25 15:59:43.482 2025/07/06 15:59:55 Backup for case backup1 succeeded STEP: Get VolumeSnapshotContent @ 07/06/25 15:59:55.147 2025/07/06 15:59:56 VolumeSnapshotContent name: [FAILED] in [It] - /home/amos/git/d/oadp-e2e-qe/e2e/app_backup/csi-backup.go:67 @ 07/06/25 15:59:56.013 < Exit [It] [tc-id:OADP-229] Verify volumesnapshotcontent are removed @ 07/06/25 15:59:56.013 (2m11.991s) > Enter [JustAfterEach] TOP-LEVEL @ 07/06/25 15:59:56.013 2025/07/06 15:59:56 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9@sha256:e1bc55532e05c21e75881cb9fc591965580494d48e7bc69b84cd062b72272d4c STEP: Get the failed spec name @ 07/06/25 15:59:56.013 2025/07/06 15:59:56 The failed spec name is: Backup restore tests Remove Backup [tc-id:OADP-229] Verify volumesnapshotcontent are removed STEP: Create a folder for all must-gather files if it doesn't exists already @ 07/06/25 15:59:56.013 STEP: Create a folder for the failed spec if it doesn't exists already @ 07/06/25 15:59:56.013 STEP: Run must-gather because the spec failed @ 07/06/25 15:59:56.013 2025/07/06 15:59:56 Log the present working directory path:- /home/amos/git/d/oadp-e2e-qe/e2e 2025/07/06 15:59:56 [adm must-gather --dest-dir /home/amos/git/d/oadp-e2e-qe/e2e/logs/It_Backup_restore_tests_Remove_Backup_tc-id_OADP-229_Verify_volumesnapshotcontent_are_removed --image registry.redhat.io/oadp/oadp-mustgather-rhel9@sha256:e1bc55532e05c21e75881cb9fc591965580494d48e7bc69b84cd062b72272d4c] 2025/07/06 16:00:23 Log all the files present in /home/amos/git/d/oadp-e2e-qe/e2e/logs directory 2025/07/06 16:00:23 It_Backup_hooks_tests_Failed_and_successful_hooks_in_backup_description_tc-id_OADP-624_csi_Mysql_application 2025/07/06 16:00:23 It_Backup_restore_tests_Remove_Backup_tc-id_OADP-229_Verify_volumesnapshotcontent_are_removed STEP: Find must-gather folder and rename it to a shorter more readable name @ 07/06/25 16:00:23.69 2025/07/06 16:00:23 Failed to find must-gather folder 2025/07/06 16:00:23 rename logs/It_Backup_restore_tests_Remove_Backup_tc-id_OADP-229_Verify_volumesnapshotcontent_are_removed/must-gather: no such file or directory [FAILED] in [JustAfterEach] - /home/amos/git/d/oadp-e2e-qe/lib/must_gather_helpers.go:125 @ 07/06/25 16:00:23.69 < Exit [JustAfterEach] TOP-LEVEL @ 07/06/25 16:00:23.69 (27.678s) > Enter [AfterEach] Backup restore tests @ 07/06/25 16:00:23.69 2025/07/06 16:00:23 Cleaning resources 2025/07/06 16:00:23 Delete secret cloud-credentials < Exit [AfterEach] Backup restore tests @ 07/06/25 16:00:23.87 (180ms) > Enter [DeferCleanup (Each)] Remove Backup @ 07/06/25 16:00:23.87 < Exit [DeferCleanup (Each)] Remove Backup @ 07/06/25 16:00:23.87 (0s) > Enter [DeferCleanup (Each)] Remove Backup @ 07/06/25 16:00:23.871 2025/07/06 16:00:23 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-229] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=18  rescued=0 ignored=0 2025/07/06 16:00:44 2025-07-06 16:00:25,592 p=122279 u=amos n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-07-06 16:00:25,592 p=122279 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:00:25,825 p=122279 u=amos n=ansible INFO| TASK [Get cluster endpoint] **************************************************** 2025-07-06 16:00:25,825 p=122279 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:00:26,047 p=122279 u=amos n=ansible INFO| TASK [Get current admin token] ************************************************* 2025-07-06 16:00:26,047 p=122279 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:00:26,058 p=122279 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 16:00:26,059 p=122279 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:26,374 p=122279 u=amos n=ansible INFO| TASK [Extract kubernetes minor version from cluster_info] ********************** 2025-07-06 16:00:26,374 p=122279 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:26,395 p=122279 u=amos n=ansible INFO| TASK [Map kubernetes minor to ocp releases] ************************************ 2025-07-06 16:00:26,395 p=122279 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:26,409 p=122279 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 16:00:26,410 p=122279 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:26,411 p=122279 u=amos n=ansible INFO| PLAY [Execute Task] ************************************************************ 2025-07-06 16:00:26,999 p=122279 u=amos n=ansible INFO| TASK [Gathering Facts] ********************************************************* 2025-07-06 16:00:26,999 p=122279 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:44,209 p=122279 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-229] *** 2025-07-06 16:00:44,209 p=122279 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:00:44,453 p=122279 u=amos n=ansible INFO| PLAY RECAP ********************************************************************* 2025-07-06 16:00:44,454 p=122279 u=amos n=ansible INFO| localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=18 rescued=0 ignored=0 < Exit [DeferCleanup (Each)] Remove Backup @ 07/06/25 16:00:44.505 (20.634s) > Enter [DeferCleanup (Each)] Remove Backup @ 07/06/25 16:00:44.505 2025/07/06 16:00:44 Cleaning setup resources for the backup 2025/07/06 16:00:44 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2025/07/06 16:00:44 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd 2025/07/06 16:00:45 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] Remove Backup @ 07/06/25 16:00:45.479 (974ms) • [FAILED] [181.658 seconds] Backup restore tests Remove Backup [It] [tc-id:OADP-229] Verify volumesnapshotcontent are removed /home/amos/git/d/oadp-e2e-qe/e2e/app_backup/csi-backup.go:51 [FAILED] Unexpected error: <*exec.ExitError | 0xc0006d4000>: exit status 1 { ProcessState: { pid: 122204, status: 256, rusage: { Utime: {Sec: 0, Usec: 131647}, Stime: {Sec: 0, Usec: 24658}, Maxrss: 102848, Ixrss: 0, Idrss: 0, Isrss: 0, Minflt: 10173, Majflt: 0, Nswap: 0, Inblock: 0, Oublock: 864, Msgsnd: 0, Msgrcv: 0, Nsignals: 0, Nvcsw: 907, Nivcsw: 10, }, }, Stderr: [101, 114, 114, 111, 114, 58, 32, 101, 114, 114, 111, 114, 32, 101, 120, 101, 99, 117, 116, 105, 110, 103, 32, 106, 115, 111, 110, 112, 97, 116, 104, 32, 34, 39, 123, 46, 105, 116, 101, 109, 115, 91, 45, 49, 93, 46, 109, 101, 116, 97, 100, 97, 116, 97, 46, 110, 97, 109, 101, 125, 39, 34, 58, 32, 69, 114, 114, 111, 114, 32, 101, 120, 101, 99, 117, 116, 105, 110, 103, 32, 116, 101, 109, 112, 108, 97, 116, 101, 58, 32, 97, 114, 114, 97, 121, 32, 105, 110, 100, 101, 120, 32, 111, 117, 116, 32, 111, 102, 32, 98, 111, 117, 110, 100, 115, 58, 32, 105, 110, 100, 101, 120, 32, 45, 49, 44, 32, 108, 101, 110, 103, 116, 104, 32, 48, 46, 32, 80, 114, 105, 110, 116, 105, 110, 103, 32, 109, 111, 114, 101, 32, 105, 110, 102, 111, 114, 109, 97, 116, 105, 111, 110, 32, 102, 111, 114, 32, 100, 101, 98, 117, 103, 103, 105, 110, 103, 32, 116, 104, 101, 32, 116, 101, 109, 112, 108, 97, 116, 101, 58, 10, 9, 116, 101, 109, 112, 108, 97, 116, 101, 32, 119, 97, 115, 58, 10, 9, 9, 39, 123, 46, 105, 116, 101, 109, 115, 91, 45, 49, 93, 46, 109, 101, 116, 97, 100, 97, 116, 97, 46, 110, 97, 109, 101, 125, 39, 10, 9, 111, 98, 106, 101, 99, 116, 32, 103, 105, 118, 101, 110, 32, 116, 111, 32, 106, 115, 111, 110, 112, 97, 116, 104, 32, 101, 110, 103, 105, 110, 101, 32, 119, 97, 115, 58, 10, 9, 9, 109, 97, 112, 91, 115, 116, 114, 105, 110, 103, 93, 105, 110, 116, 101, 114, 102, 97, 99, 101, 32, 123, 125, 123, 34, 97, 112, 105, 86, 101, 114, 115, 105, 111, 110, 34, 58, 34, 118, 49, 34, 44, 32, 34, 105, 116, 101, 109, 115, 34, 58, 91, 93, 105, 110, 116, 101, 114, 102, 97, 99, 101, 32, 123, 125, 123, 125, 44, 32, 34, 107, 105, 110, 100, 34, 58, 34, 76, 105, 115, 116, 34, 44, 32, 34, 109, 101, 116, 97, 100, 97, 116, 97, 34, 58, 109, 97, 112, 91, 115, 116, 114, 105, 110, 103, 93, 105, 110, 116, 101, 114, 102, 97, 99, 101, 32, 123, 125, 123, 34, 114, 101, 115, 111, 117, 114, 99, 101, 86, 101, 114, 115, 105, 111, 110, 34, 58, 34, 34, 125, 125, 10, 10, 10], } occurred In [It] at: /home/amos/git/d/oadp-e2e-qe/e2e/app_backup/csi-backup.go:67 @ 07/06/25 15:59:56.013 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ Backup hooks tests Failed and successful hooks in backup description [tc-id:OADP-624] [csi] Mysql application /home/amos/git/d/oadp-e2e-qe/e2e/hooks/backup_hooks.go:159 > Enter [BeforeEach] Backup hooks tests @ 07/06/25 16:00:45.48 < Exit [BeforeEach] Backup hooks tests @ 07/06/25 16:00:45.64 (161ms) > Enter [JustBeforeEach] TOP-LEVEL @ 07/06/25 16:00:45.64 < Exit [JustBeforeEach] TOP-LEVEL @ 07/06/25 16:00:45.64 (0s) > Enter [It] [tc-id:OADP-624] [csi] Mysql application @ 07/06/25 16:00:45.64 STEP: Prepare backup resources, depending on the volumes backup type @ 07/06/25 16:00:45.64 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2025/07/06 16:00:46 The 'openshift-storage' namespace exists 2025/07/06 16:00:46 Checking default storage class count 2025/07/06 16:00:47 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2025/07/06 16:00:47 Snapclass 'example-snapclass' doesn't exist, creating 2025/07/06 16:00:47 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2025/07/06 16:00:47 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd STEP: Installing application for case post-restore-hook @ 07/06/25 16:00:48.364 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-624] *** ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] Pausing for 30 seconds TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=19  changed=8  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2025/07/06 16:01:45 2025-07-06 16:00:50,208 p=122535 u=amos n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-07-06 16:00:50,208 p=122535 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:00:50,453 p=122535 u=amos n=ansible INFO| TASK [Get cluster endpoint] **************************************************** 2025-07-06 16:00:50,453 p=122535 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:00:50,732 p=122535 u=amos n=ansible INFO| TASK [Get current admin token] ************************************************* 2025-07-06 16:00:50,732 p=122535 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:00:50,747 p=122535 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 16:00:50,747 p=122535 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:51,071 p=122535 u=amos n=ansible INFO| TASK [Extract kubernetes minor version from cluster_info] ********************** 2025-07-06 16:00:51,071 p=122535 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:51,096 p=122535 u=amos n=ansible INFO| TASK [Map kubernetes minor to ocp releases] ************************************ 2025-07-06 16:00:51,096 p=122535 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:51,112 p=122535 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 16:00:51,112 p=122535 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:51,113 p=122535 u=amos n=ansible INFO| PLAY [Execute Task] ************************************************************ 2025-07-06 16:00:51,765 p=122535 u=amos n=ansible INFO| TASK [Gathering Facts] ********************************************************* 2025-07-06 16:00:51,765 p=122535 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:53,111 p=122535 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-624] *** 2025-07-06 16:00:53,112 p=122535 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:56,068 p=122535 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2025-07-06 16:00:56,068 p=122535 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:00:58,473 p=122535 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2025-07-06 16:00:58,473 p=122535 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:01:02,994 p=122535 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2025-07-06 16:01:02,994 p=122535 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:04,536 p=122535 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2025-07-06 16:01:04,537 p=122535 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:08,723 p=122535 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2025-07-06 16:01:08,723 p=122535 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:12,051 p=122535 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2025-07-06 16:01:12,052 p=122535 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:14,885 p=122535 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2025-07-06 16:01:14,885 p=122535 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:14,899 p=122535 u=amos n=ansible INFO| Pausing for 30 seconds 2025-07-06 16:01:44,901 p=122535 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** 2025-07-06 16:01:44,901 p=122535 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:01:45,002 p=122535 u=amos n=ansible INFO| PLAY RECAP ********************************************************************* 2025-07-06 16:01:45,002 p=122535 u=amos n=ansible INFO| localhost : ok=19 changed=8 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 STEP: Verify Application deployment @ 07/06/25 16:01:45.042 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2025/07/06 16:01:56 2025-07-06 16:01:46,859 p=123128 u=amos n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-07-06 16:01:46,859 p=123128 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:47,166 p=123128 u=amos n=ansible INFO| TASK [Get cluster endpoint] **************************************************** 2025-07-06 16:01:47,166 p=123128 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:47,450 p=123128 u=amos n=ansible INFO| TASK [Get current admin token] ************************************************* 2025-07-06 16:01:47,451 p=123128 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:47,466 p=123128 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 16:01:47,466 p=123128 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:01:47,814 p=123128 u=amos n=ansible INFO| TASK [Extract kubernetes minor version from cluster_info] ********************** 2025-07-06 16:01:47,814 p=123128 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:01:47,841 p=123128 u=amos n=ansible INFO| TASK [Map kubernetes minor to ocp releases] ************************************ 2025-07-06 16:01:47,841 p=123128 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:01:47,858 p=123128 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 16:01:47,858 p=123128 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:01:47,859 p=123128 u=amos n=ansible INFO| PLAY [Execute Task] ************************************************************ 2025-07-06 16:01:48,527 p=123128 u=amos n=ansible INFO| TASK [Gathering Facts] ********************************************************* 2025-07-06 16:01:48,527 p=123128 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:01:50,086 p=123128 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2025-07-06 16:01:50,086 p=123128 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:01:51,551 p=123128 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2025-07-06 16:01:51,551 p=123128 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:54,081 p=123128 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2025-07-06 16:01:54,081 p=123128 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:56,779 p=123128 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2025-07-06 16:01:56,779 p=123128 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:01:56,783 p=123128 u=amos n=ansible INFO| PLAY RECAP ********************************************************************* 2025-07-06 16:01:56,783 p=123128 u=amos n=ansible INFO| localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 STEP: Creating backup post-restore-hook @ 07/06/25 16:01:56.828 2025/07/06 16:01:56 Wait until backup post-restore-hook is completed backup phase: InProgress backup phase: PartiallyFailed STEP: Verify backup post-restore-hook has partly failed @ 07/06/25 16:02:17.331 2025/07/06 16:02:37 Backup for case post-restore-hook partly failed as expected STEP: Verify Backup Description: HooksAttempted should be 2 and HooksFailed should be 1 @ 07/06/25 16:02:38.146 2025/07/06 16:02:38 Run velero describe on the backup 2025/07/06 16:02:38 [./velero describe backup post-restore-hook-3b000652-5a69-11f0-a2c6-c69ad4ed902f -n openshift-adp --details --insecure-skip-tls-verify] 2025/07/06 16:02:39 Exec stderr: "" 2025/07/06 16:02:39 Name: post-restore-hook-3b000652-5a69-11f0-a2c6-c69ad4ed902f Namespace: openshift-adp Labels: velero.io/storage-location=dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.32.5 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=32 Phase: PartiallyFailed (run `velero backup logs post-restore-hook-3b000652-5a69-11f0-a2c6-c69ad4ed902f` for more information) Errors: Velero: message: /Error executing hook error: /command terminated with exit code 1 Cluster: Namespaces: Namespaces: Included: test-oadp-624 Excluded: Resources: Included: * Excluded: volumesnapshots.snapshot.storage.k8s.io, volumesnapshotcontents.snapshot.storage.k8s.io Cluster-scoped: auto Label selector: Or label selector: Storage Location: dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Resources: backup-hook-with-failures: Namespaces: Included: test-oadp-624 Excluded: Resources: Included: * Excluded: Label selector: Pre Exec Hook: Container: Command: /usr/bin/sh -c echo 'This is a success hook' && echo 'Info: Success hook executed' 1>&2 && exit 0 On Error: Continue Timeout: 4m0s Post Exec Hook: Container: Command: /usr/bin/sh -c echo 'This is a failure hook in PostHooks' && echo 'Error: Post failure hook triggered' 1>&2 && exit 1 On Error: Continue Timeout: 4m0s Backup Format Version: 1.1.0 Started: 2025-07-06 13:01:56 +0000 UTC Completed: 2025-07-06 13:02:09 +0000 UTC Expiration: 2025-08-05 13:01:56 +0000 UTC Total items to be backed up: 51 Items backed up: 51 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io test-oadp-624/velero-mysql-data-nsvnl: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: test-oadp-624/velero-mysql-data-nsvnl/2025-07-06T13:02:03Z Items to Update: volumesnapshots.snapshot.storage.k8s.io test-oadp-624/velero-mysql-data-nsvnl volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-eb2089cb-0aae-4420-a854-adfe832f4b05 Phase: Completed Created: 2025-07-06 13:02:03 +0000 UTC Started: 2025-07-06 13:02:03 +0000 UTC Updated: 2025-07-06 13:02:08 +0000 UTC Operation for volumesnapshots.snapshot.storage.k8s.io test-oadp-624/velero-mysql-data1-zvrvp: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: test-oadp-624/velero-mysql-data1-zvrvp/2025-07-06T13:02:08Z Items to Update: volumesnapshots.snapshot.storage.k8s.io test-oadp-624/velero-mysql-data1-zvrvp volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-39ff8dc8-f09d-4f1f-ac82-3580990a0659 Phase: Completed Created: 2025-07-06 13:02:08 +0000 UTC Started: 2025-07-06 13:02:08 +0000 UTC Updated: 2025-07-06 13:02:08 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - reclaimspacecronjobs.csiaddons.openshift.io apps/v1/Deployment: - test-oadp-624/mysql apps/v1/ReplicaSet: - test-oadp-624/mysql-64c9d6466 authorization.openshift.io/v1/RoleBinding: - test-oadp-624/admin - test-oadp-624/system:deployers - test-oadp-624/system:image-builders - test-oadp-624/system:image-pullers csiaddons.openshift.io/v1alpha1/ReclaimSpaceCronJob: - test-oadp-624/mysql-data-1751806648 - test-oadp-624/mysql-data1-1751806648 discovery.k8s.io/v1/EndpointSlice: - test-oadp-624/mysql-wf96z rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-624/admin - test-oadp-624/system:deployers - test-oadp-624/system:image-builders - test-oadp-624/system:image-pullers snapshot.storage.k8s.io/v1/VolumeSnapshot: - test-oadp-624/velero-mysql-data-nsvnl - test-oadp-624/velero-mysql-data1-zvrvp snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-39ff8dc8-f09d-4f1f-ac82-3580990a0659 - snapcontent-eb2089cb-0aae-4420-a854-adfe832f4b05 v1/ConfigMap: - test-oadp-624/kube-root-ca.crt - test-oadp-624/openshift-service-ca.crt v1/Endpoints: - test-oadp-624/mysql v1/Event: - test-oadp-624/mysql-64c9d6466-9fszf.184faadd950f5c0c - test-oadp-624/mysql-64c9d6466-9fszf.184faaddb6147ce4 - test-oadp-624/mysql-64c9d6466-9fszf.184faaddb838bc77 - test-oadp-624/mysql-64c9d6466-9fszf.184faadf8cb3fd6b - test-oadp-624/mysql-64c9d6466-9fszf.184faadf8e101cce - test-oadp-624/mysql-64c9d6466-9fszf.184faadf9165395b - test-oadp-624/mysql-64c9d6466-9fszf.184faadf91eb71b0 - test-oadp-624/mysql-64c9d6466.184faadd94438a82 - test-oadp-624/mysql-data.184faadd655706a6 - test-oadp-624/mysql-data.184faadd656507e2 - test-oadp-624/mysql-data.184faadd71b224e3 - test-oadp-624/mysql-data1.184faadd7d333eb5 - test-oadp-624/mysql-data1.184faadd7d40d8d1 - test-oadp-624/mysql-data1.184faadd8a4ba509 - test-oadp-624/mysql.184faadd9366a6c0 v1/Namespace: - test-oadp-624 v1/PersistentVolume: - pvc-1a49aa52-b8b6-4d21-90de-38dde3be59d6 - pvc-a6f85b0e-90e5-4862-91eb-e563e1d8093d v1/PersistentVolumeClaim: - test-oadp-624/mysql-data - test-oadp-624/mysql-data1 v1/Pod: - test-oadp-624/mysql-64c9d6466-9fszf v1/Secret: - test-oadp-624/builder-dockercfg-zpk6s - test-oadp-624/default-dockercfg-mdq6t - test-oadp-624/deployer-dockercfg-hfzpq - test-oadp-624/mysql v1/Service: - test-oadp-624/mysql v1/ServiceAccount: - test-oadp-624/builder - test-oadp-624/default - test-oadp-624/deployer Backup Volumes: Velero-Native Snapshots: CSI Snapshots: test-oadp-624/mysql-data: Snapshot: Operation ID: test-oadp-624/velero-mysql-data-nsvnl/2025-07-06T13:02:03Z Snapshot Content Name: snapcontent-eb2089cb-0aae-4420-a854-adfe832f4b05 Storage Snapshot ID: 0001-0011-openshift-storage-0000000000000002-c55d63eb-e9e3-4fe7-aef5-21a7a53166f6 Snapshot Size (bytes): 2147483648 CSI Driver: openshift-storage.rbd.csi.ceph.com Result: succeeded test-oadp-624/mysql-data1: Snapshot: Operation ID: test-oadp-624/velero-mysql-data1-zvrvp/2025-07-06T13:02:08Z Snapshot Content Name: snapcontent-39ff8dc8-f09d-4f1f-ac82-3580990a0659 Storage Snapshot ID: 0001-0011-openshift-storage-0000000000000002-52e3bd6c-633f-4695-9d4f-f70741abd338 Snapshot Size (bytes): 2147483648 CSI Driver: openshift-storage.rbd.csi.ceph.com Result: succeeded Pod Volume Backups: HooksAttempted: 2 HooksFailed: 1 < Exit [It] [tc-id:OADP-624] [csi] Mysql application @ 07/06/25 16:02:39.76 (1m54.12s) > Enter [JustAfterEach] TOP-LEVEL @ 07/06/25 16:02:39.76 2025/07/06 16:02:39 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9@sha256:e1bc55532e05c21e75881cb9fc591965580494d48e7bc69b84cd062b72272d4c < Exit [JustAfterEach] TOP-LEVEL @ 07/06/25 16:02:39.76 (0s) > Enter [DeferCleanup (Each)] Failed and successful hooks in backup description @ 07/06/25 16:02:39.76 2025/07/06 16:02:39 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-624] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=18  rescued=0 ignored=0 2025/07/06 16:03:03 2025-07-06 16:02:42,591 p=123566 u=amos n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-07-06 16:02:42,591 p=123566 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:02:43,112 p=123566 u=amos n=ansible INFO| TASK [Get cluster endpoint] **************************************************** 2025-07-06 16:02:43,112 p=123566 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:02:43,639 p=123566 u=amos n=ansible INFO| TASK [Get current admin token] ************************************************* 2025-07-06 16:02:43,639 p=123566 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:02:43,664 p=123566 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 16:02:43,664 p=123566 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:02:44,314 p=123566 u=amos n=ansible INFO| TASK [Extract kubernetes minor version from cluster_info] ********************** 2025-07-06 16:02:44,314 p=123566 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:02:44,366 p=123566 u=amos n=ansible INFO| TASK [Map kubernetes minor to ocp releases] ************************************ 2025-07-06 16:02:44,366 p=123566 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:02:44,405 p=123566 u=amos n=ansible INFO| TASK [set_fact] **************************************************************** 2025-07-06 16:02:44,405 p=123566 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:02:44,409 p=123566 u=amos n=ansible INFO| PLAY [Execute Task] ************************************************************ 2025-07-06 16:02:45,593 p=123566 u=amos n=ansible INFO| TASK [Gathering Facts] ********************************************************* 2025-07-06 16:02:45,593 p=123566 u=amos n=ansible INFO| ok: [localhost] 2025-07-06 16:03:03,199 p=123566 u=amos n=ansible INFO| TASK [/home/amos/git/d/oadp-e2e-qe/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-624] *** 2025-07-06 16:03:03,199 p=123566 u=amos n=ansible INFO| changed: [localhost] 2025-07-06 16:03:03,444 p=123566 u=amos n=ansible INFO| PLAY RECAP ********************************************************************* 2025-07-06 16:03:03,444 p=123566 u=amos n=ansible INFO| localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=18 rescued=0 ignored=0 < Exit [DeferCleanup (Each)] Failed and successful hooks in backup description @ 07/06/25 16:03:03.49 (23.73s) > Enter [DeferCleanup (Each)] Failed and successful hooks in backup description @ 07/06/25 16:03:03.49 2025/07/06 16:03:03 Cleaning setup resources for the backup 2025/07/06 16:03:03 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2025/07/06 16:03:03 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd 2025/07/06 16:03:04 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] Failed and successful hooks in backup description @ 07/06/25 16:03:04.437 (947ms) • [138.958 seconds] ------------------------------ SSSS > Enter [ReportAfterEach] [upstream-velero] Credentials suite @ 07/06/25 16:03:04.437 < Exit [ReportAfterEach] [upstream-velero] Credentials suite @ 07/06/25 16:03:04.437 (0s) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ [SynchronizedAfterSuite]  /home/amos/git/d/oadp-e2e-qe/e2e/e2e_suite_test.go:229 > Enter [SynchronizedAfterSuite] TOP-LEVEL @ 07/06/25 16:03:04.439 2025/07/06 16:03:04 Deleting Velero CR < Exit [SynchronizedAfterSuite] TOP-LEVEL @ 07/06/25 16:03:04.642 (203ms) > Enter [SynchronizedAfterSuite] TOP-LEVEL @ 07/06/25 16:03:04.642 < Exit [SynchronizedAfterSuite] TOP-LEVEL @ 07/06/25 16:03:04.642 (0s) [SynchronizedAfterSuite] PASSED [0.203 seconds] ------------------------------ [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report autogenerated by Ginkgo > Enter [ReportAfterSuite] TOP-LEVEL @ 07/06/25 16:03:04.642 < Exit [ReportAfterSuite] TOP-LEVEL @ 07/06/25 16:03:04.65 (8ms) [ReportAfterSuite] PASSED [0.008 seconds] ------------------------------ Summarizing 1 Failure: [FAIL] Backup restore tests Remove Backup [It] [tc-id:OADP-229] Verify volumesnapshotcontent are removed /home/amos/git/d/oadp-e2e-qe/e2e/app_backup/csi-backup.go:67 Ran 2 of 223 Specs in 321.686 seconds FAIL! -- 1 Passed | 1 Failed | 0 Pending | 221 Skipped --- FAIL: TestOADPE2E (322.26s) FAIL Ginkgo ran 1 suite in 5m25.944112739s Test Suite Failed