Extract /home/jenkins/oadp-e2e-qe.tar.gz to /alabama/cspi Extract /home/jenkins/oadp-apps-deployer.tar.gz to /alabama/oadpApps Extract /home/jenkins/mtc-python-client.tar.gz to /alabama/pyclient Create and populate /tmp/test-settings... Login as Kubeadmin to the test cluster at https://api.ci-op-2jmljgmw-2c502.cspilp.interop.ccitredhat.com:6443... WARNING: Using insecure TLS client config. Setting this option is not supported! Login successful. You have access to 78 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". Create virtual environment and install required packages... Collecting ansible_runner Downloading ansible_runner-2.4.2-py3-none-any.whl.metadata (3.2 kB) Collecting pexpect>=4.5 (from ansible_runner) Downloading pexpect-4.9.0-py2.py3-none-any.whl.metadata (2.5 kB) Collecting packaging (from ansible_runner) Downloading packaging-25.0-py3-none-any.whl.metadata (3.3 kB) Collecting python-daemon (from ansible_runner) Downloading python_daemon-3.1.2-py3-none-any.whl.metadata (4.8 kB) Collecting pyyaml (from ansible_runner) Downloading pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.4 kB) Collecting ptyprocess>=0.5 (from pexpect>=4.5->ansible_runner) Downloading ptyprocess-0.7.0-py2.py3-none-any.whl.metadata (1.3 kB) Collecting lockfile>=0.10 (from python-daemon->ansible_runner) Downloading lockfile-0.12.2-py2.py3-none-any.whl.metadata (2.4 kB) Downloading ansible_runner-2.4.2-py3-none-any.whl (79 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.8/79.8 kB 11.3 MB/s eta 0:00:00 Downloading pexpect-4.9.0-py2.py3-none-any.whl (63 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.8/63.8 kB 10.0 MB/s eta 0:00:00 Downloading packaging-25.0-py3-none-any.whl (66 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.5/66.5 kB 9.7 MB/s eta 0:00:00 Downloading python_daemon-3.1.2-py3-none-any.whl (30 kB) Downloading pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (807 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 807.9/807.9 kB 73.5 MB/s eta 0:00:00 Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB) Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB) Installing collected packages: ptyprocess, lockfile, pyyaml, python-daemon, pexpect, packaging, ansible_runner Successfully installed ansible_runner-2.4.2 lockfile-0.12.2 packaging-25.0 pexpect-4.9.0 ptyprocess-0.7.0 python-daemon-3.1.2 pyyaml-6.0.3 [notice] A new release of pip is available: 23.3.2 -> 25.3 [notice] To update, run: pip install --upgrade pip Processing /alabama/oadpApps Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Building wheels for collected packages: ocpdeployer Building wheel for ocpdeployer (pyproject.toml): started Building wheel for ocpdeployer (pyproject.toml): finished with status 'done' Created wheel for ocpdeployer: filename=ocpdeployer-0.0.1-py2.py3-none-any.whl size=241509 sha256=c2490fd350a7c9a10534d4060b62ec962ef53b280aa861dd5b546e2b9523fbd4 Stored in directory: /tmp/pip-ephem-wheel-cache-m1np6d6h/wheels/55/c3/15/eb89266a7928fafe53678a24892891bbfb18405fbd475eb4c6 Successfully built ocpdeployer Installing collected packages: ocpdeployer Successfully installed ocpdeployer-0.0.1 [notice] A new release of pip is available: 23.3.2 -> 25.3 [notice] To update, run: pip install --upgrade pip Processing /alabama/pyclient Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting suds-py3 (from mtc==0.0.1) Downloading suds_py3-1.4.5.0-py3-none-any.whl.metadata (778 bytes) Collecting requests (from mtc==0.0.1) Downloading requests-2.32.5-py3-none-any.whl.metadata (4.9 kB) Collecting jinja2 (from mtc==0.0.1) Downloading jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB) Collecting kubernetes==11.0.0 (from mtc==0.0.1) Downloading kubernetes-11.0.0-py3-none-any.whl.metadata (1.5 kB) Collecting openshift==0.11.2 (from mtc==0.0.1) Downloading openshift-0.11.2.tar.gz (19 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting certifi>=14.05.14 (from kubernetes==11.0.0->mtc==0.0.1) Downloading certifi-2025.11.12-py3-none-any.whl.metadata (2.5 kB) Collecting six>=1.9.0 (from kubernetes==11.0.0->mtc==0.0.1) Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB) Collecting python-dateutil>=2.5.3 (from kubernetes==11.0.0->mtc==0.0.1) Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB) Collecting setuptools>=21.0.0 (from kubernetes==11.0.0->mtc==0.0.1) Using cached setuptools-80.9.0-py3-none-any.whl.metadata (6.6 kB) Requirement already satisfied: pyyaml>=3.12 in /alabama/venv/lib64/python3.12/site-packages (from kubernetes==11.0.0->mtc==0.0.1) (6.0.3) Collecting google-auth>=1.0.1 (from kubernetes==11.0.0->mtc==0.0.1) Downloading google_auth-2.45.0-py2.py3-none-any.whl.metadata (6.8 kB) Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 (from kubernetes==11.0.0->mtc==0.0.1) Downloading websocket_client-1.9.0-py3-none-any.whl.metadata (8.3 kB) Collecting requests-oauthlib (from kubernetes==11.0.0->mtc==0.0.1) Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl.metadata (11 kB) Collecting urllib3>=1.24.2 (from kubernetes==11.0.0->mtc==0.0.1) Downloading urllib3-2.6.2-py3-none-any.whl.metadata (6.6 kB) Collecting python-string-utils (from openshift==0.11.2->mtc==0.0.1) Downloading python_string_utils-1.0.0-py3-none-any.whl.metadata (12 kB) Collecting ruamel.yaml>=0.15 (from openshift==0.11.2->mtc==0.0.1) Downloading ruamel_yaml-0.18.17-py3-none-any.whl.metadata (27 kB) Collecting MarkupSafe>=2.0 (from jinja2->mtc==0.0.1) Downloading markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.7 kB) Collecting charset_normalizer<4,>=2 (from requests->mtc==0.0.1) Downloading charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (37 kB) Collecting idna<4,>=2.5 (from requests->mtc==0.0.1) Downloading idna-3.11-py3-none-any.whl.metadata (8.4 kB) Collecting cachetools<7.0,>=2.0.0 (from google-auth>=1.0.1->kubernetes==11.0.0->mtc==0.0.1) Downloading cachetools-6.2.4-py3-none-any.whl.metadata (5.6 kB) Collecting pyasn1-modules>=0.2.1 (from google-auth>=1.0.1->kubernetes==11.0.0->mtc==0.0.1) Downloading pyasn1_modules-0.4.2-py3-none-any.whl.metadata (3.5 kB) Collecting rsa<5,>=3.1.4 (from google-auth>=1.0.1->kubernetes==11.0.0->mtc==0.0.1) Downloading rsa-4.9.1-py3-none-any.whl.metadata (5.6 kB) Collecting ruamel.yaml.clib>=0.2.15 (from ruamel.yaml>=0.15->openshift==0.11.2->mtc==0.0.1) Downloading ruamel_yaml_clib-0.2.15-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (3.5 kB) Collecting oauthlib>=3.0.0 (from requests-oauthlib->kubernetes==11.0.0->mtc==0.0.1) Downloading oauthlib-3.3.1-py3-none-any.whl.metadata (7.9 kB) Collecting pyasn1<0.7.0,>=0.6.1 (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes==11.0.0->mtc==0.0.1) Downloading pyasn1-0.6.1-py3-none-any.whl.metadata (8.4 kB) Downloading kubernetes-11.0.0-py3-none-any.whl (1.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 11.3 MB/s eta 0:00:00 Downloading jinja2-3.1.6-py3-none-any.whl (134 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.9/134.9 kB 1.3 MB/s eta 0:00:00 Downloading requests-2.32.5-py3-none-any.whl (64 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.7/64.7 kB 834.8 kB/s eta 0:00:00 Downloading suds_py3-1.4.5.0-py3-none-any.whl (298 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 298.8/298.8 kB 3.1 MB/s eta 0:00:00 Downloading certifi-2025.11.12-py3-none-any.whl (159 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 159.4/159.4 kB 1.6 MB/s eta 0:00:00 Downloading charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (153 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 153.5/153.5 kB 1.6 MB/s eta 0:00:00 Downloading google_auth-2.45.0-py2.py3-none-any.whl (233 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.3/233.3 kB 2.6 MB/s eta 0:00:00 Downloading idna-3.11-py3-none-any.whl (71 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.0/71.0 kB 909.9 kB/s eta 0:00:00 Downloading markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (22 kB) Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 kB 3.2 MB/s eta 0:00:00 Downloading ruamel_yaml-0.18.17-py3-none-any.whl (121 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 kB 1.7 MB/s eta 0:00:00 Using cached setuptools-80.9.0-py3-none-any.whl (1.2 MB) Downloading six-1.17.0-py2.py3-none-any.whl (11 kB) Downloading urllib3-2.6.2-py3-none-any.whl (131 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 131.2/131.2 kB 1.5 MB/s eta 0:00:00 Downloading websocket_client-1.9.0-py3-none-any.whl (82 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.6/82.6 kB 1.3 MB/s eta 0:00:00 Downloading python_string_utils-1.0.0-py3-none-any.whl (26 kB) Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl (24 kB) Downloading cachetools-6.2.4-py3-none-any.whl (11 kB) Downloading oauthlib-3.3.1-py3-none-any.whl (160 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.1/160.1 kB 1.6 MB/s eta 0:00:00 Downloading pyasn1_modules-0.4.2-py3-none-any.whl (181 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 1.5 MB/s eta 0:00:00 Downloading rsa-4.9.1-py3-none-any.whl (34 kB) Downloading ruamel_yaml_clib-0.2.15-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (788 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 788.2/788.2 kB 7.0 MB/s eta 0:00:00 Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.1/83.1 kB 672.9 kB/s eta 0:00:00 Building wheels for collected packages: mtc, openshift Building wheel for mtc (pyproject.toml): started Building wheel for mtc (pyproject.toml): finished with status 'done' Created wheel for mtc: filename=mtc-0.0.1-py3-none-any.whl size=31146 sha256=b349b538408cf165dfb5fd9d4d6ba51fe0df489e97993806aa161ea42fb45c1c Stored in directory: /tmp/pip-ephem-wheel-cache-r9zhbpxv/wheels/f1/2c/83/c09cb54cb0e821a8186cf5320758c27e7227ec862045210509 Building wheel for openshift (pyproject.toml): started Building wheel for openshift (pyproject.toml): finished with status 'done' Created wheel for openshift: filename=openshift-0.11.2-py3-none-any.whl size=19881 sha256=ad8d1ed7083c79a6e864be9805be66ffe5507f5b413a1b286b98e16e650cf335 Stored in directory: /alabama/.cache/pip/wheels/34/b7/02/4eb142942314b119c5fb3d4e595ac59486c1f3d79ff665397d Successfully built mtc openshift Installing collected packages: suds-py3, websocket-client, urllib3, six, setuptools, ruamel.yaml.clib, python-string-utils, pyasn1, oauthlib, MarkupSafe, idna, charset_normalizer, certifi, cachetools, ruamel.yaml, rsa, requests, python-dateutil, pyasn1-modules, jinja2, requests-oauthlib, google-auth, kubernetes, openshift, mtc Successfully installed MarkupSafe-3.0.3 cachetools-6.2.4 certifi-2025.11.12 charset_normalizer-3.4.4 google-auth-2.45.0 idna-3.11 jinja2-3.1.6 kubernetes-11.0.0 mtc-0.0.1 oauthlib-3.3.1 openshift-0.11.2 pyasn1-0.6.1 pyasn1-modules-0.4.2 python-dateutil-2.9.0.post0 python-string-utils-1.0.0 requests-2.32.5 requests-oauthlib-2.0.0 rsa-4.9.1 ruamel.yaml-0.18.17 ruamel.yaml.clib-0.2.15 setuptools-80.9.0 six-1.17.0 suds-py3-1.4.5.0 urllib3-2.6.2 websocket-client-1.9.0 [notice] A new release of pip is available: 23.3.2 -> 25.3 [notice] To update, run: pip install --upgrade pip go: downloading go1.24.1 (linux/amd64) go: downloading github.com/onsi/ginkgo/v2 v2.23.4 go: downloading github.com/onsi/gomega v1.36.3 go: downloading github.com/vmware-tanzu/velero v1.16.0 go: downloading github.com/openshift/oadp-operator v1.0.2-0.20250905181452-1d0b40b2ed49 go: downloading k8s.io/apimachinery v0.31.3 go: downloading k8s.io/client-go v0.31.3 go: downloading sigs.k8s.io/controller-runtime v0.19.3 go: downloading k8s.io/api v0.31.3 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/migtools/oadp-non-admin v0.0.0-20250409143544-08533a6c302d go: downloading github.com/aws/aws-sdk-go v1.44.253 go: downloading k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 go: downloading github.com/operator-framework/api v0.14.1-0.20220413143725-33310d6154f3 go: downloading github.com/kubernetes-csi/external-snapshotter/client/v4 v4.2.0 go: downloading github.com/andygrunwald/go-jira v1.16.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/apenella/go-ansible v1.1.5 go: downloading github.com/openshift/api v0.0.0-20230414143018-3367bc7e6ac7 go: downloading k8s.io/kubectl v0.30.5 go: downloading github.com/openshift/client-go v0.0.0-20211209144617-7385dd6338e3 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/go-logr/logr v1.4.2 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/gofuzz v1.2.0 go: downloading github.com/evanphx/json-patch/v5 v5.9.0 go: downloading github.com/google/go-cmp v0.7.0 go: downloading github.com/evanphx/json-patch v5.6.0+incompatible go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/spf13/pflag v1.0.6-0.20210604193023-d5e0c0615ace go: downloading k8s.io/klog/v2 v2.130.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading github.com/fatih/structs v1.1.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading github.com/google/go-querystring v1.1.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/trivago/tgo v1.0.7 go: downloading github.com/imdario/mergo v0.3.13 go: downloading golang.org/x/term v0.32.0 go: downloading golang.org/x/net v0.41.0 go: downloading github.com/gorilla/websocket v1.5.0 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/apenella/go-common-utils/data v0.0.0-20210528133155-34ba915e28c8 go: downloading github.com/apenella/go-common-utils/error v0.0.0-20210528133155-34ba915e28c8 go: downloading go.uber.org/automaxprocs v1.6.0 go: downloading golang.org/x/sys v0.33.0 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 go: downloading k8s.io/apiextensions-apiserver v0.31.3 go: downloading golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 go: downloading gopkg.in/evanphx/json-patch.v4 v4.12.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/spf13/cobra v1.8.1 go: downloading k8s.io/cli-runtime v0.31.3 go: downloading k8s.io/component-base v0.31.3 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/protobuf v1.36.5 go: downloading github.com/google/gnostic-models v0.6.8 go: downloading golang.org/x/time v0.9.0 go: downloading golang.org/x/oauth2 v0.27.0 go: downloading github.com/moby/spdystream v0.4.0 go: downloading github.com/kubernetes-csi/external-snapshotter/client/v7 v7.0.0 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/aws/aws-sdk-go-v2 v1.30.3 go: downloading github.com/aws/aws-sdk-go-v2/config v1.26.3 go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.15.11 go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.48.0 go: downloading github.com/go-task/slim-sprig/v3 v3.0.0 go: downloading golang.org/x/tools v0.33.0 go: downloading github.com/fxamacker/cbor/v2 v2.7.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/go-openapi/jsonreference v0.20.2 go: downloading github.com/go-openapi/swag v0.22.4 go: downloading golang.org/x/text v0.26.0 go: downloading github.com/inconshreveable/mousetrap v1.1.0 go: downloading github.com/jonboulle/clockwork v0.2.2 go: downloading k8s.io/component-helpers v0.30.5 go: downloading github.com/daviddengcn/go-colortext v1.0.0 go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de go: downloading github.com/distribution/reference v0.5.0 go: downloading github.com/moby/term v0.5.0 go: downloading github.com/fvbommel/sortorder v1.1.0 go: downloading sigs.k8s.io/kustomize/kustomize/v5 v5.0.4-0.20230601165947-6ce0bf390ce3 go: downloading sigs.k8s.io/kustomize/kyaml v0.17.1 go: downloading github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d go: downloading github.com/lithammer/dedent v1.1.0 go: downloading k8s.io/metrics v0.31.3 go: downloading github.com/chai2010/gettext-go v1.0.2 go: downloading github.com/MakeNowJust/heredoc v1.0.0 go: downloading github.com/mitchellh/go-wordwrap v1.0.1 go: downloading github.com/russross/blackfriday/v2 v2.1.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/stretchr/testify v1.10.0 go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c go: downloading github.com/go-logr/zapr v1.3.0 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f go: downloading github.com/aws/smithy-go v1.20.3 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.17.26 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.22.3 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.4 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.30.3 go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.4 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.15 go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.2.10 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.3 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.2.10 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.17 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.16.10 go: downloading github.com/spf13/afero v1.10.0 go: downloading github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 go: downloading github.com/go-openapi/jsonpointer v0.19.6 go: downloading github.com/x448/float16 v0.8.4 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading golang.org/x/sync v0.15.0 go: downloading sigs.k8s.io/kustomize/api v0.17.2 go: downloading github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 go: downloading github.com/peterbourgon/diskv v2.0.1+incompatible go: downloading github.com/fatih/camelcase v1.0.0 go: downloading github.com/golangplus/testing v1.0.0 go: downloading github.com/opencontainers/go-digest v1.0.0 go: downloading github.com/creack/pty v1.1.18 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/emicklei/go-restful/v3 v3.11.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.15 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/prashantv/gostub v1.1.0 go: downloading github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/go-errors/errors v1.4.2 go: downloading github.com/google/btree v1.0.1 go: downloading gomodules.xyz/jsonpatch/v2 v2.4.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/sergi/go-diff v1.2.0 go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 go: downloading github.com/xlab/treeprint v1.2.0 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading go.starlark.net v0.0.0-20230525235612-a134d8f9ddca go: downloading github.com/prometheus/common v0.62.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/klauspost/compress v1.17.11 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/jmespath/go-jmespath/internal/testify v1.5.1 storageclass.storage.k8s.io/gp2-csi annotated storageclass.storage.k8s.io/gp3-csi annotated storageclass.storage.k8s.io/odf-operator-ceph-rbd annotated storageclass.storage.k8s.io/odf-operator-ceph-rbd-virtualization annotated storageclass.storage.k8s.io/odf-operator-cephfs annotated storageclass.storage.k8s.io/openshift-storage.noobaa.io annotated storageclass.storage.k8s.io/odf-operator-ceph-rbd annotated + readonly 'RED=\e[31m' + RED='\e[31m' + readonly 'BLUE=\033[34m' + BLUE='\033[34m' + readonly 'CLEAR=\e[39m' + CLEAR='\e[39m' ++ oc get infrastructures cluster -o 'jsonpath={.status.platform}' ++ awk '{print tolower($0)}' + CLOUD_PROVIDER=aws + [[ '' == \t\r\u\e ]] + echo /home/jenkins/.kube/config /home/jenkins/.kube/config + [[ aws == *-arm* ]] + [[ aws == *-fips* ]] + E2E_TIMEOUT_MULTIPLIER=2 + export NAMESPACE=openshift-adp + NAMESPACE=openshift-adp + export PROVIDER=aws + PROVIDER=aws ++ echo aws ++ awk '{print tolower($0)}' + BACKUP_LOCATION=aws + export BACKUP_LOCATION=aws + BACKUP_LOCATION=aws + export BUCKET=ci-op-2jmljgmw-interopoadp + BUCKET=ci-op-2jmljgmw-interopoadp + OADP_CREDS_FILE=/tmp/test-settings/credentials + OADP_VSL_CREDS_FILE=/tmp/test-settings/aws_vsl_creds +++ readlink -f /alabama/cspi/test_settings/scripts/test_runner.sh ++ dirname /alabama/cspi/test_settings/scripts/test_runner.sh + readonly SCRIPT_DIR=/alabama/cspi/test_settings/scripts + SCRIPT_DIR=/alabama/cspi/test_settings/scripts ++ cd /alabama/cspi/test_settings/scripts ++ git rev-parse --show-toplevel + readonly TOP_DIR=/alabama/cspi + TOP_DIR=/alabama/cspi + echo /alabama/cspi /alabama/cspi + TESTS_FOLDER=/alabama/cspi/e2e/kubevirt-plugin ++ oc get nodes -o 'jsonpath={.items[*].metadata.labels.kubernetes\.io/arch}' ++ tr ' ' '\n' ++ sort -u ++ xargs + export NODES_ARCHITECTURE=amd64 + NODES_ARCHITECTURE=amd64 + export OADP_REPOSITORY=redhat + OADP_REPOSITORY=redhat + SKIP_DPA_CREATION=false ++ oc get ns openshift-storage ++ echo true + OPENSHIFT_STORAGE=true + '[' redhat == upstream-velero ']' + '[' true == true ']' ++ oc get sc ++ awk '$1 ~ /^.+ceph-rbd$/ {print $1}' ++ tail -1 + CEPH_RBD_STORAGE_CLASS=odf-operator-ceph-rbd + '[' -n odf-operator-ceph-rbd ']' + export CEPH_RBD_STORAGE_CLASS + echo 'ceph-rbd StorageClass found: odf-operator-ceph-rbd' ceph-rbd StorageClass found: odf-operator-ceph-rbd ++ oc get storageclass -o 'jsonpath={range .items[*]}{@.metadata.name}{" "}{@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}{end}' ++ awk '$2=="true"{print $1}' ++ wc -l + NUM_DEFAULT_STORAGE_CLASS=1 + '[' 1 -ne 1 ']' ++ oc get storageclass -o 'jsonpath={.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=='\''true'\'')].metadata.name}' + DEFAULT_SC=odf-operator-ceph-rbd + export STORAGE_CLASS=odf-operator-ceph-rbd + STORAGE_CLASS=odf-operator-ceph-rbd + '[' -n odf-operator-ceph-rbd ']' + '[' odf-operator-ceph-rbd '!=' odf-operator-ceph-rbd ']' + export STORAGE_CLASS_OPENSHIFT_STORAGE=odf-operator-ceph-rbd + STORAGE_CLASS_OPENSHIFT_STORAGE=odf-operator-ceph-rbd + echo 'Using the StorageClass for openshift-storage: odf-operator-ceph-rbd' Using the StorageClass for openshift-storage: odf-operator-ceph-rbd + [[ amd64 != \a\m\d\6\4 ]] + TEST_FILTER='!// || (// && !exclude_aws && (!/target/ || target_aws) ) ' + [[ aws =~ ^(osp|vsphere|gcp-wif|ibmcloud|azure-sts|aro-4) ]] + [[ aws =~ ^azure_sak ]] ++ oc config current-context ++ awk -F / '{print $2}' + SETTINGS_TMP=/alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443 + mkdir -p /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443 ++ oc get authentication cluster -o 'jsonpath={.spec.serviceAccountIssuer}' + IS_OIDC= + '[' '!' -z ']' + [[ aws == \a\w\s ]] + export PROVIDER=aws + PROVIDER=aws + export CREDS_SECRET_REF=cloud-credentials + CREDS_SECRET_REF=cloud-credentials ++ oc get infrastructures cluster -o 'jsonpath={.status.platformStatus.aws.region}' --allow-missing-template-keys=false + export REGION=us-east-2 + REGION=us-east-2 + settings_script=aws_settings.sh + '[' aws == aws-sts ']' + BUCKET=ci-op-2jmljgmw-interopoadp + TMP_DIR=/alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443 + source /alabama/cspi/test_settings/scripts/aws_settings.sh ++ cat ++ [[ aws == *aws* ]] ++ cat ++ echo -e '\n }\n}' +++ cat /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json ++ x='{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-2" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-2jmljgmw-interopoadp" } } } ] , "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-2" } } } ] } }' ++ echo '{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-2" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-2jmljgmw-interopoadp" } } } ] , "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-2" } } } ] } }' ++ grep -o '^[^#]*' + FILE_SETTINGS_NAME=settings.json + printf '\033[34mGenerated settings file under /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json\e[39m\n' Generated settings file under /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json + cat /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json ++ oc get volumesnapshotclass -o name + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc annotated + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/odf-operator-cephfsplugin-snapclass snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/odf-operator-cephfsplugin-snapclass annotated + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/odf-operator-rbdplugin-snapclass snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/odf-operator-rbdplugin-snapclass annotated ++ ./e2e/must-gather/get-latest-build.sh + oc get configmaps -n default must-gather-image + UPSTREAM_VERSION=99.0.0 ++ oc get OperatorCondition -n openshift-adp -o 'jsonpath={.items[*].metadata.name}' ++ awk -F v '{print $2}' + OADP_VERSION=1.5.3 + '[' -z 1.5.3 ']' + '[' 1.5.3 == 99.0.0 ']' ++ oc get sub redhat-oadp-operator -n openshift-adp -o 'jsonpath={.spec.source}' + OADP_REPO=redhat-operators + '[' -z redhat-operators ']' + '[' redhat-operators == redhat-operators ']' + REGISTRY_PATH=registry.redhat.io/oadp/oadp-mustgather-rhel9: + TAG=1.5.3 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + echo registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + '[' -z registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 ']' + export NUM_OF_OADP_INSTANCES=1 + NUM_OF_OADP_INSTANCES=1 ++ echo --skip=tc-id:OADP-555 ++ tr ' ' '\n' ++ grep '^--' ++ tr '\n' ' ' + GINKO_PARAM='--skip=tc-id:OADP-555 ' ++ echo --skip=tc-id:OADP-555 ++ tr ' ' '\n' ++ grep '^-' ++ grep -v '^--' ++ tr '\n' ' ' + TEST_PARAM= + ginkgo run --nodes=1 -mod=mod --show-node-events --flake-attempts 3 --junit-report=/logs/artifacts/junit_oadp_cnv_results.xml '--label-filter=!// || (// && !exclude_aws && (!/target/ || target_aws) ) ' --skip=tc-id:OADP-555 -p /alabama/cspi/e2e/kubevirt-plugin/ -- -credentials_file=/tmp/test-settings/credentials -vsl_credentials_file=/tmp/test-settings/aws_vsl_creds -oadp_namespace=openshift-adp -settings=/alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json -must_gather_image=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 -timeout_multiplier=2 -skip_dpa_creation=false Ginkgo detected a version mismatch between the Ginkgo CLI and the version of Ginkgo imported by your packages: Ginkgo CLI Version: 2.27.3 Mismatched package versions found: 2.23.4 used by kubevirt-plugin Ginkgo will continue to attempt to run but you may see errors (including flag parsing errors) and should either update your go.mod or your version of the Ginkgo CLI to match. To install the matching version of the CLI run go install github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file. Alternatively you can use go run github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file to invoke the matching version of the Ginkgo CLI. If you are attempting to test multiple packages that each have a different version of the Ginkgo library with a single Ginkgo CLI that is currently unsupported.  2025/12/20 16:30:20 Setting up clients 2025/12/20 16:30:20 Getting default StorageClass... 2025/12/20 16:30:20 Checking default storage class count Run the command: oc get sc 2025/12/20 16:30:20 Got default StorageClass odf-operator-ceph-rbd 2025/12/20 16:30:21 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 53m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 53m odf-operator-ceph-rbd (default) openshift-storage.rbd.csi.ceph.com Delete Immediate true 5m2s odf-operator-ceph-rbd-virtualization openshift-storage.rbd.csi.ceph.com Delete Immediate true 5m2s odf-operator-cephfs openshift-storage.cephfs.csi.ceph.com Delete Immediate true 5m2s openshift-storage.noobaa.io openshift-storage.noobaa.io/obc Delete Immediate false 2m24s 2025/12/20 16:30:21 Using velero prefix: velero-e2e-kubevirt-2d8ca307-ddc1-11f0-a23b-0a580a804e0c Running Suite: OADP E2E Virtualization Workloads Suite - /alabama/cspi/e2e/kubevirt-plugin ========================================================================================== Random Seed: 1766248184 Will run 4 of 5 specs ------------------------------ [BeforeSuite]  /alabama/cspi/e2e/kubevirt-plugin/kubevirt_suite_test.go:62 > Enter [BeforeSuite] TOP-LEVEL @ 12/20/25 16:30:21.111 < Exit [BeforeSuite] TOP-LEVEL @ 12/20/25 16:30:21.175 (64ms) [BeforeSuite] PASSED [0.064 seconds] ------------------------------ CSI: Backup/Restore Openshift Virtualization Workloads  [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed /alabama/cspi/e2e/kubevirt-plugin/backup_restore_csi.go:34 > Enter [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:30:21.176 < Exit [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:30:21.193 (17ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:30:21.193 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:30:21.193 (0s) > Enter [It] [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed @ 12/20/25 16:30:21.193 2025/12/20 16:30:21 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:30:22.237 2025/12/20 16:30:22 csi 2025/12/20 16:30:22 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "7232cbf1-f1c6-4e70-aa66-df58a19d3015", "resourceVersion": "63694", "generation": 1, "creationTimestamp": "2025-12-20T16:30:22Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:30:22Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:30:24.377 2025/12/20 16:30:24 Waiting for velero pod to be running 2025/12/20 16:30:24 pod: velero-5944c96584-crtfd is not yet running with status: {Pending [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotInitialized containers with incomplete status: [openshift-velero-plugin velero-plugin-for-aws kubevirt-velero-plugin hypershift-oadp-plugin]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] [] 2025-12-20 16:30:22 +0000 UTC [{openshift-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a 0xc000c66d7a map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000112ab0}] nil []} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 0xc000c66db8 map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000112ac0}] nil []} {kubevirt-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 0xc000c66def map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000112ad0}] nil []} {hypershift-oadp-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 0xc000c66e08 map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000112ae0}] nil []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc000c66e36 map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc000112af0} {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000112b10}] nil []}] Burstable [] []} 2025/12/20 16:30:29 pod: velero-5944c96584-crtfd is not yet running with status: {Pending [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotInitialized containers with incomplete status: [openshift-velero-plugin velero-plugin-for-aws kubevirt-velero-plugin hypershift-oadp-plugin]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] [] 2025-12-20 16:30:22 +0000 UTC [{openshift-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a 0xc000b27b6a map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000438800}] nil []} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 0xc000b27c18 map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000438810}] nil []} {kubevirt-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 0xc000b27c5f map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000438820}] nil []} {hypershift-oadp-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 0xc000b27c88 map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000438830}] nil []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc000b27cb6 map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc000438840} {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000438860}] nil []}] Burstable [] []} 2025/12/20 16:30:34 pod: velero-5944c96584-crtfd is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:29 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotInitialized containers with incomplete status: [velero-plugin-for-aws kubevirt-velero-plugin hypershift-oadp-plugin]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] 10.128.2.80 [{10.128.2.80}] 2025-12-20 16:30:22 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:29 +0000 UTC,FinishedAt:2025-12-20 16:30:29 +0000 UTC,ContainerID:cri-o://ebebffe991c47e6fa246329b6da5d1f405373a29aa7cc9cda07034abf5f9318b,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:212589b9f5b75a043c9a7094cb681a39687795c5e78e625beb6d00960ea57e70 cri-o://ebebffe991c47e6fa246329b6da5d1f405373a29aa7cc9cda07034abf5f9318b 0xc000f18929 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000439910}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 0xc000f18988 map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000439920}] nil []} {kubevirt-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 0xc000f189df map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000439930}] nil []} {hypershift-oadp-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 0xc000f189f8 map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000439940}] nil []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc000f18a26 map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc000439950} {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000439960}] nil []}] Burstable [] []} 2025/12/20 16:30:39 pod: velero-5944c96584-crtfd is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:29 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotInitialized containers with incomplete status: [kubevirt-velero-plugin hypershift-oadp-plugin]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] 10.128.2.80 [{10.128.2.80}] 2025-12-20 16:30:22 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:29 +0000 UTC,FinishedAt:2025-12-20 16:30:29 +0000 UTC,ContainerID:cri-o://ebebffe991c47e6fa246329b6da5d1f405373a29aa7cc9cda07034abf5f9318b,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:212589b9f5b75a043c9a7094cb681a39687795c5e78e625beb6d00960ea57e70 cri-o://ebebffe991c47e6fa246329b6da5d1f405373a29aa7cc9cda07034abf5f9318b 0xc000f19399 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000071040}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:35 +0000 UTC,FinishedAt:2025-12-20 16:30:35 +0000 UTC,ContainerID:cri-o://6c7378d4054549e1912e3eef511ea181c08e93fcedf2572e08fecd480f1b1312,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 cri-o://6c7378d4054549e1912e3eef511ea181c08e93fcedf2572e08fecd480f1b1312 0xc000f193f8 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000071480}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {kubevirt-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 0xc000f1948a map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000071490}] nil []} {hypershift-oadp-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 0xc000f194c8 map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc0000714a0}] nil []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc000f194f6 map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc0000714b0} {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000071bb0}] nil []}] Burstable [] []} 2025/12/20 16:30:44 pod: velero-5944c96584-crtfd is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:29 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotInitialized containers with incomplete status: [hypershift-oadp-plugin]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] 10.128.2.80 [{10.128.2.80}] 2025-12-20 16:30:22 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:29 +0000 UTC,FinishedAt:2025-12-20 16:30:29 +0000 UTC,ContainerID:cri-o://ebebffe991c47e6fa246329b6da5d1f405373a29aa7cc9cda07034abf5f9318b,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:212589b9f5b75a043c9a7094cb681a39687795c5e78e625beb6d00960ea57e70 cri-o://ebebffe991c47e6fa246329b6da5d1f405373a29aa7cc9cda07034abf5f9318b 0xc000c82729 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028c6e0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:35 +0000 UTC,FinishedAt:2025-12-20 16:30:35 +0000 UTC,ContainerID:cri-o://6c7378d4054549e1912e3eef511ea181c08e93fcedf2572e08fecd480f1b1312,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 cri-o://6c7378d4054549e1912e3eef511ea181c08e93fcedf2572e08fecd480f1b1312 0xc000c82788 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028c750}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:42 +0000 UTC,FinishedAt:2025-12-20 16:30:42 +0000 UTC,ContainerID:cri-o://01e1887d4df636eec3680223cccfc67efb2347e84332b60e3e725b6cb967cd18,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:684020e409da37fd3a41935b51ac22e506fb7a45e787ae8c0cfd6ac0dfcbff09 cri-o://01e1887d4df636eec3680223cccfc67efb2347e84332b60e3e725b6cb967cd18 0xc000c82839 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028c7d0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {hypershift-oadp-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 0xc000c82898 map[] nil [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028c7e0}] nil []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc000c828e6 map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc00028c7f0} {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028c800}] nil []}] Burstable [] []} 2025/12/20 16:30:49 pod: velero-5944c96584-crtfd is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:29 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:48 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:30:22 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] 10.128.2.80 [{10.128.2.80}] 2025-12-20 16:30:22 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:29 +0000 UTC,FinishedAt:2025-12-20 16:30:29 +0000 UTC,ContainerID:cri-o://ebebffe991c47e6fa246329b6da5d1f405373a29aa7cc9cda07034abf5f9318b,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:212589b9f5b75a043c9a7094cb681a39687795c5e78e625beb6d00960ea57e70 cri-o://ebebffe991c47e6fa246329b6da5d1f405373a29aa7cc9cda07034abf5f9318b 0xc000c83349 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028d4c0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:35 +0000 UTC,FinishedAt:2025-12-20 16:30:35 +0000 UTC,ContainerID:cri-o://6c7378d4054549e1912e3eef511ea181c08e93fcedf2572e08fecd480f1b1312,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 cri-o://6c7378d4054549e1912e3eef511ea181c08e93fcedf2572e08fecd480f1b1312 0xc000c833a8 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028d540}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:42 +0000 UTC,FinishedAt:2025-12-20 16:30:42 +0000 UTC,ContainerID:cri-o://01e1887d4df636eec3680223cccfc67efb2347e84332b60e3e725b6cb967cd18,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:684020e409da37fd3a41935b51ac22e506fb7a45e787ae8c0cfd6ac0dfcbff09 cri-o://01e1887d4df636eec3680223cccfc67efb2347e84332b60e3e725b6cb967cd18 0xc000c83459 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028d5b0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {hypershift-oadp-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:30:47 +0000 UTC,FinishedAt:2025-12-20 16:30:47 +0000 UTC,ContainerID:cri-o://f5c997fcdf95946ff09003f2986a2ff0069d10d9a2b21504e72c901a2b9a8e80,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:5d9c2ff91b5851f486fdcadcf9991007cdbe7a81f57774c6d0b3798091316af6 cri-o://f5c997fcdf95946ff09003f2986a2ff0069d10d9a2b21504e72c901a2b9a8e80 0xc000c834b8 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028d630}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc000c8356f map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc00028d660} {kube-api-access-blmgh /var/run/secrets/kubernetes.io/serviceaccount true 0xc00028d670}] nil []}] Burstable [] []} 2025/12/20 16:30:54 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:30:54 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:30:54.599 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2025/12/20 16:30:54 The 'openshift-storage' namespace exists 2025/12/20 16:30:54 Checking default storage class count 2025/12/20 16:30:54 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2025/12/20 16:30:54 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:30:54 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:30:54 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case ocp-kubevirt @ 12/20/25 16:30:54.995 2025/12/20 16:30:54 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:30:54 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: WorkerProcess for [localhost/TASK: Extract Kubernetes minor version from cluster] errantly sent data directly to stderr instead of using Display:  /usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ci-op-2jmljgmw-2c502.cspilp.interop.ccitredhat.com'. Adding certificate verification is strongly advised. Se  [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:30:58 2025-12-20 16:30:56,712 p=19994 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:30:56,712 p=19994 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:30:57,036 p=19994 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:30:57,037 p=19994 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:30:57,363 p=19994 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:30:57,363 p=19994 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:30:57,886 p=19994 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:30:57,886 p=19994 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:30:57,904 p=19994 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:30:57,904 p=19994 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:30:57,904 p=19994 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:30:57,923 p=19994 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:30:57,923 p=19994 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:30:57,938 p=19994 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:30:57,939 p=19994 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:30:58,454 p=19994 u=1001620000 n=ansible WARNING| [WARNING]: WorkerProcess for [localhost/TASK: Extract Kubernetes minor version from cluster] errantly sent data directly to stderr instead of using Display: /usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ci-op-2jmljgmw-2c502.cspilp.interop.ccitredhat.com'. Adding certificate verification is strongly advised. Se 2025-12-20 16:30:58,456 p=19994 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:30:58,456 p=19994 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:30:58,456 p=19994 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:30:58,456 p=19994 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:30:58,480 p=19994 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:30:58,481 p=19994 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:30:58,481 p=19994 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:30:58,482 p=19994 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:30:58,482 p=19994 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-185 2025/12/20 16:30:58 2025/12/20 16:30:58 Found server error pattern in ansible logs: '502' 2025/12/20 16:30:58 Detected server errors in ansible logs - treating as retryable 2025/12/20 16:30:58 ⚠️ Operation 'Deploy application ocp-kubevirt' failed with retryable server error (attempt 1/3): Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-185","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_running":true,"with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 2025/12/20 16:30:58 🕐 Waiting 1m0s before retry... 2025/12/20 16:31:58 🔄 [Attempt 2/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:31:58 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:32:01 2025-12-20 16:32:00,358 p=20174 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:32:00,358 p=20174 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:00,687 p=20174 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:32:00,688 p=20174 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:01,006 p=20174 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:32:01,007 p=20174 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:01,333 p=20174 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:32:01,334 p=20174 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:01,351 p=20174 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:32:01,351 p=20174 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:32:01,351 p=20174 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:01,370 p=20174 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:32:01,370 p=20174 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:01,385 p=20174 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:32:01,386 p=20174 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:32:01,768 p=20174 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:32:01,768 p=20174 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:32:01,769 p=20174 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:32:01,769 p=20174 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:01,792 p=20174 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:32:01,792 p=20174 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:32:01,793 p=20174 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:32:01,793 p=20174 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:32:01,793 p=20174 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-185 2025/12/20 16:32:02 2025/12/20 16:32:02 ❌ Operation 'Deploy application ocp-kubevirt' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-185","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_running":true,"with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:32:02.027 < Exit [It] [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed @ 12/20/25 16:32:02.027 (1m40.834s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:32:02.027 2025/12/20 16:32:02 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:32:02.027 2025/12/20 16:32:02 The failed spec name is: CSI: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:32:02.027 2025/12/20 16:32:02 The folder logs does not exists, creating new folder with the name: logs STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:32:02.027 2025/12/20 16:32:02 The folder logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed does not exists, creating new folder with the name: logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed STEP: Run must-gather because the spec failed @ 12/20/25 16:32:02.027 2025/12/20 16:32:02 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:32:02 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:32:30 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:32:30 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:32:30.595 < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:32:30.595 (28.568s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:32:30.595 2025/12/20 16:32:30 Cleaning app 2025/12/20 16:32:30 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:32:33 2025-12-20 16:32:32,272 p=20397 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:32:32,272 p=20397 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:32,587 p=20397 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:32:32,587 p=20397 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:32,910 p=20397 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:32:32,910 p=20397 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:33,224 p=20397 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:32:33,225 p=20397 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:33,242 p=20397 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:32:33,242 p=20397 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:32:33,242 p=20397 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:33,261 p=20397 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:32:33,262 p=20397 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:33,276 p=20397 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:32:33,276 p=20397 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:32:33,653 p=20397 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:32:33,654 p=20397 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:32:33,654 p=20397 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:32:33,654 p=20397 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:33,676 p=20397 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:32:33,676 p=20397 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:32:33,677 p=20397 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:32:33,677 p=20397 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:32:33,677 p=20397 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-185 2025/12/20 16:32:33 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:32:33.884 (3.289s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:32:33.884 2025/12/20 16:32:33 Cleaning setup resources for the backup 2025/12/20 16:32:33 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:32:33 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:32:34 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:32:34.064 (180ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:32:34.064 2025/12/20 16:32:34 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:32:34.125 (61ms) Attempt #1 Failed. Retrying ↺ @ 12/20/25 16:32:34.125 > Enter [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:32:34.125 < Exit [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:32:34.155 (30ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:32:34.155 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:32:34.155 (0s) > Enter [It] [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed @ 12/20/25 16:32:34.155 2025/12/20 16:32:34 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:32:35.221 2025/12/20 16:32:35 csi 2025/12/20 16:32:35 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "bccdbaa5-7215-4b4e-8fde-2877d6396d40", "resourceVersion": "66513", "generation": 1, "creationTimestamp": "2025-12-20T16:32:35Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:32:35Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:32:37.353 2025/12/20 16:32:37 Waiting for velero pod to be running 2025/12/20 16:32:37 pod: velero-5944c96584-k44dm is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:32:37 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:32:35 +0000 UTC ContainersNotInitialized containers with incomplete status: [velero-plugin-for-aws kubevirt-velero-plugin hypershift-oadp-plugin]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:32:35 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:32:35 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:32:35 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] 10.128.2.83 [{10.128.2.83}] 2025-12-20 16:32:35 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:32:36 +0000 UTC,FinishedAt:2025-12-20 16:32:36 +0000 UTC,ContainerID:cri-o://637ddbc0e99656a442756586a3c9e5cf08081cbed341503d6d1c3b47b1e5a397,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:212589b9f5b75a043c9a7094cb681a39687795c5e78e625beb6d00960ea57e70 cri-o://637ddbc0e99656a442756586a3c9e5cf08081cbed341503d6d1c3b47b1e5a397 0xc001010969 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-ss4mh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000ba6c00}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 0xc0010109c8 map[] nil [{plugins /target false } {kube-api-access-ss4mh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000ba6c10}] nil []} {kubevirt-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 0xc001010a1f map[] nil [{plugins /target false } {kube-api-access-ss4mh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000ba6c20}] nil []} {hypershift-oadp-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 0xc001010a38 map[] nil [{plugins /target false } {kube-api-access-ss4mh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000ba6c30}] nil []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc001010a66 map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc000ba6c40} {kube-api-access-ss4mh /var/run/secrets/kubernetes.io/serviceaccount true 0xc000ba6c50}] nil []}] Burstable [] []} 2025/12/20 16:32:42 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:32:42 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:32:42.434 2025/12/20 16:32:42 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:32:42 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:32:42 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case ocp-kubevirt @ 12/20/25 16:32:42.583 2025/12/20 16:32:42 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:32:42 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:32:45 2025-12-20 16:32:44,262 p=20576 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:32:44,262 p=20576 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:44,566 p=20576 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:32:44,566 p=20576 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:44,867 p=20576 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:32:44,867 p=20576 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:45,168 p=20576 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:32:45,168 p=20576 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:45,185 p=20576 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:32:45,185 p=20576 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:32:45,185 p=20576 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:45,204 p=20576 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:32:45,205 p=20576 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:45,218 p=20576 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:32:45,219 p=20576 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:32:45,596 p=20576 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:32:45,597 p=20576 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:32:45,597 p=20576 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:32:45,597 p=20576 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:45,619 p=20576 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:32:45,619 p=20576 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:32:45,620 p=20576 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:32:45,620 p=20576 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:32:45,620 p=20576 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-185 2025/12/20 16:32:45 2025/12/20 16:32:45 ❌ Operation 'Deploy application ocp-kubevirt' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-185","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_running":true,"with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:32:45.83 < Exit [It] [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed @ 12/20/25 16:32:45.831 (11.676s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:32:45.831 2025/12/20 16:32:45 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:32:45.831 2025/12/20 16:32:45 The failed spec name is: CSI: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:32:45.831 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:32:45.831 STEP: Run must-gather because the spec failed @ 12/20/25 16:32:45.831 2025/12/20 16:32:45 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:32:45 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:32:56 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:32:56 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:32:56.893 The folder logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed/must-gather already exists, skipping renaming the folder < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:32:56.893 (11.063s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:32:56.893 2025/12/20 16:32:56 Cleaning app 2025/12/20 16:32:56 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:32:59 2025-12-20 16:32:58,538 p=20800 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:32:58,539 p=20800 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:58,836 p=20800 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:32:58,836 p=20800 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:59,139 p=20800 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:32:59,139 p=20800 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:59,444 p=20800 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:32:59,444 p=20800 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:32:59,462 p=20800 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:32:59,462 p=20800 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:32:59,462 p=20800 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:59,481 p=20800 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:32:59,481 p=20800 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:59,495 p=20800 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:32:59,496 p=20800 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:32:59,871 p=20800 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:32:59,871 p=20800 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:32:59,871 p=20800 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:32:59,872 p=20800 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:32:59,893 p=20800 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:32:59,894 p=20800 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:32:59,894 p=20800 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:32:59,895 p=20800 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:32:59,895 p=20800 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-185 2025/12/20 16:33:00 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:00.09 (3.196s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:00.09 2025/12/20 16:33:00 Cleaning setup resources for the backup 2025/12/20 16:33:00 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:33:00 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:33:00 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:00.16 (70ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:00.16 2025/12/20 16:33:00 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:00.183 (23ms) Attempt #2 Failed. Retrying ↺ @ 12/20/25 16:33:00.183 > Enter [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:33:00.183 < Exit [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:33:00.207 (24ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:33:00.207 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:33:00.207 (0s) > Enter [It] [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed @ 12/20/25 16:33:00.207 2025/12/20 16:33:00 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:33:00.228 2025/12/20 16:33:00 csi 2025/12/20 16:33:00 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "79850c40-1dde-4ea6-9095-d7ae7953573e", "resourceVersion": "67003", "generation": 1, "creationTimestamp": "2025-12-20T16:33:00Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:33:00Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:33:00.329 2025/12/20 16:33:00 Waiting for velero pod to be running 2025/12/20 16:33:00 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:33:00 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "79850c40-1dde-4ea6-9095-d7ae7953573e", "resourceVersion": "67003", "generation": 1, "creationTimestamp": "2025-12-20T16:33:00Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:33:00Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:33:05 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:33:05.385 2025/12/20 16:33:05 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:33:05 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:33:05 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case ocp-kubevirt @ 12/20/25 16:33:05.598 2025/12/20 16:33:05 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:33:05 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:33:08 2025-12-20 16:33:07,236 p=20980 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:33:07,236 p=20980 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:07,539 p=20980 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:33:07,539 p=20980 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:07,843 p=20980 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:33:07,844 p=20980 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:08,146 p=20980 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:33:08,147 p=20980 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:08,164 p=20980 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:33:08,164 p=20980 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:33:08,164 p=20980 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:08,183 p=20980 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:33:08,183 p=20980 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:08,198 p=20980 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:33:08,198 p=20980 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:33:08,573 p=20980 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:33:08,573 p=20980 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:33:08,574 p=20980 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:33:08,574 p=20980 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:08,595 p=20980 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:33:08,596 p=20980 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:33:08,596 p=20980 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:33:08,597 p=20980 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:33:08,597 p=20980 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-185 2025/12/20 16:33:08 2025/12/20 16:33:08 ❌ Operation 'Deploy application ocp-kubevirt' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-185","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_running":true,"with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:33:08.794 < Exit [It] [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed @ 12/20/25 16:33:08.794 (8.587s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:33:08.794 2025/12/20 16:33:08 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:33:08.794 2025/12/20 16:33:08 The failed spec name is: CSI: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:33:08.794 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:33:08.794 STEP: Run must-gather because the spec failed @ 12/20/25 16:33:08.794 2025/12/20 16:33:08 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:33:08 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:33:26 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:33:26 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:33:26.55 The folder logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed/must-gather already exists, skipping renaming the folder < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:33:26.55 (17.756s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:26.55 2025/12/20 16:33:26 Cleaning app 2025/12/20 16:33:26 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:33:29 2025-12-20 16:33:28,204 p=21204 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:33:28,205 p=21204 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:28,511 p=21204 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:33:28,511 p=21204 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:28,815 p=21204 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:33:28,815 p=21204 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:29,118 p=21204 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:33:29,119 p=21204 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:29,136 p=21204 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:33:29,136 p=21204 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:33:29,136 p=21204 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:29,155 p=21204 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:33:29,156 p=21204 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:29,170 p=21204 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:33:29,171 p=21204 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:33:29,547 p=21204 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:33:29,547 p=21204 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:33:29,547 p=21204 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:33:29,548 p=21204 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:29,569 p=21204 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:33:29,570 p=21204 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:33:29,570 p=21204 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:33:29,570 p=21204 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:33:29,571 p=21204 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-185 2025/12/20 16:33:29 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:29.768 (3.218s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:29.768 2025/12/20 16:33:29 Cleaning setup resources for the backup 2025/12/20 16:33:29 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:33:29 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:33:29 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:29.834 (67ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:29.834 2025/12/20 16:33:29 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:29.861 (27ms) • [FAILED] [188.686 seconds] CSI: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed /alabama/cspi/e2e/kubevirt-plugin/backup_restore_csi.go:34 [FAILED] Unexpected error: <*errors.Error | 0xc000ca1c40>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-185","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_running":true,"with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"test-oadp-185\",\"non_admin_user\":false,\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"vm_running\":true,\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:33:08.794 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ CSI: Backup/Restore Openshift Virtualization Workloads  [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored /alabama/cspi/e2e/kubevirt-plugin/backup_restore_csi.go:51 > Enter [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:33:29.861 < Exit [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:33:29.886 (24ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:33:29.886 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:33:29.886 (0s) > Enter [It] [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored @ 12/20/25 16:33:29.886 2025/12/20 16:33:29 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:33:29.907 2025/12/20 16:33:29 csi 2025/12/20 16:33:29 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "43d0dab5-bc8f-4f27-9b97-bdec23215071", "resourceVersion": "67591", "generation": 1, "creationTimestamp": "2025-12-20T16:33:29Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:33:29Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:33:30 2025/12/20 16:33:30 Waiting for velero pod to be running 2025/12/20 16:33:30 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:33:30 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "43d0dab5-bc8f-4f27-9b97-bdec23215071", "resourceVersion": "67591", "generation": 1, "creationTimestamp": "2025-12-20T16:33:29Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:33:29Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:33:35 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:33:35.055 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2025/12/20 16:33:35 The 'openshift-storage' namespace exists 2025/12/20 16:33:35 Checking default storage class count 2025/12/20 16:33:35 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2025/12/20 16:33:35 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:33:35 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:33:35 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case ocp-kubevirt @ 12/20/25 16:33:35.559 2025/12/20 16:33:35 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:33:35 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:33:38 2025-12-20 16:33:37,196 p=21407 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:33:37,196 p=21407 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:37,495 p=21407 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:33:37,495 p=21407 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:37,796 p=21407 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:33:37,796 p=21407 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:38,095 p=21407 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:33:38,095 p=21407 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:38,113 p=21407 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:33:38,113 p=21407 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:33:38,113 p=21407 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:38,133 p=21407 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:33:38,133 p=21407 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:38,147 p=21407 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:33:38,148 p=21407 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:33:38,531 p=21407 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:33:38,531 p=21407 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:33:38,531 p=21407 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:33:38,532 p=21407 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:38,553 p=21407 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:33:38,553 p=21407 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:33:38,554 p=21407 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:33:38,554 p=21407 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:33:38,554 p=21407 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-186 2025/12/20 16:33:38 2025/12/20 16:33:38 ❌ Operation 'Deploy application ocp-kubevirt' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-186","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_running":false,"with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:33:38.757 < Exit [It] [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored @ 12/20/25 16:33:38.757 (8.871s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:33:38.757 2025/12/20 16:33:38 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:33:38.757 2025/12/20 16:33:38 The failed spec name is: CSI: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:33:38.757 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:33:38.757 2025/12/20 16:33:38 The folder logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored does not exists, creating new folder with the name: logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored STEP: Run must-gather because the spec failed @ 12/20/25 16:33:38.757 2025/12/20 16:33:38 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:33:38 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:33:49 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:33:49 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed 2025/12/20 16:33:49 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:33:49.796 < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:33:49.796 (11.039s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:49.796 2025/12/20 16:33:49 Cleaning app 2025/12/20 16:33:49 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:33:52 2025-12-20 16:33:51,431 p=21630 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:33:51,431 p=21630 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:51,737 p=21630 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:33:51,737 p=21630 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:52,042 p=21630 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:33:52,042 p=21630 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:52,348 p=21630 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:33:52,349 p=21630 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:33:52,366 p=21630 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:33:52,366 p=21630 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:33:52,366 p=21630 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:52,386 p=21630 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:33:52,386 p=21630 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:52,400 p=21630 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:33:52,401 p=21630 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:33:52,777 p=21630 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:33:52,778 p=21630 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:33:52,778 p=21630 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:33:52,778 p=21630 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:33:52,799 p=21630 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:33:52,800 p=21630 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:33:52,800 p=21630 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:33:52,801 p=21630 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:33:52,801 p=21630 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-186 2025/12/20 16:33:52 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:52.995 (3.2s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:52.996 2025/12/20 16:33:52 Cleaning setup resources for the backup 2025/12/20 16:33:52 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:33:52 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:33:53 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:53.065 (69ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:53.065 2025/12/20 16:33:53 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:33:53.092 (28ms) Attempt #1 Failed. Retrying ↺ @ 12/20/25 16:33:53.092 > Enter [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:33:53.092 < Exit [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:33:53.128 (36ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:33:53.129 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:33:53.129 (0s) > Enter [It] [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored @ 12/20/25 16:33:53.129 2025/12/20 16:33:53 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:33:53.154 2025/12/20 16:33:53 csi 2025/12/20 16:33:53 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "39c9173f-bb91-4929-9e52-018cd8a4326a", "resourceVersion": "68065", "generation": 1, "creationTimestamp": "2025-12-20T16:33:53Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:33:53Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:33:53.253 2025/12/20 16:33:53 Waiting for velero pod to be running 2025/12/20 16:33:53 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:33:53 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "39c9173f-bb91-4929-9e52-018cd8a4326a", "resourceVersion": "68065", "generation": 1, "creationTimestamp": "2025-12-20T16:33:53Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:33:53Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:33:58 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:33:58.31 2025/12/20 16:33:58 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:33:58 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:33:58 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case ocp-kubevirt @ 12/20/25 16:33:58.58 2025/12/20 16:33:58 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:33:58 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:34:01 2025-12-20 16:34:00,223 p=21810 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:34:00,224 p=21810 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:00,528 p=21810 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:34:00,528 p=21810 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:00,828 p=21810 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:34:00,829 p=21810 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:01,131 p=21810 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:34:01,131 p=21810 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:01,148 p=21810 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:34:01,149 p=21810 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:34:01,149 p=21810 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:01,168 p=21810 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:34:01,168 p=21810 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:01,182 p=21810 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:34:01,183 p=21810 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:34:01,555 p=21810 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:34:01,555 p=21810 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:34:01,555 p=21810 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:34:01,555 p=21810 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:01,577 p=21810 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:34:01,577 p=21810 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:34:01,578 p=21810 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:34:01,578 p=21810 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:34:01,578 p=21810 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-186 2025/12/20 16:34:01 2025/12/20 16:34:01 ❌ Operation 'Deploy application ocp-kubevirt' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-186","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_running":false,"with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:34:01.77 < Exit [It] [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored @ 12/20/25 16:34:01.77 (8.641s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:34:01.77 2025/12/20 16:34:01 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:34:01.77 2025/12/20 16:34:01 The failed spec name is: CSI: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:34:01.77 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:34:01.77 STEP: Run must-gather because the spec failed @ 12/20/25 16:34:01.77 2025/12/20 16:34:01 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:34:01 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:34:12 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:34:12 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed 2025/12/20 16:34:12 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:34:12.875 The folder logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored/must-gather already exists, skipping renaming the folder < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:34:12.875 (11.105s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:12.875 2025/12/20 16:34:12 Cleaning app 2025/12/20 16:34:12 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:34:15 2025-12-20 16:34:14,526 p=22035 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:34:14,527 p=22035 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:14,840 p=22035 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:34:14,840 p=22035 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:15,148 p=22035 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:34:15,148 p=22035 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:15,452 p=22035 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:34:15,452 p=22035 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:15,469 p=22035 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:34:15,470 p=22035 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:34:15,470 p=22035 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:15,489 p=22035 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:34:15,489 p=22035 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:15,503 p=22035 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:34:15,504 p=22035 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:34:15,883 p=22035 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:34:15,884 p=22035 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:34:15,884 p=22035 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:34:15,884 p=22035 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:15,906 p=22035 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:34:15,907 p=22035 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:34:15,907 p=22035 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:34:15,907 p=22035 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:34:15,907 p=22035 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-186 2025/12/20 16:34:16 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:16.133 (3.259s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:16.133 2025/12/20 16:34:16 Cleaning setup resources for the backup 2025/12/20 16:34:16 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:34:16 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:34:16 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:16.198 (64ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:16.198 2025/12/20 16:34:16 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:16.221 (22ms) Attempt #2 Failed. Retrying ↺ @ 12/20/25 16:34:16.221 > Enter [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:34:16.221 < Exit [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:34:16.248 (28ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:34:16.248 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:34:16.248 (0s) > Enter [It] [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored @ 12/20/25 16:34:16.248 2025/12/20 16:34:16 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:34:16.283 2025/12/20 16:34:16 csi 2025/12/20 16:34:16 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "76856860-d508-4a25-a2a3-2379668b08ec", "resourceVersion": "68602", "generation": 1, "creationTimestamp": "2025-12-20T16:34:16Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:34:16Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:34:16.38 2025/12/20 16:34:16 Waiting for velero pod to be running 2025/12/20 16:34:16 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:34:16 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "76856860-d508-4a25-a2a3-2379668b08ec", "resourceVersion": "68602", "generation": 1, "creationTimestamp": "2025-12-20T16:34:16Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:34:16Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:34:21 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:34:21.432 2025/12/20 16:34:21 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:34:21 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:34:21 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case ocp-kubevirt @ 12/20/25 16:34:21.605 2025/12/20 16:34:21 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:34:21 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:34:24 2025-12-20 16:34:23,424 p=22215 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:34:23,425 p=22215 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:23,760 p=22215 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:34:23,760 p=22215 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:24,097 p=22215 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:34:24,097 p=22215 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:24,416 p=22215 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:34:24,416 p=22215 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:24,433 p=22215 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:34:24,434 p=22215 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:34:24,434 p=22215 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:24,454 p=22215 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:34:24,454 p=22215 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:24,469 p=22215 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:34:24,470 p=22215 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:34:24,893 p=22215 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:34:24,894 p=22215 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:34:24,894 p=22215 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:34:24,894 p=22215 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:24,918 p=22215 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:34:24,919 p=22215 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:34:24,919 p=22215 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:34:24,920 p=22215 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:34:24,920 p=22215 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-186 2025/12/20 16:34:25 2025/12/20 16:34:25 ❌ Operation 'Deploy application ocp-kubevirt' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-186","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_running":false,"with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:34:25.144 < Exit [It] [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored @ 12/20/25 16:34:25.144 (8.896s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:34:25.144 2025/12/20 16:34:25 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:34:25.144 2025/12/20 16:34:25 The failed spec name is: CSI: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:34:25.144 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:34:25.144 STEP: Run must-gather because the spec failed @ 12/20/25 16:34:25.144 2025/12/20 16:34:25 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:34:25 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:34:36 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:34:36 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed 2025/12/20 16:34:36 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:34:36.375 The folder logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored/must-gather already exists, skipping renaming the folder < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:34:36.375 (11.231s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:36.375 2025/12/20 16:34:36 Cleaning app 2025/12/20 16:34:36 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:34:39 2025-12-20 16:34:38,026 p=22436 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:34:38,026 p=22436 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:38,331 p=22436 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:34:38,332 p=22436 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:38,633 p=22436 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:34:38,633 p=22436 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:38,941 p=22436 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:34:38,941 p=22436 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:38,959 p=22436 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:34:38,960 p=22436 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:34:38,960 p=22436 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:38,980 p=22436 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:34:38,980 p=22436 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:38,996 p=22436 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:34:38,996 p=22436 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:34:39,375 p=22436 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:34:39,376 p=22436 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:34:39,376 p=22436 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:34:39,376 p=22436 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:39,398 p=22436 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:34:39,399 p=22436 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:34:39,399 p=22436 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:34:39,399 p=22436 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:34:39,399 p=22436 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-186 2025/12/20 16:34:39 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:39.597 (3.222s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:39.597 2025/12/20 16:34:39 Cleaning setup resources for the backup 2025/12/20 16:34:39 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:34:39 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:34:39 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:39.746 (149ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:39.747 2025/12/20 16:34:39 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:39.768 (22ms) • [FAILED] [69.907 seconds] CSI: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored /alabama/cspi/e2e/kubevirt-plugin/backup_restore_csi.go:51 [FAILED] Unexpected error: <*errors.Error | 0xc00026cdc0>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-186","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_running":false,"with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"test-oadp-186\",\"non_admin_user\":false,\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"vm_running\":false,\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:34:25.144 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ CSI: Backup/Restore Openshift Virtualization Workloads  [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume /alabama/cspi/e2e/kubevirt-plugin/backup_restore_csi.go:68 > Enter [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:34:39.769 < Exit [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:34:39.792 (23ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:34:39.792 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:34:39.792 (0s) > Enter [It] [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume @ 12/20/25 16:34:39.792 2025/12/20 16:34:39 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:34:39.818 2025/12/20 16:34:39 csi 2025/12/20 16:34:39 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "0987f11f-3f87-437d-90ae-2f4db927c480", "resourceVersion": "69084", "generation": 1, "creationTimestamp": "2025-12-20T16:34:39Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:34:39Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:34:39.914 2025/12/20 16:34:39 Waiting for velero pod to be running 2025/12/20 16:34:39 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:34:39 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "0987f11f-3f87-437d-90ae-2f4db927c480", "resourceVersion": "69084", "generation": 1, "creationTimestamp": "2025-12-20T16:34:39Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:34:39Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:34:44 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:34:44.967 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2025/12/20 16:34:45 The 'openshift-storage' namespace exists 2025/12/20 16:34:45 Checking default storage class count 2025/12/20 16:34:45 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2025/12/20 16:34:45 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:34:45 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:34:45 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case ocp-datavolume @ 12/20/25 16:34:45.463 2025/12/20 16:34:45 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-datavolume 2025/12/20 16:34:45 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:34:48 2025-12-20 16:34:47,088 p=22638 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:34:47,089 p=22638 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:47,392 p=22638 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:34:47,392 p=22638 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:47,696 p=22638 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:34:47,696 p=22638 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:47,999 p=22638 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:34:47,999 p=22638 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:34:48,016 p=22638 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:34:48,017 p=22638 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:34:48,017 p=22638 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:48,038 p=22638 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:34:48,038 p=22638 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:48,053 p=22638 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:34:48,054 p=22638 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:34:48,434 p=22638 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:34:48,434 p=22638 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:34:48,434 p=22638 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:34:48,434 p=22638 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:34:48,456 p=22638 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:34:48,456 p=22638 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:34:48,457 p=22638 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:34:48,457 p=22638 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:34:48,457 p=22638 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-187 2025/12/20 16:34:48 2025/12/20 16:34:48 ❌ Operation 'Deploy application ocp-datavolume' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-187","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:34:48.666 < Exit [It] [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume @ 12/20/25 16:34:48.666 (8.874s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:34:48.666 2025/12/20 16:34:48 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:34:48.666 2025/12/20 16:34:48 The failed spec name is: CSI: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:34:48.666 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:34:48.666 2025/12/20 16:34:48 The folder logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume does not exists, creating new folder with the name: logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume STEP: Run must-gather because the spec failed @ 12/20/25 16:34:48.666 2025/12/20 16:34:48 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:34:48 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:34:59 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:34:59 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed 2025/12/20 16:34:59 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored 2025/12/20 16:34:59 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:34:59.676 < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:34:59.676 (11.01s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:34:59.676 2025/12/20 16:34:59 Cleaning app 2025/12/20 16:34:59 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:35:02 2025-12-20 16:35:01,327 p=22864 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:35:01,328 p=22864 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:01,632 p=22864 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:35:01,632 p=22864 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:01,937 p=22864 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:35:01,937 p=22864 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:02,240 p=22864 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:35:02,241 p=22864 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:02,258 p=22864 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:35:02,258 p=22864 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:35:02,258 p=22864 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:02,277 p=22864 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:35:02,277 p=22864 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:02,292 p=22864 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:35:02,292 p=22864 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:35:02,692 p=22864 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:35:02,692 p=22864 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:35:02,693 p=22864 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:35:02,693 p=22864 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:02,714 p=22864 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:35:02,715 p=22864 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:35:02,715 p=22864 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:35:02,716 p=22864 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:35:02,716 p=22864 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-187 2025/12/20 16:35:02 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:02.925 (3.249s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:02.925 2025/12/20 16:35:02 Cleaning setup resources for the backup 2025/12/20 16:35:02 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:35:02 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:35:02 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:02.987 (63ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:02.988 2025/12/20 16:35:02 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:03.01 (22ms) Attempt #1 Failed. Retrying ↺ @ 12/20/25 16:35:03.01 > Enter [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:35:03.01 < Exit [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:35:03.034 (25ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:35:03.034 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:35:03.034 (0s) > Enter [It] [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume @ 12/20/25 16:35:03.034 2025/12/20 16:35:03 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:35:03.061 2025/12/20 16:35:03 csi 2025/12/20 16:35:03 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "eef0f4aa-aeab-4512-bba8-a304da2ed967", "resourceVersion": "69579", "generation": 1, "creationTimestamp": "2025-12-20T16:35:03Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:35:03Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:35:03.15 2025/12/20 16:35:03 Waiting for velero pod to be running 2025/12/20 16:35:03 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:35:03 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "eef0f4aa-aeab-4512-bba8-a304da2ed967", "resourceVersion": "69579", "generation": 1, "creationTimestamp": "2025-12-20T16:35:03Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:35:03Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:35:08 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:35:08.201 2025/12/20 16:35:08 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:35:08 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:35:08 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case ocp-datavolume @ 12/20/25 16:35:08.385 2025/12/20 16:35:08 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-datavolume 2025/12/20 16:35:08 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:35:11 2025-12-20 16:35:10,022 p=23043 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:35:10,022 p=23043 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:10,327 p=23043 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:35:10,327 p=23043 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:10,631 p=23043 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:35:10,631 p=23043 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:10,936 p=23043 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:35:10,937 p=23043 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:10,954 p=23043 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:35:10,955 p=23043 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:35:10,955 p=23043 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:10,974 p=23043 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:35:10,975 p=23043 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:10,989 p=23043 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:35:10,990 p=23043 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:35:11,367 p=23043 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:35:11,368 p=23043 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:35:11,368 p=23043 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:35:11,368 p=23043 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:11,390 p=23043 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:35:11,391 p=23043 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:35:11,391 p=23043 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:35:11,391 p=23043 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:35:11,392 p=23043 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-187 2025/12/20 16:35:11 2025/12/20 16:35:11 ❌ Operation 'Deploy application ocp-datavolume' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-187","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:35:11.597 < Exit [It] [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume @ 12/20/25 16:35:11.597 (8.563s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:35:11.598 2025/12/20 16:35:11 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:35:11.598 2025/12/20 16:35:11 The failed spec name is: CSI: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:35:11.598 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:35:11.598 STEP: Run must-gather because the spec failed @ 12/20/25 16:35:11.598 2025/12/20 16:35:11 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:35:11 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:35:22 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:35:22 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed 2025/12/20 16:35:22 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored 2025/12/20 16:35:22 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:35:22.707 The folder logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume/must-gather already exists, skipping renaming the folder < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:35:22.707 (11.11s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:22.707 2025/12/20 16:35:22 Cleaning app 2025/12/20 16:35:22 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:35:25 2025-12-20 16:35:24,366 p=23266 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:35:24,366 p=23266 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:24,667 p=23266 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:35:24,667 p=23266 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:24,971 p=23266 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:35:24,971 p=23266 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:25,274 p=23266 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:35:25,274 p=23266 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:25,291 p=23266 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:35:25,292 p=23266 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:35:25,292 p=23266 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:25,311 p=23266 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:35:25,311 p=23266 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:25,325 p=23266 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:35:25,326 p=23266 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:35:25,698 p=23266 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:35:25,699 p=23266 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:35:25,699 p=23266 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:35:25,699 p=23266 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:25,721 p=23266 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:35:25,721 p=23266 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:35:25,722 p=23266 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:35:25,722 p=23266 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:35:25,722 p=23266 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-187 2025/12/20 16:35:25 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:25.932 (3.224s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:25.932 2025/12/20 16:35:25 Cleaning setup resources for the backup 2025/12/20 16:35:25 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:35:25 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:35:25 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:25.994 (62ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:25.994 2025/12/20 16:35:25 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:26.018 (23ms) Attempt #2 Failed. Retrying ↺ @ 12/20/25 16:35:26.018 > Enter [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:35:26.018 < Exit [BeforeEach] CSI: Backup/Restore Openshift Virtualization Workloads @ 12/20/25 16:35:26.049 (32ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:35:26.049 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:35:26.049 (0s) > Enter [It] [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume @ 12/20/25 16:35:26.049 2025/12/20 16:35:26 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:35:26.088 2025/12/20 16:35:26 csi 2025/12/20 16:35:26 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "e92f8503-fed1-4bb7-9028-6784b0c5cda1", "resourceVersion": "70045", "generation": 1, "creationTimestamp": "2025-12-20T16:35:26Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:35:26Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:35:26.177 2025/12/20 16:35:26 Waiting for velero pod to be running 2025/12/20 16:35:26 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:35:26 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "e92f8503-fed1-4bb7-9028-6784b0c5cda1", "resourceVersion": "70045", "generation": 1, "creationTimestamp": "2025-12-20T16:35:26Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:35:26Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:35:31 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:35:31.23 2025/12/20 16:35:31 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:35:31 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:35:31 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case ocp-datavolume @ 12/20/25 16:35:31.497 2025/12/20 16:35:31 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-datavolume 2025/12/20 16:35:31 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:35:34 2025-12-20 16:35:33,147 p=23447 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:35:33,147 p=23447 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:33,446 p=23447 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:35:33,446 p=23447 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:33,753 p=23447 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:35:33,753 p=23447 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:34,050 p=23447 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:35:34,050 p=23447 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:34,067 p=23447 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:35:34,068 p=23447 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:35:34,068 p=23447 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:34,086 p=23447 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:35:34,087 p=23447 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:34,100 p=23447 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:35:34,101 p=23447 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:35:34,475 p=23447 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:35:34,476 p=23447 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:35:34,476 p=23447 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:35:34,476 p=23447 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:34,498 p=23447 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:35:34,498 p=23447 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:35:34,498 p=23447 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:35:34,499 p=23447 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:35:34,499 p=23447 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-187 2025/12/20 16:35:34 2025/12/20 16:35:34 ❌ Operation 'Deploy application ocp-datavolume' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-187","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:35:34.706 < Exit [It] [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume @ 12/20/25 16:35:34.706 (8.657s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:35:34.706 2025/12/20 16:35:34 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:35:34.706 2025/12/20 16:35:34 The failed spec name is: CSI: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:35:34.706 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:35:34.706 STEP: Run must-gather because the spec failed @ 12/20/25 16:35:34.706 2025/12/20 16:35:34 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:35:34 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:35:45 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:35:45 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed 2025/12/20 16:35:45 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored 2025/12/20 16:35:45 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:35:45.833 The folder logs/It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume/must-gather already exists, skipping renaming the folder < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:35:45.833 (11.127s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:45.833 2025/12/20 16:35:45 Cleaning app 2025/12/20 16:35:45 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:35:48 2025-12-20 16:35:47,480 p=23671 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:35:47,480 p=23671 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:47,780 p=23671 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:35:47,780 p=23671 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:48,085 p=23671 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:35:48,085 p=23671 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:48,385 p=23671 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:35:48,385 p=23671 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:35:48,402 p=23671 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:35:48,403 p=23671 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:35:48,403 p=23671 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:48,422 p=23671 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:35:48,422 p=23671 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:48,436 p=23671 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:35:48,437 p=23671 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:35:48,822 p=23671 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:35:48,823 p=23671 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:35:48,823 p=23671 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:35:48,823 p=23671 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:35:48,845 p=23671 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:35:48,845 p=23671 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:35:48,846 p=23671 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:35:48,846 p=23671 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:35:48,846 p=23671 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-187 2025/12/20 16:35:49 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:49.054 (3.221s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:49.054 2025/12/20 16:35:49 Cleaning setup resources for the backup 2025/12/20 16:35:49 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:35:49 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:35:49 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:49.118 (64ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:49.119 2025/12/20 16:35:49 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:35:49.143 (25ms) • [FAILED] [69.375 seconds] CSI: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume /alabama/cspi/e2e/kubevirt-plugin/backup_restore_csi.go:68 [FAILED] Unexpected error: <*errors.Error | 0xc000d55300>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-187","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"test-oadp-187\",\"non_admin_user\":false,\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:35:34.706 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ S ------------------------------ Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads  [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode /alabama/cspi/e2e/kubevirt-plugin/backup_restore_datamover.go:129 > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:35:49.143 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:35:49.144 (0s) > Enter [It] [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode @ 12/20/25 16:35:49.144 2025/12/20 16:35:49 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:35:49.17 2025/12/20 16:35:49 native-datamover 2025/12/20 16:35:49 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "e96d3cd7-f83f-4dac-b725-c16477cb69fc", "resourceVersion": "70512", "generation": 1, "creationTimestamp": "2025-12-20T16:35:49Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:35:49Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:35:49.269 2025/12/20 16:35:49 Waiting for velero pod to be running 2025/12/20 16:35:49 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:35:49 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "e96d3cd7-f83f-4dac-b725-c16477cb69fc", "resourceVersion": "70512", "generation": 1, "creationTimestamp": "2025-12-20T16:35:49Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:35:49Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:35:54 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:35:54.32 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2025/12/20 16:35:54 The 'openshift-storage' namespace exists 2025/12/20 16:35:54 Checking default storage class count 2025/12/20 16:35:54 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2025/12/20 16:35:54 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:35:54 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:35:54 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:35:54 Checking for correct number of running NodeAgent pods... 2025/12/20 16:35:54 pod: node-agent-9zqcm is not yet running with status: {Pending [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:35:49 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:35:49 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:35:49 +0000 UTC ContainersNotReady containers with unready status: [node-agent]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:35:49 +0000 UTC ContainersNotReady containers with unready status: [node-agent]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:35:49 +0000 UTC }] 10.0.81.187 [{10.0.81.187}] [] 2025-12-20 16:35:49 +0000 UTC [] [{node-agent {&ContainerStateWaiting{Reason:ContainerCreating,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc000c673ea map[] nil [{host-pods /host_pods false } {host-plugins /var/lib/kubelet/plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc000ba6db0} {credentials /tmp/credentials false } {home-velero /home/velero false } {tmp /tmp false } {kube-api-access-p6qtj /var/run/secrets/kubernetes.io/serviceaccount true 0xc000ba6dc0}] nil []}] Burstable [] []} STEP: Installing application for case ocp-kubevirt @ 12/20/25 16:35:59.814 2025/12/20 16:35:59 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:35:59 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:36:02 2025-12-20 16:36:01,441 p=23870 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:36:01,442 p=23870 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:36:01,742 p=23870 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:36:01,742 p=23870 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:36:02,046 p=23870 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:36:02,046 p=23870 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:36:02,346 p=23870 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:36:02,347 p=23870 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:36:02,364 p=23870 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:36:02,364 p=23870 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:36:02,365 p=23870 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:36:02,384 p=23870 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:36:02,384 p=23870 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:36:02,398 p=23870 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:36:02,398 p=23870 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:36:02,776 p=23870 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:36:02,776 p=23870 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:36:02,776 p=23870 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:36:02,776 p=23870 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:36:02,798 p=23870 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:36:02,798 p=23870 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:36:02,798 p=23870 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:36:02,799 p=23870 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:36:02,799 p=23870 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:36:02 2025/12/20 16:36:02 ⚠️ Operation 'Deploy application ocp-kubevirt' failed with retryable server error (attempt 1/3): Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 2025/12/20 16:36:02 🕐 Waiting 1m0s before retry... 2025/12/20 16:37:02 🔄 [Attempt 2/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:37:02 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:37:06 2025-12-20 16:37:04,637 p=24049 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:37:04,638 p=24049 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:37:04,939 p=24049 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:37:04,940 p=24049 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:37:05,243 p=24049 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:37:05,244 p=24049 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:37:05,552 p=24049 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:37:05,552 p=24049 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:37:05,570 p=24049 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:37:05,570 p=24049 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:37:05,571 p=24049 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:37:05,590 p=24049 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:37:05,590 p=24049 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:37:05,604 p=24049 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:37:05,605 p=24049 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:37:05,977 p=24049 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:37:05,977 p=24049 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:37:05,978 p=24049 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:37:05,978 p=24049 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:37:05,999 p=24049 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:37:05,999 p=24049 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:37:06,000 p=24049 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:37:06,000 p=24049 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:37:06,000 p=24049 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:37:06 2025/12/20 16:37:06 ⚠️ Operation 'Deploy application ocp-kubevirt' failed with retryable server error (attempt 2/3): Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 2025/12/20 16:37:06 🕐 Waiting 2m0s before retry... 2025/12/20 16:39:06 🔄 [Attempt 3/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:39:06 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:39:09 2025-12-20 16:39:07,914 p=24229 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:39:07,915 p=24229 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:08,219 p=24229 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:39:08,220 p=24229 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:08,527 p=24229 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:39:08,527 p=24229 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:08,828 p=24229 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:39:08,828 p=24229 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:08,845 p=24229 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:39:08,845 p=24229 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:39:08,845 p=24229 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:39:08,865 p=24229 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:39:08,865 p=24229 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:39:08,879 p=24229 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:39:08,880 p=24229 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:39:09,260 p=24229 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:39:09,260 p=24229 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:39:09,261 p=24229 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:39:09,261 p=24229 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:39:09,282 p=24229 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:39:09,283 p=24229 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:39:09,283 p=24229 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:39:09,284 p=24229 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:39:09,284 p=24229 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:39:09 2025/12/20 16:39:09 ❌ Operation 'Deploy application ocp-kubevirt' failed after 3 attempts. Final error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:39:09.53 < Exit [It] [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode @ 12/20/25 16:39:09.53 (3m20.387s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:39:09.53 2025/12/20 16:39:09 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:39:09.53 2025/12/20 16:39:09 The failed spec name is: Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:39:09.53 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:39:09.53 2025/12/20 16:39:09 The folder logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode does not exists, creating new folder with the name: logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode STEP: Run must-gather because the spec failed @ 12/20/25 16:39:09.53 2025/12/20 16:39:09 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:39:09 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:39:20 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:39:20 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed 2025/12/20 16:39:20 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored 2025/12/20 16:39:20 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume 2025/12/20 16:39:20 It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:39:20.637 < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:39:20.637 (11.107s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:39:20.637 2025/12/20 16:39:20 Cleaning app 2025/12/20 16:39:20 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:39:23 2025-12-20 16:39:22,273 p=24452 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:39:22,273 p=24452 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:22,581 p=24452 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:39:22,581 p=24452 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:22,880 p=24452 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:39:22,880 p=24452 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:23,180 p=24452 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:39:23,180 p=24452 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:23,197 p=24452 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:39:23,198 p=24452 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:39:23,198 p=24452 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:39:23,217 p=24452 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:39:23,217 p=24452 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:39:23,233 p=24452 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:39:23,233 p=24452 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:39:23,611 p=24452 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:39:23,612 p=24452 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:39:23,612 p=24452 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:39:23,612 p=24452 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:39:23,633 p=24452 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:39:23,634 p=24452 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:39:23,634 p=24452 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:39:23,634 p=24452 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:39:23,634 p=24452 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:39:23 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:39:23.832 (3.194s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:39:23.832 2025/12/20 16:39:23 Cleaning setup resources for the backup 2025/12/20 16:39:23 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:39:23 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:39:23 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:39:23.926 (95ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:39:23.926 2025/12/20 16:39:23 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:39:23.953 (27ms) Attempt #1 Failed. Retrying ↺ @ 12/20/25 16:39:23.953 > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:39:23.953 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:39:23.953 (0s) > Enter [It] [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode @ 12/20/25 16:39:23.953 2025/12/20 16:39:23 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:39:23.979 2025/12/20 16:39:23 native-datamover 2025/12/20 16:39:24 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "53882a22-e26c-4cfc-a499-1fefbf1501f7", "resourceVersion": "73756", "generation": 1, "creationTimestamp": "2025-12-20T16:39:24Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:39:24Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:39:24.086 2025/12/20 16:39:24 Waiting for velero pod to be running 2025/12/20 16:39:24 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:39:24 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "53882a22-e26c-4cfc-a499-1fefbf1501f7", "resourceVersion": "73756", "generation": 1, "creationTimestamp": "2025-12-20T16:39:24Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:39:24Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:39:29 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:39:29.136 2025/12/20 16:39:29 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:39:29 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:39:29 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:39:29 Checking for correct number of running NodeAgent pods... STEP: Installing application for case ocp-kubevirt @ 12/20/25 16:39:29.347 2025/12/20 16:39:29 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:39:29 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:39:32 2025-12-20 16:39:30,972 p=24633 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:39:30,972 p=24633 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:31,278 p=24633 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:39:31,279 p=24633 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:31,590 p=24633 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:39:31,590 p=24633 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:31,891 p=24633 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:39:31,891 p=24633 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:39:31,908 p=24633 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:39:31,909 p=24633 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:39:31,909 p=24633 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:39:31,928 p=24633 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:39:31,928 p=24633 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:39:31,942 p=24633 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:39:31,942 p=24633 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:39:32,318 p=24633 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:39:32,319 p=24633 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:39:32,319 p=24633 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:39:32,319 p=24633 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:39:32,341 p=24633 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:39:32,341 p=24633 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:39:32,341 p=24633 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:39:32,342 p=24633 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:39:32,342 p=24633 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:39:32 2025/12/20 16:39:32 ⚠️ Operation 'Deploy application ocp-kubevirt' failed with retryable server error (attempt 1/3): Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 2025/12/20 16:39:32 🕐 Waiting 1m0s before retry... 2025/12/20 16:40:32 🔄 [Attempt 2/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:40:32 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:40:35 2025-12-20 16:40:34,176 p=24813 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:40:34,177 p=24813 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:40:34,477 p=24813 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:40:34,477 p=24813 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:40:34,779 p=24813 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:40:34,779 p=24813 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:40:35,084 p=24813 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:40:35,084 p=24813 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:40:35,101 p=24813 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:40:35,101 p=24813 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:40:35,101 p=24813 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:40:35,120 p=24813 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:40:35,120 p=24813 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:40:35,135 p=24813 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:40:35,135 p=24813 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:40:35,516 p=24813 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:40:35,516 p=24813 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:40:35,516 p=24813 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:40:35,516 p=24813 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:40:35,538 p=24813 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:40:35,539 p=24813 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:40:35,539 p=24813 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:40:35,539 p=24813 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:40:35,539 p=24813 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:40:35 2025/12/20 16:40:35 ⚠️ Operation 'Deploy application ocp-kubevirt' failed with retryable server error (attempt 2/3): Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 2025/12/20 16:40:35 🕐 Waiting 2m0s before retry... 2025/12/20 16:42:35 🔄 [Attempt 3/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:42:35 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:42:38 2025-12-20 16:42:37,383 p=24992 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:42:37,383 p=24992 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:42:37,684 p=24992 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:42:37,684 p=24992 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:42:37,987 p=24992 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:42:37,987 p=24992 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:42:38,290 p=24992 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:42:38,290 p=24992 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:42:38,308 p=24992 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:42:38,308 p=24992 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:42:38,308 p=24992 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:42:38,327 p=24992 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:42:38,327 p=24992 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:42:38,342 p=24992 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:42:38,343 p=24992 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:42:38,722 p=24992 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:42:38,723 p=24992 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:42:38,723 p=24992 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:42:38,723 p=24992 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:42:38,744 p=24992 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:42:38,745 p=24992 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:42:38,745 p=24992 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:42:38,746 p=24992 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:42:38,746 p=24992 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:42:38 2025/12/20 16:42:38 ❌ Operation 'Deploy application ocp-kubevirt' failed after 3 attempts. Final error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:42:38.966 < Exit [It] [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode @ 12/20/25 16:42:38.966 (3m15.013s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:42:38.966 2025/12/20 16:42:38 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:42:38.966 2025/12/20 16:42:38 The failed spec name is: Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:42:38.966 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:42:38.967 STEP: Run must-gather because the spec failed @ 12/20/25 16:42:38.967 2025/12/20 16:42:38 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:42:38 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:42:50 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:42:50 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed 2025/12/20 16:42:50 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored 2025/12/20 16:42:50 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume 2025/12/20 16:42:50 It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:42:50.08 The folder logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode/must-gather already exists, skipping renaming the folder < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:42:50.08 (11.113s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:42:50.08 2025/12/20 16:42:50 Cleaning app 2025/12/20 16:42:50 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:42:53 2025-12-20 16:42:51,727 p=25218 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:42:51,727 p=25218 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:42:52,032 p=25218 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:42:52,032 p=25218 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:42:52,333 p=25218 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:42:52,333 p=25218 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:42:52,636 p=25218 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:42:52,637 p=25218 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:42:52,654 p=25218 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:42:52,654 p=25218 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:42:52,654 p=25218 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:42:52,673 p=25218 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:42:52,674 p=25218 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:42:52,688 p=25218 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:42:52,688 p=25218 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:42:53,063 p=25218 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:42:53,063 p=25218 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:42:53,063 p=25218 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:42:53,063 p=25218 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:42:53,085 p=25218 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:42:53,085 p=25218 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:42:53,086 p=25218 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:42:53,086 p=25218 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:42:53,086 p=25218 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:42:53 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:42:53.286 (3.207s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:42:53.286 2025/12/20 16:42:53 Cleaning setup resources for the backup 2025/12/20 16:42:53 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:42:53 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:42:53 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:42:53.378 (92ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:42:53.379 2025/12/20 16:42:53 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:42:53.402 (23ms) Attempt #2 Failed. Retrying ↺ @ 12/20/25 16:42:53.402 > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:42:53.402 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:42:53.402 (0s) > Enter [It] [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode @ 12/20/25 16:42:53.402 2025/12/20 16:42:53 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:42:53.419 2025/12/20 16:42:53 native-datamover 2025/12/20 16:42:53 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "e9bd4ea2-ff77-40bc-8656-791e709504f8", "resourceVersion": "76885", "generation": 1, "creationTimestamp": "2025-12-20T16:42:53Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:42:53Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:42:53.532 2025/12/20 16:42:53 Waiting for velero pod to be running 2025/12/20 16:42:53 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:42:53 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "e9bd4ea2-ff77-40bc-8656-791e709504f8", "resourceVersion": "76885", "generation": 1, "creationTimestamp": "2025-12-20T16:42:53Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:42:53Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:42:58 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:42:58.586 2025/12/20 16:42:58 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:42:58 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:42:58 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:42:58 Checking for correct number of running NodeAgent pods... STEP: Installing application for case ocp-kubevirt @ 12/20/25 16:42:58.896 2025/12/20 16:42:58 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:42:58 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:43:01 2025-12-20 16:43:00,545 p=25398 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:43:00,546 p=25398 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:43:00,854 p=25398 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:43:00,854 p=25398 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:43:01,156 p=25398 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:43:01,156 p=25398 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:43:01,465 p=25398 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:43:01,465 p=25398 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:43:01,483 p=25398 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:43:01,483 p=25398 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:43:01,483 p=25398 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:43:01,503 p=25398 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:43:01,503 p=25398 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:43:01,518 p=25398 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:43:01,518 p=25398 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:43:01,898 p=25398 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:43:01,899 p=25398 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:43:01,899 p=25398 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:43:01,899 p=25398 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:43:01,921 p=25398 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:43:01,921 p=25398 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:43:01,922 p=25398 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:43:01,922 p=25398 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:43:01,922 p=25398 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:43:02 2025/12/20 16:43:02 ⚠️ Operation 'Deploy application ocp-kubevirt' failed with retryable server error (attempt 1/3): Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 2025/12/20 16:43:02 🕐 Waiting 1m0s before retry... 2025/12/20 16:44:02 🔄 [Attempt 2/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:44:02 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:44:05 2025-12-20 16:44:03,797 p=25577 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:44:03,797 p=25577 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:44:04,103 p=25577 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:44:04,103 p=25577 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:44:04,405 p=25577 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:44:04,405 p=25577 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:44:04,705 p=25577 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:44:04,705 p=25577 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:44:04,722 p=25577 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:44:04,723 p=25577 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:44:04,723 p=25577 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:44:04,742 p=25577 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:44:04,742 p=25577 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:44:04,756 p=25577 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:44:04,757 p=25577 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:44:05,131 p=25577 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:44:05,131 p=25577 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:44:05,131 p=25577 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:44:05,131 p=25577 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:44:05,153 p=25577 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:44:05,154 p=25577 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:44:05,154 p=25577 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:44:05,154 p=25577 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:44:05,154 p=25577 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:44:05 2025/12/20 16:44:05 ⚠️ Operation 'Deploy application ocp-kubevirt' failed with retryable server error (attempt 2/3): Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 2025/12/20 16:44:05 🕐 Waiting 2m0s before retry... 2025/12/20 16:46:05 🔄 [Attempt 3/3] Executing operation: Deploy application ocp-kubevirt 2025/12/20 16:46:05 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:46:08 2025-12-20 16:46:07,060 p=25756 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:46:07,060 p=25756 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:07,367 p=25756 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:46:07,368 p=25756 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:07,675 p=25756 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:46:07,675 p=25756 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:07,975 p=25756 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:46:07,975 p=25756 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:07,993 p=25756 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:46:07,993 p=25756 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:46:07,993 p=25756 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:08,012 p=25756 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:46:08,012 p=25756 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:08,027 p=25756 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:46:08,027 p=25756 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:46:08,402 p=25756 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:46:08,403 p=25756 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:46:08,403 p=25756 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:46:08,403 p=25756 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:08,424 p=25756 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:46:08,425 p=25756 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:46:08,425 p=25756 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:46:08,426 p=25756 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:46:08,426 p=25756 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:46:08 2025/12/20 16:46:08 ❌ Operation 'Deploy application ocp-kubevirt' failed after 3 attempts. Final error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:46:08.642 < Exit [It] [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode @ 12/20/25 16:46:08.642 (3m15.24s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:46:08.642 2025/12/20 16:46:08 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:46:08.642 2025/12/20 16:46:08 The failed spec name is: Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:46:08.642 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:46:08.642 STEP: Run must-gather because the spec failed @ 12/20/25 16:46:08.642 2025/12/20 16:46:08 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2025/12/20 16:46:08 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:46:19 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2025/12/20 16:46:19 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-185_kubevirt_Tier0_vf_vm_Backing_up_started_VM_should_succeed 2025/12/20 16:46:19 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-186_kubevirt_Tier0_vf_vm_Stopped_VM_should_be_restored 2025/12/20 16:46:19 It_CSI_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-187_kubevirt_Tier0_vf_vm_Backup-restore_data_volume 2025/12/20 16:46:19 It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:46:19.926 The folder logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Tier1_Started_VM_should_over_ceph_filesytem_mode/must-gather already exists, skipping renaming the folder < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:46:19.926 (11.284s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:46:19.926 2025/12/20 16:46:19 Cleaning app 2025/12/20 16:46:19 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:46:22 2025-12-20 16:46:21,564 p=25979 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:46:21,564 p=25979 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:21,867 p=25979 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:46:21,868 p=25979 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:22,168 p=25979 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:46:22,168 p=25979 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:22,475 p=25979 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:46:22,476 p=25979 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:22,493 p=25979 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:46:22,494 p=25979 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:46:22,494 p=25979 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:22,512 p=25979 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:46:22,513 p=25979 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:22,527 p=25979 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:46:22,528 p=25979 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:46:22,902 p=25979 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:46:22,903 p=25979 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:46:22,903 p=25979 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:46:22,903 p=25979 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:22,924 p=25979 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:46:22,925 p=25979 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:46:22,925 p=25979 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:46:22,926 p=25979 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:46:22,926 p=25979 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2025/12/20 16:46:23 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:46:23.121 (3.195s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:46:23.121 2025/12/20 16:46:23 Cleaning setup resources for the backup 2025/12/20 16:46:23 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:46:23 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:46:23 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:46:23.248 (127ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:46:23.248 2025/12/20 16:46:23 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:46:23.273 (25ms) • [FAILED] [634.130 seconds] Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode /alabama/cspi/e2e/kubevirt-plugin/backup_restore_datamover.go:129 [FAILED] Unexpected error: <*fmt.wrapError | 0xc000128000>: operation 'Deploy application ocp-kubevirt' failed after 3 attempts: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-401","non_admin_user":false,"storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","user_kubeconfig":"/home/jenkins/.kube/config","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { msg: "operation 'Deploy application ocp-kubevirt' failed after 3 attempts: Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"test-oadp-401\",\"non_admin_user\":false,\"storage_class\":\"odf-operator-cephfs\",\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"vm_disk_pvc_volume_mode\":\"Filesystem\",\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", err: <*errors.Error | 0xc000a72000>{ context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"test-oadp-401\",\"non_admin_user\":false,\"storage_class\":\"odf-operator-cephfs\",\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"vm_disk_pvc_volume_mode\":\"Filesystem\",\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, }, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:46:08.642 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ [AfterSuite]  /alabama/cspi/e2e/kubevirt-plugin/kubevirt_suite_test.go:105 > Enter [AfterSuite] TOP-LEVEL @ 12/20/25 16:46:23.274 < Exit [AfterSuite] TOP-LEVEL @ 12/20/25 16:46:23.298 (24ms) [AfterSuite] PASSED [0.024 seconds] ------------------------------ [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report autogenerated by Ginkgo > Enter [ReportAfterSuite] TOP-LEVEL @ 12/20/25 16:46:23.298 < Exit [ReportAfterSuite] TOP-LEVEL @ 12/20/25 16:46:23.303 (5ms) [ReportAfterSuite] PASSED [0.005 seconds] ------------------------------ Summarizing 4 Failures: [FAIL] CSI: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-185] [kubevirt] [Tier0] [vf:vm] Backing up started VM should succeed /alabama/cspi/test_common/backup_restore_app_case.go:49 [FAIL] CSI: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-186] [kubevirt] [Tier0] [vf:vm] Stopped VM should be restored /alabama/cspi/test_common/backup_restore_app_case.go:49 [FAIL] CSI: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-187] [kubevirt] [Tier0] [vf:vm] Backup-restore data volume /alabama/cspi/test_common/backup_restore_app_case.go:49 [FAIL] Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-401] [kubevirt] [Tier1] Started VM should over ceph filesytem mode /alabama/cspi/test_common/backup_restore_app_case.go:49 Ran 4 of 5 Specs in 962.187 seconds FAIL! -- 0 Passed | 4 Failed | 0 Pending | 1 Skipped --- FAIL: TestOADPE2E (962.19s) FAIL Ginkgo ran 1 suite in 16m38.341608128s Test Suite Failed + readonly 'RED=\e[31m' + RED='\e[31m' + readonly 'BLUE=\033[34m' + BLUE='\033[34m' + readonly 'CLEAR=\e[39m' + CLEAR='\e[39m' ++ oc get infrastructures cluster -o 'jsonpath={.status.platform}' ++ awk '{print tolower($0)}' + CLOUD_PROVIDER=aws + [[ '' == \t\r\u\e ]] + echo /home/jenkins/.kube/config /home/jenkins/.kube/config + [[ aws == *-arm* ]] + [[ aws == *-fips* ]] + E2E_TIMEOUT_MULTIPLIER=2 + export NAMESPACE=openshift-adp + NAMESPACE=openshift-adp + export PROVIDER=aws + PROVIDER=aws ++ echo aws ++ awk '{print tolower($0)}' + BACKUP_LOCATION=aws + export BACKUP_LOCATION=aws + BACKUP_LOCATION=aws + export BUCKET=ci-op-2jmljgmw-interopoadp + BUCKET=ci-op-2jmljgmw-interopoadp + OADP_CREDS_FILE=/tmp/test-settings/credentials + OADP_VSL_CREDS_FILE=/tmp/test-settings/aws_vsl_creds +++ readlink -f /alabama/cspi/test_settings/scripts/test_runner.sh ++ dirname /alabama/cspi/test_settings/scripts/test_runner.sh + readonly SCRIPT_DIR=/alabama/cspi/test_settings/scripts + SCRIPT_DIR=/alabama/cspi/test_settings/scripts ++ cd /alabama/cspi/test_settings/scripts ++ git rev-parse --show-toplevel + readonly TOP_DIR=/alabama/cspi + TOP_DIR=/alabama/cspi + echo /alabama/cspi /alabama/cspi + TESTS_FOLDER=/alabama/cspi/e2e ++ oc get nodes -o 'jsonpath={.items[*].metadata.labels.kubernetes\.io/arch}' ++ tr ' ' '\n' ++ sort -u ++ xargs + export NODES_ARCHITECTURE=amd64 + NODES_ARCHITECTURE=amd64 + export OADP_REPOSITORY=redhat + OADP_REPOSITORY=redhat + SKIP_DPA_CREATION=false ++ oc get ns openshift-storage ++ echo true + OPENSHIFT_STORAGE=true + '[' redhat == upstream-velero ']' + '[' true == true ']' ++ oc get sc ++ awk '$1 ~ /^.+ceph-rbd$/ {print $1}' ++ tail -1 + CEPH_RBD_STORAGE_CLASS=odf-operator-ceph-rbd + '[' -n odf-operator-ceph-rbd ']' + export CEPH_RBD_STORAGE_CLASS + echo 'ceph-rbd StorageClass found: odf-operator-ceph-rbd' ceph-rbd StorageClass found: odf-operator-ceph-rbd ++ oc get storageclass -o 'jsonpath={range .items[*]}{@.metadata.name}{" "}{@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}{end}' ++ awk '$2=="true"{print $1}' ++ wc -l + NUM_DEFAULT_STORAGE_CLASS=1 + '[' 1 -ne 1 ']' ++ oc get storageclass -o 'jsonpath={.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=='\''true'\'')].metadata.name}' + DEFAULT_SC=odf-operator-ceph-rbd + export STORAGE_CLASS=odf-operator-ceph-rbd + STORAGE_CLASS=odf-operator-ceph-rbd + '[' -n odf-operator-ceph-rbd ']' + '[' odf-operator-ceph-rbd '!=' odf-operator-ceph-rbd ']' + export STORAGE_CLASS_OPENSHIFT_STORAGE=odf-operator-ceph-rbd + STORAGE_CLASS_OPENSHIFT_STORAGE=odf-operator-ceph-rbd + echo 'Using the StorageClass for openshift-storage: odf-operator-ceph-rbd' Using the StorageClass for openshift-storage: odf-operator-ceph-rbd + [[ amd64 != \a\m\d\6\4 ]] + TEST_FILTER='!// || (// && !exclude_aws && (!/target/ || target_aws) ) ' + [[ aws =~ ^(osp|vsphere|gcp-wif|ibmcloud|azure-sts|aro-4) ]] + [[ aws =~ ^azure_sak ]] ++ oc config current-context ++ awk -F / '{print $2}' + SETTINGS_TMP=/alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443 + mkdir -p /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443 ++ oc get authentication cluster -o 'jsonpath={.spec.serviceAccountIssuer}' + IS_OIDC= + '[' '!' -z ']' + [[ aws == \a\w\s ]] + export PROVIDER=aws + PROVIDER=aws + export CREDS_SECRET_REF=cloud-credentials + CREDS_SECRET_REF=cloud-credentials ++ oc get infrastructures cluster -o 'jsonpath={.status.platformStatus.aws.region}' --allow-missing-template-keys=false + export REGION=us-east-2 + REGION=us-east-2 + settings_script=aws_settings.sh + '[' aws == aws-sts ']' + BUCKET=ci-op-2jmljgmw-interopoadp + TMP_DIR=/alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443 + source /alabama/cspi/test_settings/scripts/aws_settings.sh ++ cat ++ [[ aws == *aws* ]] ++ cat ++ echo -e '\n }\n}' +++ cat /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json ++ x='{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-2" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-2jmljgmw-interopoadp" } } } ] , "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-2" } } } ] } }' ++ echo '{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-2" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-2jmljgmw-interopoadp" } } } ] , "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-2" } } } ] } }' ++ grep -o '^[^#]*' + FILE_SETTINGS_NAME=settings.json + printf '\033[34mGenerated settings file under /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json\e[39m\n' Generated settings file under /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json + cat /alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json ++ oc get volumesnapshotclass -o name + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc annotated + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/odf-operator-cephfsplugin-snapclass snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/odf-operator-cephfsplugin-snapclass annotated + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/odf-operator-rbdplugin-snapclass snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/odf-operator-rbdplugin-snapclass annotated ++ ./e2e/must-gather/get-latest-build.sh + oc get configmaps -n default must-gather-image + UPSTREAM_VERSION=99.0.0 ++ oc get OperatorCondition -n openshift-adp -o 'jsonpath={.items[*].metadata.name}' ++ awk -F v '{print $2}' + OADP_VERSION=1.5.3 + '[' -z 1.5.3 ']' + '[' 1.5.3 == 99.0.0 ']' ++ oc get sub redhat-oadp-operator -n openshift-adp -o 'jsonpath={.spec.source}' + OADP_REPO=redhat-operators + '[' -z redhat-operators ']' + '[' redhat-operators == redhat-operators ']' + REGISTRY_PATH=registry.redhat.io/oadp/oadp-mustgather-rhel9: + TAG=1.5.3 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + echo registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 + '[' -z registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 ']' + export NUM_OF_OADP_INSTANCES=1 + NUM_OF_OADP_INSTANCES=1 ++ echo --focus=interop ++ tr ' ' '\n' ++ grep '^--' ++ tr '\n' ' ' + GINKO_PARAM='--focus=interop ' ++ echo --focus=interop ++ tr ' ' '\n' ++ grep '^-' ++ grep -v '^--' ++ tr '\n' ' ' + TEST_PARAM= + ginkgo run --nodes=1 -mod=mod --show-node-events --flake-attempts 3 --junit-report=/logs/artifacts/junit_oadp_interop_results.xml '--label-filter=!// || (// && !exclude_aws && (!/target/ || target_aws) ) ' --focus=interop -p /alabama/cspi/e2e/ -- -credentials_file=/tmp/test-settings/credentials -vsl_credentials_file=/tmp/test-settings/aws_vsl_creds -oadp_namespace=openshift-adp -settings=/alabama/cspi/output_files/api-ci-op-2jmljgmw-2c502-cspilp-interop-ccitredhat-com:6443/settings.json -must_gather_image=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 -timeout_multiplier=2 -skip_dpa_creation=false Ginkgo detected a version mismatch between the Ginkgo CLI and the version of Ginkgo imported by your packages: Ginkgo CLI Version: 2.27.3 Mismatched package versions found: 2.23.4 used by e2e Ginkgo will continue to attempt to run but you may see errors (including flag parsing errors) and should either update your go.mod or your version of the Ginkgo CLI to match. To install the matching version of the CLI run go install github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file. Alternatively you can use go run github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file to invoke the matching version of the Ginkgo CLI. If you are attempting to test multiple packages that each have a different version of the Ginkgo library with a single Ginkgo CLI that is currently unsupported.  2025/12/20 16:46:29 Setting up clients 2025/12/20 16:46:29 Getting default StorageClass... 2025/12/20 16:46:29 Checking default storage class count Run the command: oc get sc 2025/12/20 16:46:29 Got default StorageClass odf-operator-ceph-rbd 2025/12/20 16:46:30 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 70m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 70m odf-operator-ceph-rbd (default) openshift-storage.rbd.csi.ceph.com Delete Immediate true 21m odf-operator-ceph-rbd-virtualization openshift-storage.rbd.csi.ceph.com Delete Immediate true 21m odf-operator-cephfs openshift-storage.cephfs.csi.ceph.com Delete Immediate true 21m openshift-storage.noobaa.io openshift-storage.noobaa.io/obc Delete Immediate false 18m 2025/12/20 16:46:30 Using velero prefix: velero-e2e-6f1c3409-ddc3-11f0-8171-0a580a804e0c 2025/12/20 16:46:30 Checking default storage class count Running Suite: OADP E2E Suite - /alabama/cspi/e2e ================================================= Random Seed: 1766249185 Will run 6 of 193 specs ------------------------------ [SynchronizedBeforeSuite]  /alabama/cspi/e2e/e2e_suite_test.go:85 > Enter [SynchronizedBeforeSuite] TOP-LEVEL @ 12/20/25 16:46:30.217 < Exit [SynchronizedBeforeSuite] TOP-LEVEL @ 12/20/25 16:46:30.232 (15ms) > Enter [SynchronizedBeforeSuite] TOP-LEVEL @ 12/20/25 16:46:30.232 2025/12/20 16:46:30 The VSL credentials file: /tmp/test-settings/aws_vsl_creds doesn't exists 2025/12/20 16:46:30 The error message is: stat /tmp/test-settings/aws_vsl_creds: no such file or directory < Exit [SynchronizedBeforeSuite] TOP-LEVEL @ 12/20/25 16:46:30.294 (63ms) [SynchronizedBeforeSuite] PASSED [0.078 seconds] ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ [skip-disconnected] Restore hooks tests Successful Init hook [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia /alabama/cspi/e2e/hooks/restore_hooks.go:134 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:46:30.295 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:46:30.295 (0s) > Enter [BeforeEach] [skip-disconnected] Restore hooks tests @ 12/20/25 16:46:30.295 < Exit [BeforeEach] [skip-disconnected] Restore hooks tests @ 12/20/25 16:46:30.312 (17ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:46:30.312 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:46:30.312 (0s) > Enter [It] [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia @ 12/20/25 16:46:30.312 2025/12/20 16:46:30 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:46:30.339 2025/12/20 16:46:30 kopia 2025/12/20 16:46:30 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "14df3a3b-1ea1-402e-b620-28f59c7d9291", "resourceVersion": "80243", "generation": 1, "creationTimestamp": "2025-12-20T16:46:30Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:46:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:46:30.43 2025/12/20 16:46:30 Waiting for velero pod to be running 2025/12/20 16:46:35 pod: velero-54467cd776-qxrhx is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:46:31 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:46:34 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:46:30 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:46:30 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:46:30 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] 10.128.2.112 [{10.128.2.112}] 2025-12-20 16:46:30 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:46:31 +0000 UTC,FinishedAt:2025-12-20 16:46:31 +0000 UTC,ContainerID:cri-o://0098c7a7aa10462d03b7e5a8ea3bf3d40ffa0c99bc7c41e36fc7ed80392deb3f,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:212589b9f5b75a043c9a7094cb681a39687795c5e78e625beb6d00960ea57e70 cri-o://0098c7a7aa10462d03b7e5a8ea3bf3d40ffa0c99bc7c41e36fc7ed80392deb3f 0xc00103c439 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-hkvrz /var/run/secrets/kubernetes.io/serviceaccount true 0xc000c33300}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:46:31 +0000 UTC,FinishedAt:2025-12-20 16:46:31 +0000 UTC,ContainerID:cri-o://8d04b9d84a444a3256f269f10e0d22dd5c35c88a93a046c2a4b5bb5aa67f8469,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 cri-o://8d04b9d84a444a3256f269f10e0d22dd5c35c88a93a046c2a4b5bb5aa67f8469 0xc00103c498 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-hkvrz /var/run/secrets/kubernetes.io/serviceaccount true 0xc000c33370}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:46:32 +0000 UTC,FinishedAt:2025-12-20 16:46:32 +0000 UTC,ContainerID:cri-o://b9f4e3228f88a0fe048c42066c153b090629415a8f0cc26b4ae4a621ada00311,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:684020e409da37fd3a41935b51ac22e506fb7a45e787ae8c0cfd6ac0dfcbff09 cri-o://b9f4e3228f88a0fe048c42066c153b090629415a8f0cc26b4ae4a621ada00311 0xc00103c549 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-hkvrz /var/run/secrets/kubernetes.io/serviceaccount true 0xc000c333e0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {hypershift-oadp-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:46:33 +0000 UTC,FinishedAt:2025-12-20 16:46:33 +0000 UTC,ContainerID:cri-o://2d947454e757f7b18be054efad101520e28670eb85f17117854807e6875d6932,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:5d9c2ff91b5851f486fdcadcf9991007cdbe7a81f57774c6d0b3798091316af6 cri-o://2d947454e757f7b18be054efad101520e28670eb85f17117854807e6875d6932 0xc00103c5a8 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-hkvrz /var/run/secrets/kubernetes.io/serviceaccount true 0xc000c33450}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc00103c65f map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc000c33460} {kube-api-access-hkvrz /var/run/secrets/kubernetes.io/serviceaccount true 0xc000c33470}] nil []}] Burstable [] []} 2025/12/20 16:46:40 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:46:40 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:46:40.526 2025/12/20 16:46:40 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql-hooks-e2e @ 12/20/25 16:46:40.613 2025/12/20 16:46:40 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:46:40 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:46:43 2025-12-20 16:46:42,272 p=27208 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:46:42,272 p=27208 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:42,573 p=27208 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:46:42,573 p=27208 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:42,872 p=27208 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:46:42,872 p=27208 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:43,173 p=27208 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:46:43,173 p=27208 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:43,190 p=27208 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:46:43,191 p=27208 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:46:43,191 p=27208 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:43,210 p=27208 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:46:43,211 p=27208 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:43,225 p=27208 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:46:43,226 p=27208 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:46:43,600 p=27208 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:46:43,600 p=27208 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:46:43,600 p=27208 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:46:43,601 p=27208 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:43,622 p=27208 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:46:43,623 p=27208 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:46:43,623 p=27208 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:46:43,624 p=27208 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:46:43,624 p=27208 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-164 2025/12/20 16:46:43 2025/12/20 16:46:43 ❌ Operation 'Deploy application ocp-mysql' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-164","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:46:43.832 < Exit [It] [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia @ 12/20/25 16:46:43.832 (13.52s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:46:43.832 2025/12/20 16:46:43 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:46:43.832 2025/12/20 16:46:43 The failed spec name is: [skip-disconnected] Restore hooks tests Successful Init hook [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:46:43.832 2025/12/20 16:46:43 The folder logs does not exists, creating new folder with the name: logs STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:46:43.833 2025/12/20 16:46:43 The folder logs/It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia does not exists, creating new folder with the name: logs/It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Run must-gather because the spec failed @ 12/20/25 16:46:43.833 2025/12/20 16:46:43 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:46:43 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:46:54 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:46:54 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:46:54.91 Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:46:54.929 (11.097s) > Enter [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:46:54.929 2025/12/20 16:46:54 Cleaning app 2025/12/20 16:46:54 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:46:57 2025-12-20 16:46:56,581 p=27431 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:46:56,581 p=27431 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:56,884 p=27431 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:46:56,884 p=27431 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:57,184 p=27431 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:46:57,184 p=27431 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:57,488 p=27431 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:46:57,488 p=27431 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:46:57,506 p=27431 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:46:57,506 p=27431 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:46:57,507 p=27431 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:57,527 p=27431 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:46:57,527 p=27431 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:57,542 p=27431 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:46:57,542 p=27431 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:46:57,921 p=27431 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:46:57,922 p=27431 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:46:57,922 p=27431 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:46:57,922 p=27431 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:46:57,943 p=27431 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:46:57,944 p=27431 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:46:57,944 p=27431 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:46:57,945 p=27431 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:46:57,945 p=27431 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-164 2025/12/20 16:46:58 < Exit [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:46:58.144 (3.215s) > Enter [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:46:58.144 2025/12/20 16:46:58 Cleaning setup resources for the backup < Exit [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:46:58.144 (0s) > Enter [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:46:58.144 2025/12/20 16:46:58 Deleting DPA CR < Exit [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:46:58.162 (18ms) Attempt #1 Failed. Retrying ↺ @ 12/20/25 16:46:58.162 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:46:58.162 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:46:58.162 (0s) > Enter [BeforeEach] [skip-disconnected] Restore hooks tests @ 12/20/25 16:46:58.162 < Exit [BeforeEach] [skip-disconnected] Restore hooks tests @ 12/20/25 16:46:58.182 (21ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:46:58.182 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:46:58.182 (0s) > Enter [It] [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia @ 12/20/25 16:46:58.182 2025/12/20 16:46:58 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:46:58.203 2025/12/20 16:46:58 kopia 2025/12/20 16:46:58 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "55083aa3-5cb9-4494-8d4a-add0b13df4a5", "resourceVersion": "80772", "generation": 1, "creationTimestamp": "2025-12-20T16:46:58Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:46:58Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:46:58.306 2025/12/20 16:46:58 Waiting for velero pod to be running 2025/12/20 16:46:58 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:46:58 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "55083aa3-5cb9-4494-8d4a-add0b13df4a5", "resourceVersion": "80772", "generation": 1, "creationTimestamp": "2025-12-20T16:46:58Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:46:58Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:47:03 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:47:03.37 2025/12/20 16:47:03 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql-hooks-e2e @ 12/20/25 16:47:03.414 2025/12/20 16:47:03 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:47:03 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:47:06 2025-12-20 16:47:05,077 p=27610 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:47:05,078 p=27610 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:05,383 p=27610 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:47:05,383 p=27610 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:05,686 p=27610 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:47:05,686 p=27610 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:05,985 p=27610 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:47:05,985 p=27610 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:06,003 p=27610 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:47:06,003 p=27610 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:47:06,003 p=27610 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:06,023 p=27610 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:47:06,023 p=27610 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:06,038 p=27610 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:47:06,038 p=27610 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:47:06,411 p=27610 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:47:06,412 p=27610 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:47:06,412 p=27610 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:47:06,412 p=27610 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:06,434 p=27610 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:47:06,434 p=27610 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:47:06,435 p=27610 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:47:06,435 p=27610 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:47:06,435 p=27610 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-164 2025/12/20 16:47:06 2025/12/20 16:47:06 ❌ Operation 'Deploy application ocp-mysql' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-164","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:47:06.631 < Exit [It] [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia @ 12/20/25 16:47:06.631 (8.449s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:47:06.631 2025/12/20 16:47:06 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:47:06.632 2025/12/20 16:47:06 The failed spec name is: [skip-disconnected] Restore hooks tests Successful Init hook [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:47:06.632 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:47:06.632 STEP: Run must-gather because the spec failed @ 12/20/25 16:47:06.632 2025/12/20 16:47:06 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:47:06 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:47:17 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:47:17 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:47:17.776 The folder logs/It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:47:17.793 (11.161s) > Enter [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:17.793 2025/12/20 16:47:17 Cleaning app 2025/12/20 16:47:17 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:47:20 2025-12-20 16:47:19,450 p=27834 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:47:19,451 p=27834 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:19,752 p=27834 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:47:19,752 p=27834 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:20,052 p=27834 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:47:20,052 p=27834 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:20,354 p=27834 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:47:20,354 p=27834 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:20,371 p=27834 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:47:20,372 p=27834 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:47:20,372 p=27834 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:20,391 p=27834 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:47:20,391 p=27834 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:20,405 p=27834 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:47:20,406 p=27834 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:47:20,775 p=27834 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:47:20,776 p=27834 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:47:20,776 p=27834 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:47:20,776 p=27834 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:20,798 p=27834 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:47:20,798 p=27834 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:47:20,798 p=27834 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:47:20,799 p=27834 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:47:20,799 p=27834 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-164 2025/12/20 16:47:21 < Exit [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:21.01 (3.217s) > Enter [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:21.01 2025/12/20 16:47:21 Cleaning setup resources for the backup < Exit [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:21.01 (0s) > Enter [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:21.01 2025/12/20 16:47:21 Deleting DPA CR < Exit [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:21.028 (18ms) Attempt #2 Failed. Retrying ↺ @ 12/20/25 16:47:21.028 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:47:21.028 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:47:21.028 (0s) > Enter [BeforeEach] [skip-disconnected] Restore hooks tests @ 12/20/25 16:47:21.028 < Exit [BeforeEach] [skip-disconnected] Restore hooks tests @ 12/20/25 16:47:21.056 (28ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:47:21.056 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:47:21.056 (0s) > Enter [It] [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia @ 12/20/25 16:47:21.056 2025/12/20 16:47:21 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:47:21.078 2025/12/20 16:47:21 kopia 2025/12/20 16:47:21 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "1fe077d9-35e0-4a20-b2e4-f0db0f8ef5c4", "resourceVersion": "81320", "generation": 1, "creationTimestamp": "2025-12-20T16:47:21Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:47:21Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:47:21.192 2025/12/20 16:47:21 Waiting for velero pod to be running 2025/12/20 16:47:21 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:47:21 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "1fe077d9-35e0-4a20-b2e4-f0db0f8ef5c4", "resourceVersion": "81320", "generation": 1, "creationTimestamp": "2025-12-20T16:47:21Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:47:21Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:47:26 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:47:26.261 2025/12/20 16:47:26 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql-hooks-e2e @ 12/20/25 16:47:26.307 2025/12/20 16:47:26 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:47:26 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:47:29 2025-12-20 16:47:27,974 p=28013 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:47:27,974 p=28013 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:28,279 p=28013 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:47:28,279 p=28013 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:28,581 p=28013 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:47:28,581 p=28013 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:28,884 p=28013 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:47:28,884 p=28013 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:28,901 p=28013 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:47:28,901 p=28013 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:47:28,902 p=28013 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:28,921 p=28013 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:47:28,921 p=28013 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:28,935 p=28013 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:47:28,936 p=28013 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:47:29,321 p=28013 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:47:29,322 p=28013 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:47:29,322 p=28013 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:47:29,322 p=28013 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:29,344 p=28013 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:47:29,344 p=28013 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:47:29,345 p=28013 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:47:29,345 p=28013 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:47:29,345 p=28013 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-164 2025/12/20 16:47:29 2025/12/20 16:47:29 ❌ Operation 'Deploy application ocp-mysql' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-164","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:47:29.551 < Exit [It] [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia @ 12/20/25 16:47:29.551 (8.494s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:47:29.551 2025/12/20 16:47:29 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:47:29.551 2025/12/20 16:47:29 The failed spec name is: [skip-disconnected] Restore hooks tests Successful Init hook [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:47:29.551 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:47:29.551 STEP: Run must-gather because the spec failed @ 12/20/25 16:47:29.551 2025/12/20 16:47:29 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:47:29 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:47:40 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:47:40 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:47:40.683 The folder logs/It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:47:40.704 (11.153s) > Enter [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:40.704 2025/12/20 16:47:40 Cleaning app 2025/12/20 16:47:40 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:47:43 2025-12-20 16:47:42,359 p=28237 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:47:42,360 p=28237 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:42,658 p=28237 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:47:42,659 p=28237 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:42,960 p=28237 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:47:42,960 p=28237 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:43,258 p=28237 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:47:43,259 p=28237 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:43,276 p=28237 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:47:43,277 p=28237 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:47:43,277 p=28237 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:43,296 p=28237 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:47:43,296 p=28237 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:43,311 p=28237 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:47:43,311 p=28237 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:47:43,685 p=28237 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:47:43,685 p=28237 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:47:43,685 p=28237 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:47:43,685 p=28237 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:43,706 p=28237 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:47:43,707 p=28237 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:47:43,707 p=28237 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:47:43,708 p=28237 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:47:43,708 p=28237 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-164 2025/12/20 16:47:43 < Exit [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:43.914 (3.21s) > Enter [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:43.914 2025/12/20 16:47:43 Cleaning setup resources for the backup < Exit [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:43.914 (0s) > Enter [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:43.914 2025/12/20 16:47:43 Deleting DPA CR < Exit [DeferCleanup (Each)] Successful Init hook @ 12/20/25 16:47:43.932 (17ms) • [FAILED] [73.636 seconds] [skip-disconnected] Restore hooks tests Successful Init hook [It] [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia /alabama/cspi/e2e/hooks/restore_hooks.go:134 [FAILED] Unexpected error: <*errors.Error | 0xc000c0de40>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-164","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"test-oadp-164\",\"non_admin_user\":false,\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:47:29.551 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ Backup restore tests Application backup [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia [mr-check] /alabama/cspi/e2e/app_backup/backup_restore.go:48 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:47:43.932 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:47:43.932 (0s) > Enter [BeforeEach] Backup restore tests @ 12/20/25 16:47:43.932 < Exit [BeforeEach] Backup restore tests @ 12/20/25 16:47:43.954 (22ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:47:43.954 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:47:43.954 (0s) > Enter [It] [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia @ 12/20/25 16:47:43.954 2025/12/20 16:47:43 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:47:43.976 2025/12/20 16:47:43 kopia 2025/12/20 16:47:44 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "88a63a7c-cfe8-49fd-8466-194f6d8f1c98", "resourceVersion": "81828", "generation": 1, "creationTimestamp": "2025-12-20T16:47:44Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:47:44Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:47:44.067 2025/12/20 16:47:44 Waiting for velero pod to be running 2025/12/20 16:47:44 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:47:44 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "88a63a7c-cfe8-49fd-8466-194f6d8f1c98", "resourceVersion": "81828", "generation": 1, "creationTimestamp": "2025-12-20T16:47:44Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:47:44Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:47:49 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:47:49.13 2025/12/20 16:47:49 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 12/20/25 16:47:49.173 2025/12/20 16:47:49 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:47:49 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:47:52 2025-12-20 16:47:50,860 p=28417 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:47:50,860 p=28417 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:51,162 p=28417 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:47:51,162 p=28417 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:51,481 p=28417 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:47:51,481 p=28417 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:51,788 p=28417 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:47:51,788 p=28417 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:47:51,806 p=28417 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:47:51,806 p=28417 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:47:51,806 p=28417 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:51,826 p=28417 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:47:51,827 p=28417 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:51,841 p=28417 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:47:51,842 p=28417 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:47:52,218 p=28417 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:47:52,218 p=28417 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:47:52,218 p=28417 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:47:52,219 p=28417 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:47:52,240 p=28417 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:47:52,241 p=28417 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:47:52,241 p=28417 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:47:52,242 p=28417 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:47:52,242 p=28417 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-437-kopia 2025/12/20 16:47:52 2025/12/20 16:47:52 ❌ Operation 'Deploy application ocp-mysql' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-437-kopia","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:47:52.45 < Exit [It] [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia @ 12/20/25 16:47:52.45 (8.496s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:47:52.45 2025/12/20 16:47:52 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:47:52.45 2025/12/20 16:47:52 The failed spec name is: Backup restore tests Application backup [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:47:52.45 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:47:52.45 2025/12/20 16:47:52 The folder logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check does not exists, creating new folder with the name: logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check STEP: Run must-gather because the spec failed @ 12/20/25 16:47:52.45 2025/12/20 16:47:52 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:47:52 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:48:03 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:48:03 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:48:03 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:48:03.504 Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:48:03.522 (11.073s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:03.522 2025/12/20 16:48:03 Cleaning app 2025/12/20 16:48:03 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:48:06 2025-12-20 16:48:05,220 p=28640 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:48:05,220 p=28640 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:05,535 p=28640 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:48:05,535 p=28640 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:05,848 p=28640 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:48:05,848 p=28640 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:06,153 p=28640 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:48:06,153 p=28640 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:06,171 p=28640 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:48:06,171 p=28640 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:48:06,171 p=28640 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:06,191 p=28640 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:48:06,192 p=28640 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:06,206 p=28640 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:48:06,207 p=28640 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:48:06,605 p=28640 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:48:06,605 p=28640 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:48:06,606 p=28640 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:48:06,606 p=28640 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:06,627 p=28640 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:48:06,628 p=28640 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:48:06,628 p=28640 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:48:06,628 p=28640 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:48:06,629 p=28640 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-437-kopia 2025/12/20 16:48:06 < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:06.838 (3.316s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:06.838 2025/12/20 16:48:06 Cleaning setup resources for the backup < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:06.839 (0s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:06.839 2025/12/20 16:48:06 Deleting DPA CR < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:06.859 (20ms) Attempt #1 Failed. Retrying ↺ @ 12/20/25 16:48:06.859 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:48:06.859 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:48:06.859 (0s) > Enter [BeforeEach] Backup restore tests @ 12/20/25 16:48:06.859 < Exit [BeforeEach] Backup restore tests @ 12/20/25 16:48:06.882 (23ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:48:06.882 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:48:06.882 (0s) > Enter [It] [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia @ 12/20/25 16:48:06.882 2025/12/20 16:48:06 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:48:06.907 2025/12/20 16:48:06 kopia 2025/12/20 16:48:06 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "6247ce68-ed31-42f4-86ad-99c83093bc33", "resourceVersion": "82338", "generation": 1, "creationTimestamp": "2025-12-20T16:48:06Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:48:06Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:48:06.998 2025/12/20 16:48:06 Waiting for velero pod to be running 2025/12/20 16:48:07 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:48:07 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "6247ce68-ed31-42f4-86ad-99c83093bc33", "resourceVersion": "82338", "generation": 1, "creationTimestamp": "2025-12-20T16:48:06Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:48:06Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:48:12 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:48:12.06 2025/12/20 16:48:12 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 12/20/25 16:48:12.103 2025/12/20 16:48:12 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:48:12 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:48:15 2025-12-20 16:48:13,792 p=28819 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:48:13,792 p=28819 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:14,119 p=28819 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:48:14,120 p=28819 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:14,435 p=28819 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:48:14,435 p=28819 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:14,738 p=28819 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:48:14,739 p=28819 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:14,756 p=28819 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:48:14,756 p=28819 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:48:14,756 p=28819 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:14,775 p=28819 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:48:14,776 p=28819 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:14,790 p=28819 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:48:14,791 p=28819 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:48:15,177 p=28819 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:48:15,177 p=28819 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:48:15,178 p=28819 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:48:15,178 p=28819 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:15,200 p=28819 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:48:15,201 p=28819 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:48:15,201 p=28819 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:48:15,202 p=28819 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:48:15,202 p=28819 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-437-kopia 2025/12/20 16:48:15 2025/12/20 16:48:15 ❌ Operation 'Deploy application ocp-mysql' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-437-kopia","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:48:15.448 < Exit [It] [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia @ 12/20/25 16:48:15.448 (8.567s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:48:15.448 2025/12/20 16:48:15 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:48:15.448 2025/12/20 16:48:15 The failed spec name is: Backup restore tests Application backup [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:48:15.448 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:48:15.448 STEP: Run must-gather because the spec failed @ 12/20/25 16:48:15.448 2025/12/20 16:48:15 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:48:15 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:48:26 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:48:26 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:48:26 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:48:26.67 The folder logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:48:26.687 (11.239s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:26.687 2025/12/20 16:48:26 Cleaning app 2025/12/20 16:48:26 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:48:29 2025-12-20 16:48:28,400 p=29043 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:48:28,400 p=29043 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:28,710 p=29043 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:48:28,710 p=29043 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:29,015 p=29043 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:48:29,015 p=29043 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:29,338 p=29043 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:48:29,339 p=29043 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:29,356 p=29043 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:48:29,356 p=29043 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:48:29,356 p=29043 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:29,376 p=29043 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:48:29,376 p=29043 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:29,391 p=29043 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:48:29,392 p=29043 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:48:29,788 p=29043 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:48:29,789 p=29043 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:48:29,789 p=29043 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:48:29,789 p=29043 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:29,811 p=29043 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:48:29,811 p=29043 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:48:29,812 p=29043 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:48:29,812 p=29043 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:48:29,812 p=29043 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-437-kopia 2025/12/20 16:48:30 < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:30.01 (3.323s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:30.01 2025/12/20 16:48:30 Cleaning setup resources for the backup < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:30.01 (0s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:30.011 2025/12/20 16:48:30 Deleting DPA CR < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:48:30.029 (18ms) Attempt #2 Failed. Retrying ↺ @ 12/20/25 16:48:30.029 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:48:30.029 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:48:30.029 (0s) > Enter [BeforeEach] Backup restore tests @ 12/20/25 16:48:30.029 < Exit [BeforeEach] Backup restore tests @ 12/20/25 16:48:30.05 (21ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:48:30.05 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:48:30.05 (0s) > Enter [It] [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia @ 12/20/25 16:48:30.05 2025/12/20 16:48:30 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:48:30.088 2025/12/20 16:48:30 kopia 2025/12/20 16:48:30 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "8079e886-6e16-4f29-915b-b9cf5dada651", "resourceVersion": "82835", "generation": 1, "creationTimestamp": "2025-12-20T16:48:30Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:48:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:48:30.197 2025/12/20 16:48:30 Waiting for velero pod to be running 2025/12/20 16:48:30 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:48:30 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "8079e886-6e16-4f29-915b-b9cf5dada651", "resourceVersion": "82835", "generation": 1, "creationTimestamp": "2025-12-20T16:48:30Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:48:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:48:35 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:48:35.262 2025/12/20 16:48:35 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 12/20/25 16:48:35.31 2025/12/20 16:48:35 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:48:35 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:48:38 2025-12-20 16:48:37,024 p=29219 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:48:37,025 p=29219 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:37,339 p=29219 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:48:37,339 p=29219 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:37,643 p=29219 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:48:37,643 p=29219 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:37,947 p=29219 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:48:37,948 p=29219 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:48:37,965 p=29219 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:48:37,965 p=29219 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:48:37,966 p=29219 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:37,984 p=29219 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:48:37,984 p=29219 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:37,999 p=29219 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:48:37,999 p=29219 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:48:38,379 p=29219 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:48:38,379 p=29219 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:48:38,380 p=29219 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:48:38,380 p=29219 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:48:38,401 p=29219 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:48:38,401 p=29219 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:48:38,402 p=29219 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:48:38,402 p=29219 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:48:38,402 p=29219 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-437-kopia 2025/12/20 16:48:38 2025/12/20 16:48:38 Found server error pattern in ansible logs: '401' 2025/12/20 16:48:38 Detected server errors in ansible logs - treating as retryable 2025/12/20 16:48:38 ⚠️ Operation 'Deploy application ocp-mysql' failed with retryable server error (attempt 1/3): Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-437-kopia","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 2025/12/20 16:48:38 🕐 Waiting 1m0s before retry... 2025/12/20 16:49:38 🔄 [Attempt 2/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:49:38 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:49:41 2025-12-20 16:49:40,283 p=29401 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:49:40,283 p=29401 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:49:40,594 p=29401 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:49:40,594 p=29401 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:49:40,912 p=29401 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:49:40,912 p=29401 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:49:41,216 p=29401 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:49:41,216 p=29401 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:49:41,234 p=29401 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:49:41,235 p=29401 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:49:41,235 p=29401 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:49:41,254 p=29401 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:49:41,254 p=29401 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:49:41,269 p=29401 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:49:41,270 p=29401 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:49:41,663 p=29401 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:49:41,663 p=29401 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:49:41,663 p=29401 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:49:41,663 p=29401 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:49:41,686 p=29401 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:49:41,687 p=29401 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:49:41,687 p=29401 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:49:41,688 p=29401 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:49:41,688 p=29401 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-437-kopia 2025/12/20 16:49:41 2025/12/20 16:49:41 Found server error pattern in ansible logs: '401' 2025/12/20 16:49:41 Detected server errors in ansible logs - treating as retryable 2025/12/20 16:49:41 ⚠️ Operation 'Deploy application ocp-mysql' failed with retryable server error (attempt 2/3): Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-437-kopia","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 2025/12/20 16:49:41 🕐 Waiting 2m0s before retry... 2025/12/20 16:51:41 🔄 [Attempt 3/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:51:41 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:51:45 2025-12-20 16:51:43,623 p=29580 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:51:43,623 p=29580 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:51:43,933 p=29580 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:51:43,933 p=29580 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:51:44,241 p=29580 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:51:44,242 p=29580 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:51:44,557 p=29580 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:51:44,557 p=29580 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:51:44,575 p=29580 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:51:44,575 p=29580 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:51:44,575 p=29580 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:51:44,594 p=29580 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:51:44,595 p=29580 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:51:44,609 p=29580 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:51:44,609 p=29580 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:51:45,009 p=29580 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:51:45,009 p=29580 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:51:45,009 p=29580 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:51:45,009 p=29580 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:51:45,031 p=29580 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:51:45,032 p=29580 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:51:45,032 p=29580 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:51:45,032 p=29580 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:51:45,032 p=29580 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-437-kopia 2025/12/20 16:51:45 2025/12/20 16:51:45 ❌ Operation 'Deploy application ocp-mysql' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-437-kopia","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:51:45.276 < Exit [It] [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia @ 12/20/25 16:51:45.276 (3m15.226s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:51:45.276 2025/12/20 16:51:45 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:51:45.276 2025/12/20 16:51:45 The failed spec name is: Backup restore tests Application backup [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:51:45.276 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:51:45.276 STEP: Run must-gather because the spec failed @ 12/20/25 16:51:45.276 2025/12/20 16:51:45 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:51:45 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:51:56 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:51:56 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:51:56 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:51:56.576 The folder logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:51:56.619 (11.343s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:51:56.619 2025/12/20 16:51:56 Cleaning app 2025/12/20 16:51:56 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:52:00 2025-12-20 16:51:58,449 p=29802 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:51:58,449 p=29802 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:51:58,905 p=29802 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:51:58,906 p=29802 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:51:59,342 p=29802 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:51:59,342 p=29802 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:51:59,792 p=29802 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:51:59,792 p=29802 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:51:59,813 p=29802 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:51:59,813 p=29802 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:51:59,813 p=29802 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:51:59,842 p=29802 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:51:59,842 p=29802 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:51:59,865 p=29802 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:51:59,865 p=29802 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:52:00,360 p=29802 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:52:00,361 p=29802 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:52:00,361 p=29802 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:52:00,361 p=29802 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:00,393 p=29802 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:52:00,394 p=29802 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:52:00,394 p=29802 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:52:00,395 p=29802 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:52:00,395 p=29802 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-437-kopia 2025/12/20 16:52:00 < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:00.66 (4.04s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:00.66 2025/12/20 16:52:00 Cleaning setup resources for the backup < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:00.66 (0s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:00.66 2025/12/20 16:52:00 Deleting DPA CR < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:00.68 (20ms) • [FAILED] [256.748 seconds] Backup restore tests Application backup [It] [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia [mr-check] /alabama/cspi/e2e/app_backup/backup_restore.go:48 [FAILED] Unexpected error: <*errors.Error | 0xc00158a180>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-437-kopia","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"test-oadp-437-kopia\",\"non_admin_user\":false,\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:51:45.276 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ S ------------------------------ Backup restore tests Application backup [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI [exclude_aro-4] /alabama/cspi/e2e/app_backup/backup_restore.go:78 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:52:00.68 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:52:00.68 (0s) > Enter [BeforeEach] Backup restore tests @ 12/20/25 16:52:00.68 < Exit [BeforeEach] Backup restore tests @ 12/20/25 16:52:00.697 (16ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:52:00.697 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:52:00.697 (0s) > Enter [It] [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI @ 12/20/25 16:52:00.697 2025/12/20 16:52:00 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:52:00.73 2025/12/20 16:52:00 csi 2025/12/20 16:52:00 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "0269db22-9cc6-4981-b44d-b44f738163a4", "resourceVersion": "86001", "generation": 1, "creationTimestamp": "2025-12-20T16:52:00Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:52:00Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:52:00.865 2025/12/20 16:52:00 Waiting for velero pod to be running 2025/12/20 16:52:00 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:52:00 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "0269db22-9cc6-4981-b44d-b44f738163a4", "resourceVersion": "86001", "generation": 1, "creationTimestamp": "2025-12-20T16:52:00Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:52:00Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:52:05 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:52:05.936 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2025/12/20 16:52:06 The 'openshift-storage' namespace exists 2025/12/20 16:52:06 Checking default storage class count 2025/12/20 16:52:06 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2025/12/20 16:52:06 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:52:06 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:52:06 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case django-persistent @ 12/20/25 16:52:06.386 2025/12/20 16:52:06 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-django 2025/12/20 16:52:06 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:52:10 2025-12-20 16:52:08,573 p=29972 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:52:08,573 p=29972 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:08,986 p=29972 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:52:08,987 p=29972 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:09,414 p=29972 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:52:09,414 p=29972 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:09,823 p=29972 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:52:09,824 p=29972 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:09,842 p=29972 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:52:09,843 p=29972 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:52:09,843 p=29972 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:09,867 p=29972 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:52:09,867 p=29972 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:09,885 p=29972 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:52:09,885 p=29972 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:52:10,387 p=29972 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:52:10,387 p=29972 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:52:10,388 p=29972 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:52:10,388 p=29972 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:10,414 p=29972 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:52:10,415 p=29972 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:52:10,415 p=29972 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:52:10,416 p=29972 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:52:10,416 p=29972 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-122 2025/12/20 16:52:10 2025/12/20 16:52:10 ❌ Operation 'Deploy application ocp-django' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-122","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:52:10.706 < Exit [It] [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI @ 12/20/25 16:52:10.706 (10.009s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:52:10.706 2025/12/20 16:52:10 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:52:10.706 2025/12/20 16:52:10 The failed spec name is: Backup restore tests Application backup [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:52:10.706 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:52:10.706 2025/12/20 16:52:10 The folder logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 does not exists, creating new folder with the name: logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 STEP: Run must-gather because the spec failed @ 12/20/25 16:52:10.706 2025/12/20 16:52:10 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:52:10 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:52:21 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:52:21 It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 2025/12/20 16:52:21 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:52:21 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:52:21.79 Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:52:21.808 (11.102s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:21.809 2025/12/20 16:52:21 Reset number of visits to 0 2025/12/20 16:52:21 Cleaning app 2025/12/20 16:52:21 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:52:24 2025-12-20 16:52:23,510 p=30164 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:52:23,510 p=30164 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:23,839 p=30164 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:52:23,840 p=30164 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:24,154 p=30164 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:52:24,155 p=30164 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:24,469 p=30164 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:52:24,469 p=30164 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:24,487 p=30164 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:52:24,487 p=30164 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:52:24,487 p=30164 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:24,506 p=30164 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:52:24,506 p=30164 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:24,521 p=30164 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:52:24,521 p=30164 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:52:24,891 p=30164 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:52:24,892 p=30164 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:52:24,892 p=30164 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:52:24,892 p=30164 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:24,913 p=30164 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:52:24,913 p=30164 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:52:24,914 p=30164 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:52:24,914 p=30164 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:52:24,914 p=30164 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-122 2025/12/20 16:52:25 < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:25.114 (3.306s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:25.115 2025/12/20 16:52:25 Cleaning setup resources for the backup 2025/12/20 16:52:25 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:52:25 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:52:25 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:25.173 (58ms) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:25.173 2025/12/20 16:52:25 Deleting DPA CR < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:25.197 (24ms) Attempt #1 Failed. Retrying ↺ @ 12/20/25 16:52:25.197 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:52:25.197 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:52:25.197 (0s) > Enter [BeforeEach] Backup restore tests @ 12/20/25 16:52:25.197 < Exit [BeforeEach] Backup restore tests @ 12/20/25 16:52:25.223 (26ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:52:25.223 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:52:25.223 (0s) > Enter [It] [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI @ 12/20/25 16:52:25.223 2025/12/20 16:52:25 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:52:25.253 2025/12/20 16:52:25 csi 2025/12/20 16:52:25 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "a4bc43e7-4d7f-478f-b837-eb07dc18bf6e", "resourceVersion": "86539", "generation": 1, "creationTimestamp": "2025-12-20T16:52:25Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:52:25Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:52:25.343 2025/12/20 16:52:25 Waiting for velero pod to be running 2025/12/20 16:52:25 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:52:25 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "a4bc43e7-4d7f-478f-b837-eb07dc18bf6e", "resourceVersion": "86539", "generation": 1, "creationTimestamp": "2025-12-20T16:52:25Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:52:25Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:52:30 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:52:30.394 2025/12/20 16:52:30 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:52:30 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:52:30 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case django-persistent @ 12/20/25 16:52:30.656 2025/12/20 16:52:30 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-django 2025/12/20 16:52:30 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:52:33 2025-12-20 16:52:32,322 p=30341 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:52:32,323 p=30341 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:32,630 p=30341 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:52:32,630 p=30341 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:32,934 p=30341 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:52:32,934 p=30341 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:33,242 p=30341 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:52:33,242 p=30341 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:33,260 p=30341 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:52:33,260 p=30341 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:52:33,260 p=30341 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:33,280 p=30341 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:52:33,280 p=30341 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:33,295 p=30341 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:52:33,295 p=30341 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:52:33,676 p=30341 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:52:33,677 p=30341 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:52:33,677 p=30341 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:52:33,677 p=30341 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:33,698 p=30341 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:52:33,698 p=30341 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:52:33,699 p=30341 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:52:33,699 p=30341 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:52:33,699 p=30341 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-122 2025/12/20 16:52:33 2025/12/20 16:52:33 ❌ Operation 'Deploy application ocp-django' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-122","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:52:33.913 < Exit [It] [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI @ 12/20/25 16:52:33.913 (8.69s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:52:33.913 2025/12/20 16:52:33 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:52:33.913 2025/12/20 16:52:33 The failed spec name is: Backup restore tests Application backup [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:52:33.913 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:52:33.913 STEP: Run must-gather because the spec failed @ 12/20/25 16:52:33.913 2025/12/20 16:52:33 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:52:33 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:52:45 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:52:45 It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 2025/12/20 16:52:45 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:52:45 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:52:45.244 The folder logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:52:45.262 (11.349s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:45.262 2025/12/20 16:52:45 Reset number of visits to 0 2025/12/20 16:52:45 Cleaning app 2025/12/20 16:52:45 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:52:48 2025-12-20 16:52:47,179 p=30563 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:52:47,179 p=30563 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:47,544 p=30563 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:52:47,545 p=30563 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:47,944 p=30563 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:52:47,944 p=30563 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:48,344 p=30563 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:52:48,344 p=30563 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:48,363 p=30563 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:52:48,364 p=30563 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:52:48,364 p=30563 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:48,383 p=30563 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:52:48,383 p=30563 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:48,398 p=30563 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:52:48,398 p=30563 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:52:48,786 p=30563 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:52:48,787 p=30563 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:52:48,787 p=30563 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:52:48,787 p=30563 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:48,809 p=30563 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:52:48,809 p=30563 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:52:48,809 p=30563 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:52:48,810 p=30563 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:52:48,810 p=30563 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-122 2025/12/20 16:52:49 < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:49.003 (3.741s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:49.003 2025/12/20 16:52:49 Cleaning setup resources for the backup 2025/12/20 16:52:49 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:52:49 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:52:49 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:49.067 (65ms) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:49.067 2025/12/20 16:52:49 Deleting DPA CR < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:52:49.091 (24ms) Attempt #2 Failed. Retrying ↺ @ 12/20/25 16:52:49.091 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:52:49.091 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:52:49.091 (0s) > Enter [BeforeEach] Backup restore tests @ 12/20/25 16:52:49.091 < Exit [BeforeEach] Backup restore tests @ 12/20/25 16:52:49.118 (26ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:52:49.118 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:52:49.118 (0s) > Enter [It] [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI @ 12/20/25 16:52:49.118 2025/12/20 16:52:49 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:52:49.158 2025/12/20 16:52:49 csi 2025/12/20 16:52:49 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "ce50ef04-8e23-4e24-b326-3e07329c8aec", "resourceVersion": "87022", "generation": 1, "creationTimestamp": "2025-12-20T16:52:49Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:52:49Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:52:49.256 2025/12/20 16:52:49 Waiting for velero pod to be running 2025/12/20 16:52:49 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:52:49 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "ce50ef04-8e23-4e24-b326-3e07329c8aec", "resourceVersion": "87022", "generation": 1, "creationTimestamp": "2025-12-20T16:52:49Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:52:49Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:52:54 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:52:54.311 2025/12/20 16:52:54 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:52:54 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:52:54 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd STEP: Installing application for case django-persistent @ 12/20/25 16:52:54.49 2025/12/20 16:52:54 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-django 2025/12/20 16:52:54 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:52:57 2025-12-20 16:52:56,149 p=30738 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:52:56,149 p=30738 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:56,452 p=30738 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:52:56,452 p=30738 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:56,758 p=30738 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:52:56,758 p=30738 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:57,065 p=30738 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:52:57,066 p=30738 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:52:57,083 p=30738 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:52:57,083 p=30738 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:52:57,083 p=30738 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:57,102 p=30738 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:52:57,103 p=30738 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:57,117 p=30738 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:52:57,117 p=30738 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:52:57,498 p=30738 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:52:57,499 p=30738 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:52:57,499 p=30738 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:52:57,499 p=30738 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:52:57,521 p=30738 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:52:57,522 p=30738 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:52:57,522 p=30738 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:52:57,523 p=30738 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:52:57,523 p=30738 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-122 2025/12/20 16:52:57 2025/12/20 16:52:57 ❌ Operation 'Deploy application ocp-django' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-122","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:52:57.734 < Exit [It] [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI @ 12/20/25 16:52:57.734 (8.616s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:52:57.734 2025/12/20 16:52:57 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:52:57.734 2025/12/20 16:52:57 The failed spec name is: Backup restore tests Application backup [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:52:57.734 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:52:57.734 STEP: Run must-gather because the spec failed @ 12/20/25 16:52:57.734 2025/12/20 16:52:57 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:52:57 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:53:08 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:53:08 It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 2025/12/20 16:53:08 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:53:08 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:53:08.969 The folder logs/It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:53:08.988 (11.254s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:53:08.988 2025/12/20 16:53:08 Reset number of visits to 0 2025/12/20 16:53:08 Cleaning app 2025/12/20 16:53:08 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:53:12 2025-12-20 16:53:10,650 p=30963 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:53:10,651 p=30963 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:10,956 p=30963 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:53:10,956 p=30963 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:11,260 p=30963 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:53:11,260 p=30963 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:11,565 p=30963 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:53:11,566 p=30963 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:11,583 p=30963 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:53:11,584 p=30963 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:53:11,584 p=30963 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:11,603 p=30963 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:53:11,603 p=30963 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:11,618 p=30963 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:53:11,618 p=30963 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:53:11,999 p=30963 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:53:12,000 p=30963 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:53:12,000 p=30963 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:53:12,000 p=30963 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:12,022 p=30963 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:53:12,023 p=30963 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:53:12,023 p=30963 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:53:12,024 p=30963 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:53:12,024 p=30963 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-122 2025/12/20 16:53:12 < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:53:12.229 (3.242s) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:53:12.229 2025/12/20 16:53:12 Cleaning setup resources for the backup 2025/12/20 16:53:12 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:53:12 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:53:12 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:53:12.289 (60ms) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:53:12.289 2025/12/20 16:53:12 Deleting DPA CR < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:53:12.31 (21ms) • [FAILED] [71.630 seconds] Backup restore tests Application backup [It] [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI [exclude_aro-4] /alabama/cspi/e2e/app_backup/backup_restore.go:78 [FAILED] Unexpected error: <*errors.Error | 0xc000aaa240>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-122","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"test-oadp-122\",\"non_admin_user\":false,\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:52:57.734 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ S ------------------------------ Backup restore tests Application backup [tc-id:OADP-352][interop][skip-disconnected][smoke] [Tier0] [vf:volume-snapshot-location] Django application with BSL&VSL [vsl] /alabama/cspi/e2e/app_backup/backup_restore.go:112 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:53:12.311 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:53:12.311 (0s) > Enter [BeforeEach] Backup restore tests @ 12/20/25 16:53:12.311 < Exit [BeforeEach] Backup restore tests @ 12/20/25 16:53:12.326 (16ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:53:12.326 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:53:12.327 (0s) > Enter [It] [tc-id:OADP-352][interop][skip-disconnected][smoke] [Tier0] [vf:volume-snapshot-location] Django application with BSL&VSL @ 12/20/25 16:53:12.327 2025/12/20 16:53:12 Check if VSL custom credentials exist 2025/12/20 16:53:12 Check if the cloud provider is AWS 2025/12/20 16:53:12 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:53:12.372 2025/12/20 16:53:12 vsl 2025/12/20 16:53:12 Check if VSL custom credentials exist 2025/12/20 16:53:12 Check if the cloud provider is AWS 2025/12/20 16:53:12 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "094f14f1-eca2-47e7-ae23-5e92146a3cc8", "resourceVersion": "87537", "generation": 1, "creationTimestamp": "2025-12-20T16:53:12Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:53:12Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-2" } } } ], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:53:12.498 2025/12/20 16:53:12 Waiting for velero pod to be running 2025/12/20 16:53:12 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:53:12 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "094f14f1-eca2-47e7-ae23-5e92146a3cc8", "resourceVersion": "87537", "generation": 1, "creationTimestamp": "2025-12-20T16:53:12Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:53:12Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-2" } } } ], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:53:17 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:53:17.561 2025/12/20 16:53:17 Checking default storage class count [SKIPPED] in [It] - /alabama/cspi/lib/backup.go:408 @ 12/20/25 16:53:17.589 < Exit [It] [tc-id:OADP-352][interop][skip-disconnected][smoke] [Tier0] [vf:volume-snapshot-location] Django application with BSL&VSL @ 12/20/25 16:53:17.589 (5.263s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:53:17.589 2025/12/20 16:53:17 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:53:17.606 (17ms) > Enter [DeferCleanup (Each)] Application backup @ 12/20/25 16:53:17.607 2025/12/20 16:53:17 Deleting DPA CR < Exit [DeferCleanup (Each)] Application backup @ 12/20/25 16:53:17.625 (18ms) S [SKIPPED] [5.314 seconds] Backup restore tests Application backup [It] [tc-id:OADP-352][interop][skip-disconnected][smoke] [Tier0] [vf:volume-snapshot-location] Django application with BSL&VSL [vsl] /alabama/cspi/e2e/app_backup/backup_restore.go:112 [SKIPPED] Skipping VSL test because the default StorageClass provisioner openshift-storage.rbd.csi.ceph.com is not supported In [It] at: /alabama/cspi/lib/backup.go:408 @ 12/20/25 16:53:17.589 ------------------------------ SSSSSSSSSSSSS ------------------------------ Incremental backup restore tests Incremental restore pod count [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update /alabama/cspi/e2e/incremental_restore/backup_restore_incremental.go:94 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:53:17.625 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:53:17.625 (0s) > Enter [BeforeEach] Incremental backup restore tests @ 12/20/25 16:53:17.625 < Exit [BeforeEach] Incremental backup restore tests @ 12/20/25 16:53:17.658 (32ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:53:17.658 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:53:17.658 (0s) > Enter [It] [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update @ 12/20/25 16:53:17.658 2025/12/20 16:53:17 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:53:17.677 2025/12/20 16:53:17 csi 2025/12/20 16:53:17 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "8f22c43e-fff5-4d4e-bbb0-ae715e577256", "resourceVersion": "87709", "generation": 1, "creationTimestamp": "2025-12-20T16:53:17Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:53:17Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:53:17.775 2025/12/20 16:53:17 Waiting for velero pod to be running 2025/12/20 16:53:17 pod: velero-7d4f678c7b-glt2d is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:14 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:17 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:12 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:12 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:12 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] 10.128.2.134 [{10.128.2.134}] 2025-12-20 16:53:12 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:53:13 +0000 UTC,FinishedAt:2025-12-20 16:53:13 +0000 UTC,ContainerID:cri-o://350e2e9f0cacb4c0832003369a7f37be2559b40e69fa50c02e07141b71e7afc9,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:212589b9f5b75a043c9a7094cb681a39687795c5e78e625beb6d00960ea57e70 cri-o://350e2e9f0cacb4c0832003369a7f37be2559b40e69fa50c02e07141b71e7afc9 0xc001188259 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-8tn9r /var/run/secrets/kubernetes.io/serviceaccount true 0xc00128f3d0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:53:14 +0000 UTC,FinishedAt:2025-12-20 16:53:14 +0000 UTC,ContainerID:cri-o://45ae28cded0fc94d6b528122c8dcef8f327d896acfa5765d93df2577ed2f95bb,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 cri-o://45ae28cded0fc94d6b528122c8dcef8f327d896acfa5765d93df2577ed2f95bb 0xc0011882b8 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-8tn9r /var/run/secrets/kubernetes.io/serviceaccount true 0xc00128f440}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:53:15 +0000 UTC,FinishedAt:2025-12-20 16:53:15 +0000 UTC,ContainerID:cri-o://9a9ea3f241db280372401be088f40dadcb0f3e294ad914387d36ebf5bc0117fc,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:684020e409da37fd3a41935b51ac22e506fb7a45e787ae8c0cfd6ac0dfcbff09 cri-o://9a9ea3f241db280372401be088f40dadcb0f3e294ad914387d36ebf5bc0117fc 0xc001188369 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-8tn9r /var/run/secrets/kubernetes.io/serviceaccount true 0xc00128f4b0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {hypershift-oadp-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:53:16 +0000 UTC,FinishedAt:2025-12-20 16:53:16 +0000 UTC,ContainerID:cri-o://a0c17f7e3daec392811c54066ce63a453bf5b6d500f815fc681928120bb769bc,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:5d9c2ff91b5851f486fdcadcf9991007cdbe7a81f57774c6d0b3798091316af6 cri-o://a0c17f7e3daec392811c54066ce63a453bf5b6d500f815fc681928120bb769bc 0xc0011883c8 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-8tn9r /var/run/secrets/kubernetes.io/serviceaccount true 0xc00128f520}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc00118847f map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc00128f530} {cloud-credentials /credentials false } {kube-api-access-8tn9r /var/run/secrets/kubernetes.io/serviceaccount true 0xc00128f540}] nil []}] Burstable [] []} 2025/12/20 16:53:22 pod: velero-5944c96584-nv9w9 is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:19 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:22 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:17 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:17 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:53:17 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] 10.128.2.135 [{10.128.2.135}] 2025-12-20 16:53:17 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:53:18 +0000 UTC,FinishedAt:2025-12-20 16:53:18 +0000 UTC,ContainerID:cri-o://824f3a280f671c5552b3c0b58bb146863c2ea4c1c27f190228443581ab75d50f,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:212589b9f5b75a043c9a7094cb681a39687795c5e78e625beb6d00960ea57e70 cri-o://824f3a280f671c5552b3c0b58bb146863c2ea4c1c27f190228443581ab75d50f 0xc000fdbba9 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-6nt4m /var/run/secrets/kubernetes.io/serviceaccount true 0xc00092f0f0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:53:19 +0000 UTC,FinishedAt:2025-12-20 16:53:19 +0000 UTC,ContainerID:cri-o://b35328b4b33a24c26ff673345d7467882dd1ce10e82c2e6b424d833b922b90b7,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 cri-o://b35328b4b33a24c26ff673345d7467882dd1ce10e82c2e6b424d833b922b90b7 0xc000fdbc08 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-6nt4m /var/run/secrets/kubernetes.io/serviceaccount true 0xc00092f160}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:53:20 +0000 UTC,FinishedAt:2025-12-20 16:53:20 +0000 UTC,ContainerID:cri-o://8ad0180e40e83494db3aed2cb44c879a10b0ca6ce90d505539afdf6402f39835,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:684020e409da37fd3a41935b51ac22e506fb7a45e787ae8c0cfd6ac0dfcbff09 cri-o://8ad0180e40e83494db3aed2cb44c879a10b0ca6ce90d505539afdf6402f39835 0xc000fdbcb9 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-6nt4m /var/run/secrets/kubernetes.io/serviceaccount true 0xc00092f1d0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {hypershift-oadp-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:53:21 +0000 UTC,FinishedAt:2025-12-20 16:53:21 +0000 UTC,ContainerID:cri-o://cfb17cb3ab447d7b54f5f92285ff8463f036049b0f5f62514c8d3a6d98bdcbfa,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:5d9c2ff91b5851f486fdcadcf9991007cdbe7a81f57774c6d0b3798091316af6 cri-o://cfb17cb3ab447d7b54f5f92285ff8463f036049b0f5f62514c8d3a6d98bdcbfa 0xc000fdbd18 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-6nt4m /var/run/secrets/kubernetes.io/serviceaccount true 0xc00092f240}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc000fdbdcf map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc00092f250} {kube-api-access-6nt4m /var/run/secrets/kubernetes.io/serviceaccount true 0xc00092f260}] nil []}] Burstable [] []} 2025/12/20 16:53:27 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:53:27 DPA successfully reconciled STEP: Installing application for case todolist-backup @ 12/20/25 16:53:27.876 2025/12/20 16:53:27 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-todolist-mariadb 2025/12/20 16:53:27 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:53:30 2025-12-20 16:53:29,549 p=31143 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:53:29,549 p=31143 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:29,855 p=31143 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:53:29,855 p=31143 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:30,156 p=31143 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:53:30,156 p=31143 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:30,467 p=31143 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:53:30,467 p=31143 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:30,487 p=31143 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:53:30,488 p=31143 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:53:30,488 p=31143 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:30,511 p=31143 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:53:30,511 p=31143 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:30,526 p=31143 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:53:30,527 p=31143 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:53:30,909 p=31143 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:53:30,910 p=31143 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:53:30,910 p=31143 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:53:30,910 p=31143 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:30,931 p=31143 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:53:30,932 p=31143 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:53:30,932 p=31143 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:53:30,933 p=31143 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:53:30,933 p=31143 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n todolist-mariadb-csi-policy-update 2025/12/20 16:53:31 2025/12/20 16:53:31 ❌ Operation 'Deploy application ocp-todolist-mariadb' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"todolist-mariadb-csi-policy-update","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:53:31.127 < Exit [It] [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update @ 12/20/25 16:53:31.127 (13.469s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:53:31.127 2025/12/20 16:53:31 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:53:31.127 2025/12/20 16:53:31 The failed spec name is: Incremental backup restore tests Incremental restore pod count [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:53:31.127 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:53:31.127 2025/12/20 16:53:31 The folder logs/It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update does not exists, creating new folder with the name: logs/It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update STEP: Run must-gather because the spec failed @ 12/20/25 16:53:31.127 2025/12/20 16:53:31 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:53:31 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:53:42 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:53:42 It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 2025/12/20 16:53:42 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:53:42 It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update 2025/12/20 16:53:42 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:53:42.258 Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:53:42.275 (11.148s) > Enter [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:53:42.275 2025/12/20 16:53:42 Cleaning app 2025/12/20 16:53:42 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:53:45 2025-12-20 16:53:43,974 p=31367 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:53:43,975 p=31367 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:44,284 p=31367 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:53:44,284 p=31367 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:44,601 p=31367 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:53:44,602 p=31367 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:44,914 p=31367 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:53:44,914 p=31367 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:44,932 p=31367 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:53:44,933 p=31367 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:53:44,933 p=31367 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:44,955 p=31367 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:53:44,956 p=31367 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:44,971 p=31367 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:53:44,971 p=31367 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:53:45,368 p=31367 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:53:45,368 p=31367 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:53:45,368 p=31367 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:53:45,369 p=31367 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:45,392 p=31367 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:53:45,393 p=31367 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:53:45,393 p=31367 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:53:45,394 p=31367 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:53:45,394 p=31367 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n todolist-mariadb-csi-policy-update 2025/12/20 16:53:45 < Exit [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:53:45.611 (3.336s) > Enter [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:53:45.611 2025/12/20 16:53:45 Deleting DPA CR < Exit [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:53:45.628 (17ms) Attempt #1 Failed. Retrying ↺ @ 12/20/25 16:53:45.628 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:53:45.628 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:53:45.628 (0s) > Enter [BeforeEach] Incremental backup restore tests @ 12/20/25 16:53:45.628 < Exit [BeforeEach] Incremental backup restore tests @ 12/20/25 16:53:45.647 (19ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:53:45.647 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:53:45.647 (0s) > Enter [It] [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update @ 12/20/25 16:53:45.647 2025/12/20 16:53:45 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:53:45.676 2025/12/20 16:53:45 csi 2025/12/20 16:53:45 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "5640c7dc-e609-4829-8099-556e3fef1152", "resourceVersion": "88183", "generation": 1, "creationTimestamp": "2025-12-20T16:53:45Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:53:45Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:53:45.767 2025/12/20 16:53:45 Waiting for velero pod to be running 2025/12/20 16:53:45 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:53:45 DPA successfully reconciled STEP: Installing application for case todolist-backup @ 12/20/25 16:53:45.841 2025/12/20 16:53:45 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-todolist-mariadb 2025/12/20 16:53:45 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:53:48 2025-12-20 16:53:47,545 p=31547 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:53:47,546 p=31547 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:47,848 p=31547 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:53:47,848 p=31547 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:48,152 p=31547 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:53:48,152 p=31547 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:48,463 p=31547 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:53:48,463 p=31547 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:53:48,481 p=31547 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:53:48,481 p=31547 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:53:48,481 p=31547 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:48,501 p=31547 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:53:48,501 p=31547 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:48,517 p=31547 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:53:48,518 p=31547 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:53:48,908 p=31547 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:53:48,908 p=31547 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:53:48,908 p=31547 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:53:48,909 p=31547 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:53:48,931 p=31547 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:53:48,931 p=31547 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:53:48,932 p=31547 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:53:48,932 p=31547 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:53:48,932 p=31547 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n todolist-mariadb-csi-policy-update 2025/12/20 16:53:49 2025/12/20 16:53:49 ❌ Operation 'Deploy application ocp-todolist-mariadb' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"todolist-mariadb-csi-policy-update","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:53:49.152 < Exit [It] [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update @ 12/20/25 16:53:49.152 (3.504s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:53:49.152 2025/12/20 16:53:49 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:53:49.152 2025/12/20 16:53:49 The failed spec name is: Incremental backup restore tests Incremental restore pod count [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:53:49.152 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:53:49.152 STEP: Run must-gather because the spec failed @ 12/20/25 16:53:49.152 2025/12/20 16:53:49 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:53:49 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:54:00 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:54:00 It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 2025/12/20 16:54:00 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:54:00 It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update 2025/12/20 16:54:00 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:54:00.454 The folder logs/It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:54:00.524 (11.372s) > Enter [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:54:00.524 2025/12/20 16:54:00 Cleaning app 2025/12/20 16:54:00 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:54:03 2025-12-20 16:54:02,198 p=31771 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:54:02,199 p=31771 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:02,507 p=31771 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:54:02,507 p=31771 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:02,821 p=31771 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:54:02,821 p=31771 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:03,127 p=31771 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:54:03,128 p=31771 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:03,145 p=31771 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:54:03,145 p=31771 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:54:03,145 p=31771 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:03,165 p=31771 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:54:03,165 p=31771 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:03,179 p=31771 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:54:03,179 p=31771 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:54:03,564 p=31771 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:54:03,565 p=31771 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:54:03,565 p=31771 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:54:03,565 p=31771 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:03,586 p=31771 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:54:03,587 p=31771 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:54:03,588 p=31771 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:54:03,588 p=31771 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:54:03,588 p=31771 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n todolist-mariadb-csi-policy-update 2025/12/20 16:54:03 < Exit [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:54:03.796 (3.271s) > Enter [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:54:03.796 2025/12/20 16:54:03 Deleting DPA CR < Exit [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:54:03.819 (23ms) Attempt #2 Failed. Retrying ↺ @ 12/20/25 16:54:03.819 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:54:03.819 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:54:03.819 (0s) > Enter [BeforeEach] Incremental backup restore tests @ 12/20/25 16:54:03.819 < Exit [BeforeEach] Incremental backup restore tests @ 12/20/25 16:54:03.853 (34ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:54:03.853 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:54:03.853 (0s) > Enter [It] [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update @ 12/20/25 16:54:03.853 2025/12/20 16:54:03 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:54:03.875 2025/12/20 16:54:03 csi 2025/12/20 16:54:03 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "77284060-5fa0-46b1-b3ef-96d643668f90", "resourceVersion": "88602", "generation": 1, "creationTimestamp": "2025-12-20T16:54:03Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:54:03Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:54:04.021 2025/12/20 16:54:04 Waiting for velero pod to be running 2025/12/20 16:54:04 pod: velero-5944c96584-h4dq2 is not yet running with status: {Pending [] [] [] [] [] Burstable [] []} 2025/12/20 16:54:09 pod: velero-5944c96584-h4dq2 is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:05 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:08 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:04 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:04 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:04 +0000 UTC }] 10.0.22.21 [{10.0.22.21}] 10.128.2.138 [{10.128.2.138}] 2025-12-20 16:54:04 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:54:04 +0000 UTC,FinishedAt:2025-12-20 16:54:04 +0000 UTC,ContainerID:cri-o://63b86b0d81131e6cc85ef389770de7be5afec495f2c479e97fbd7334165ff957,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:9b888c75a1ddf274b1f0aa053ddbbc9acc3270417712456ffa29ef81a883b38a registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:212589b9f5b75a043c9a7094cb681a39687795c5e78e625beb6d00960ea57e70 cri-o://63b86b0d81131e6cc85ef389770de7be5afec495f2c479e97fbd7334165ff957 0xc00103d509 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-rv57q /var/run/secrets/kubernetes.io/serviceaccount true 0xc00062ff50}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:54:05 +0000 UTC,FinishedAt:2025-12-20 16:54:05 +0000 UTC,ContainerID:cri-o://ed1c2a950d6de91f5eddeb270e6e609bfcf9ecef098f37fc88263f5a0439be8c,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:2a099649d3d7101a0b6dff9f6372bc76903ba6ecb6578dbeef8a8b3853f48026 cri-o://ed1c2a950d6de91f5eddeb270e6e609bfcf9ecef098f37fc88263f5a0439be8c 0xc00103d568 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-rv57q /var/run/secrets/kubernetes.io/serviceaccount true 0xc000438320}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:54:06 +0000 UTC,FinishedAt:2025-12-20 16:54:06 +0000 UTC,ContainerID:cri-o://1e1c51b799423eebbe7bf29b31240a782c92d23347e4c905b0eee036665a9c96,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:77d6057b1783872ec65d7d6a88d8a1212c38aacc982c7efd46ed2eb77b5e9e70 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:684020e409da37fd3a41935b51ac22e506fb7a45e787ae8c0cfd6ac0dfcbff09 cri-o://1e1c51b799423eebbe7bf29b31240a782c92d23347e4c905b0eee036665a9c96 0xc00103d619 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-rv57q /var/run/secrets/kubernetes.io/serviceaccount true 0xc0004383f0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []} {hypershift-oadp-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2025-12-20 16:54:07 +0000 UTC,FinishedAt:2025-12-20 16:54:07 +0000 UTC,ContainerID:cri-o://d28f1f18dfb06aa731816c7eab9892957b05c3370837675a5af24b61853ac480,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f8798364f94bb1bf4db264b0439c5a280b40cafa408c47b4160d9cc9c5d87e53 registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:5d9c2ff91b5851f486fdcadcf9991007cdbe7a81f57774c6d0b3798091316af6 cri-o://d28f1f18dfb06aa731816c7eab9892957b05c3370837675a5af24b61853ac480 0xc00103d678 map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{134217728 0} {} BinarySI}] &ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{134217728 0} {} BinarySI},},Claims:[]ResourceClaim{},} [{plugins /target false } {kube-api-access-rv57q /var/run/secrets/kubernetes.io/serviceaccount true 0xc0004384a0}] &ContainerUser{Linux:&LinuxContainerUser{UID:1000740000,GID:0,SupplementalGroups:[0 1000740000],},} []}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc00103d72f map[] nil [{plugins /plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc0004384b0} {kube-api-access-rv57q /var/run/secrets/kubernetes.io/serviceaccount true 0xc0004384d0}] nil []}] Burstable [] []} 2025/12/20 16:54:14 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:54:14 DPA successfully reconciled STEP: Installing application for case todolist-backup @ 12/20/25 16:54:14.202 2025/12/20 16:54:14 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-todolist-mariadb 2025/12/20 16:54:14 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:54:17 2025-12-20 16:54:15,892 p=31949 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:54:15,893 p=31949 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:16,204 p=31949 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:54:16,205 p=31949 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:16,520 p=31949 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:54:16,521 p=31949 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:16,834 p=31949 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:54:16,834 p=31949 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:16,851 p=31949 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:54:16,852 p=31949 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:54:16,852 p=31949 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:16,871 p=31949 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:54:16,871 p=31949 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:16,886 p=31949 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:54:16,886 p=31949 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:54:17,278 p=31949 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:54:17,278 p=31949 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:54:17,278 p=31949 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:54:17,279 p=31949 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:17,300 p=31949 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:54:17,301 p=31949 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:54:17,301 p=31949 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:54:17,302 p=31949 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:54:17,302 p=31949 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n todolist-mariadb-csi-policy-update 2025/12/20 16:54:17 2025/12/20 16:54:17 ❌ Operation 'Deploy application ocp-todolist-mariadb' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"todolist-mariadb-csi-policy-update","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:54:17.525 < Exit [It] [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update @ 12/20/25 16:54:17.525 (13.672s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:54:17.525 2025/12/20 16:54:17 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:54:17.525 2025/12/20 16:54:17 The failed spec name is: Incremental backup restore tests Incremental restore pod count [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:54:17.525 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:54:17.525 STEP: Run must-gather because the spec failed @ 12/20/25 16:54:17.525 2025/12/20 16:54:17 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:54:17 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:54:28 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:54:28 It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 2025/12/20 16:54:28 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:54:28 It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update 2025/12/20 16:54:28 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:54:28.856 The folder logs/It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:54:28.876 (11.351s) > Enter [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:54:28.876 2025/12/20 16:54:28 Cleaning app 2025/12/20 16:54:28 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:54:32 2025-12-20 16:54:30,564 p=32171 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:54:30,564 p=32171 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:30,869 p=32171 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:54:30,869 p=32171 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:31,178 p=32171 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:54:31,178 p=32171 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:31,486 p=32171 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:54:31,486 p=32171 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:31,503 p=32171 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:54:31,503 p=32171 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:54:31,504 p=32171 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:31,523 p=32171 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:54:31,523 p=32171 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:31,537 p=32171 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:54:31,537 p=32171 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:54:31,924 p=32171 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:54:31,924 p=32171 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:54:31,925 p=32171 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:54:31,925 p=32171 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:31,946 p=32171 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:54:31,947 p=32171 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:54:31,948 p=32171 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:54:31,948 p=32171 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:54:31,949 p=32171 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n todolist-mariadb-csi-policy-update 2025/12/20 16:54:32 < Exit [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:54:32.15 (3.274s) > Enter [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:54:32.15 2025/12/20 16:54:32 Deleting DPA CR < Exit [DeferCleanup (Each)] Incremental restore pod count @ 12/20/25 16:54:32.169 (19ms) • [FAILED] [74.544 seconds] Incremental backup restore tests Incremental restore pod count [It] [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update /alabama/cspi/e2e/incremental_restore/backup_restore_incremental.go:94 [FAILED] Unexpected error: <*errors.Error | 0xc000f48380>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"todolist-mariadb-csi-policy-update","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"todolist-mariadb-csi-policy-update\",\"non_admin_user\":false,\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:54:17.525 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ SSSSSS > Enter [ReportAfterEach] [upstream-velero] Credentials suite @ 12/20/25 16:54:32.17 < Exit [ReportAfterEach] [upstream-velero] Credentials suite @ 12/20/25 16:54:32.17 (0s) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ [datamover] DataMover: Backup/Restore stateful application with CSI  [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:34 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:54:32.17 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:54:32.17 (0s) > Enter [BeforeEach] [datamover] DataMover: Backup/Restore stateful application with CSI @ 12/20/25 16:54:32.17 < Exit [BeforeEach] [datamover] DataMover: Backup/Restore stateful application with CSI @ 12/20/25 16:54:32.211 (40ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:54:32.211 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:54:32.211 (0s) > Enter [It] [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application @ 12/20/25 16:54:32.211 2025/12/20 16:54:32 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:54:32.232 2025/12/20 16:54:32 native-datamover 2025/12/20 16:54:32 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "fb932db4-74c8-4748-8133-c3934a1867de", "resourceVersion": "89129", "generation": 1, "creationTimestamp": "2025-12-20T16:54:32Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:54:32Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:54:32.315 2025/12/20 16:54:32 Waiting for velero pod to be running 2025/12/20 16:54:32 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:54:32 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:54:32.373 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2025/12/20 16:54:32 The 'openshift-storage' namespace exists 2025/12/20 16:54:32 Checking default storage class count 2025/12/20 16:54:32 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2025/12/20 16:54:32 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:54:32 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:54:32 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:54:32 Checking for correct number of running NodeAgent pods... 2025/12/20 16:54:32 pod: node-agent-m8rb8 is not yet running with status: {Pending [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:32 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:32 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:32 +0000 UTC ContainersNotReady containers with unready status: [node-agent]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:32 +0000 UTC ContainersNotReady containers with unready status: [node-agent]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-12-20 16:54:32 +0000 UTC }] 10.0.81.187 [{10.0.81.187}] [] 2025-12-20 16:54:32 +0000 UTC [] [{node-agent {&ContainerStateWaiting{Reason:ContainerCreating,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:dd5897b1579599cfa8d8a03a1c8d3cf68e3927c4ba5dee4e33c2a2d4c9e6ed89 0xc000e2c7fa map[] nil [{host-pods /host_pods false } {host-plugins /var/lib/kubelet/plugins false } {scratch /scratch false } {certs /etc/ssl/certs false } {bound-sa-token /var/run/secrets/openshift/serviceaccount true 0xc00140eac0} {credentials /tmp/credentials false } {home-velero /home/velero false } {tmp /tmp false } {kube-api-access-mdlc7 /var/run/secrets/kubernetes.io/serviceaccount true 0xc00140ead0}] nil []}] Burstable [] []} STEP: Installing application for case mysql @ 12/20/25 16:54:37.983 2025/12/20 16:54:37 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:54:37 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:54:41 2025-12-20 16:54:39,667 p=32372 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:54:39,668 p=32372 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:39,972 p=32372 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:54:39,972 p=32372 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:40,283 p=32372 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:54:40,284 p=32372 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:40,588 p=32372 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:54:40,588 p=32372 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:40,606 p=32372 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:54:40,607 p=32372 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:54:40,607 p=32372 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:40,627 p=32372 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:54:40,627 p=32372 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:40,642 p=32372 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:54:40,642 p=32372 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:54:41,028 p=32372 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:54:41,029 p=32372 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:54:41,029 p=32372 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:54:41,029 p=32372 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:41,052 p=32372 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:54:41,052 p=32372 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:54:41,053 p=32372 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:54:41,053 p=32372 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:54:41,053 p=32372 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-439 2025/12/20 16:54:41 2025/12/20 16:54:41 ❌ Operation 'Deploy application ocp-mysql' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-439","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:54:41.254 < Exit [It] [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application @ 12/20/25 16:54:41.254 (9.043s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:54:41.254 2025/12/20 16:54:41 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:54:41.254 2025/12/20 16:54:41 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:54:41.254 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:54:41.254 2025/12/20 16:54:41 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application does not exists, creating new folder with the name: logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application STEP: Run must-gather because the spec failed @ 12/20/25 16:54:41.255 2025/12/20 16:54:41 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:54:41 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:54:52 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:54:52 It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 2025/12/20 16:54:52 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:54:52 It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update 2025/12/20 16:54:52 It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application 2025/12/20 16:54:52 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:54:52.354 Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:54:52.38 (11.126s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:54:52.38 2025/12/20 16:54:52 Cleaning app 2025/12/20 16:54:52 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:54:55 2025-12-20 16:54:54,044 p=32596 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:54:54,044 p=32596 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:54,350 p=32596 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:54:54,350 p=32596 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:54,650 p=32596 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:54:54,651 p=32596 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:54,951 p=32596 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:54:54,951 p=32596 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:54:54,969 p=32596 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:54:54,969 p=32596 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:54:54,969 p=32596 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:54,989 p=32596 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:54:54,989 p=32596 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:55,003 p=32596 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:54:55,003 p=32596 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:54:55,385 p=32596 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:54:55,385 p=32596 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:54:55,386 p=32596 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:54:55,386 p=32596 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:54:55,407 p=32596 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:54:55,408 p=32596 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:54:55,408 p=32596 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:54:55,408 p=32596 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:54:55,409 p=32596 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-439 2025/12/20 16:54:55 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:54:55.616 (3.235s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:54:55.616 2025/12/20 16:54:55 Cleaning setup resources for the backup 2025/12/20 16:54:55 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:54:55 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:54:55 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:54:55.678 (62ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:54:55.678 2025/12/20 16:54:55 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:54:55.698 (20ms) Attempt #1 Failed. Retrying ↺ @ 12/20/25 16:54:55.698 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:54:55.698 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:54:55.698 (0s) > Enter [BeforeEach] [datamover] DataMover: Backup/Restore stateful application with CSI @ 12/20/25 16:54:55.698 < Exit [BeforeEach] [datamover] DataMover: Backup/Restore stateful application with CSI @ 12/20/25 16:54:55.725 (27ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:54:55.725 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:54:55.725 (0s) > Enter [It] [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application @ 12/20/25 16:54:55.725 2025/12/20 16:54:55 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:54:55.763 2025/12/20 16:54:55 native-datamover 2025/12/20 16:54:55 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "66bece7c-dc08-46ec-b724-820ec414ce29", "resourceVersion": "89650", "generation": 1, "creationTimestamp": "2025-12-20T16:54:55Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:54:55Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:54:55.87 2025/12/20 16:54:55 Waiting for velero pod to be running 2025/12/20 16:54:55 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:54:55 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "66bece7c-dc08-46ec-b724-820ec414ce29", "resourceVersion": "89650", "generation": 1, "creationTimestamp": "2025-12-20T16:54:55Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:54:55Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:55:00 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:55:00.919 2025/12/20 16:55:00 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:55:00 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:55:00 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:55:01 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 12/20/25 16:55:01.139 2025/12/20 16:55:01 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:55:01 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:55:04 2025-12-20 16:55:02,809 p=32776 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:55:02,809 p=32776 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:03,111 p=32776 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:55:03,112 p=32776 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:03,419 p=32776 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:55:03,420 p=32776 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:03,719 p=32776 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:55:03,720 p=32776 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:03,737 p=32776 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:55:03,738 p=32776 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:55:03,738 p=32776 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:03,757 p=32776 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:55:03,757 p=32776 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:03,771 p=32776 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:55:03,772 p=32776 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:55:04,154 p=32776 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:55:04,155 p=32776 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:55:04,155 p=32776 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:55:04,155 p=32776 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:04,177 p=32776 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:55:04,178 p=32776 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:55:04,178 p=32776 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:55:04,178 p=32776 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:55:04,179 p=32776 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-439 2025/12/20 16:55:04 2025/12/20 16:55:04 ❌ Operation 'Deploy application ocp-mysql' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-439","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:55:04.389 < Exit [It] [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application @ 12/20/25 16:55:04.389 (8.664s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:55:04.389 2025/12/20 16:55:04 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:55:04.389 2025/12/20 16:55:04 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:55:04.389 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:55:04.389 STEP: Run must-gather because the spec failed @ 12/20/25 16:55:04.389 2025/12/20 16:55:04 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:55:04 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:55:15 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:55:15 It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 2025/12/20 16:55:15 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:55:15 It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update 2025/12/20 16:55:15 It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application 2025/12/20 16:55:15 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:55:15.653 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:55:15.673 (11.284s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:15.673 2025/12/20 16:55:15 Cleaning app 2025/12/20 16:55:15 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:55:18 2025-12-20 16:55:17,328 p=32999 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:55:17,329 p=32999 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:17,631 p=32999 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:55:17,631 p=32999 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:17,938 p=32999 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:55:17,938 p=32999 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:18,245 p=32999 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:55:18,245 p=32999 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:18,263 p=32999 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:55:18,264 p=32999 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:55:18,264 p=32999 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:18,283 p=32999 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:55:18,283 p=32999 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:18,298 p=32999 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:55:18,298 p=32999 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:55:18,679 p=32999 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:55:18,680 p=32999 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:55:18,680 p=32999 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:55:18,680 p=32999 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:18,701 p=32999 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:55:18,702 p=32999 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:55:18,702 p=32999 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:55:18,703 p=32999 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:55:18,703 p=32999 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-439 2025/12/20 16:55:18 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:18.911 (3.238s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:18.911 2025/12/20 16:55:18 Cleaning setup resources for the backup 2025/12/20 16:55:18 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:55:18 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:55:18 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:19.047 (136ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:19.047 2025/12/20 16:55:19 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:19.071 (25ms) Attempt #2 Failed. Retrying ↺ @ 12/20/25 16:55:19.071 > Enter [BeforeEach] TOP-LEVEL @ 12/20/25 16:55:19.071 < Exit [BeforeEach] TOP-LEVEL @ 12/20/25 16:55:19.071 (0s) > Enter [BeforeEach] [datamover] DataMover: Backup/Restore stateful application with CSI @ 12/20/25 16:55:19.071 < Exit [BeforeEach] [datamover] DataMover: Backup/Restore stateful application with CSI @ 12/20/25 16:55:19.09 (19ms) > Enter [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:55:19.09 < Exit [JustBeforeEach] TOP-LEVEL @ 12/20/25 16:55:19.09 (0s) > Enter [It] [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application @ 12/20/25 16:55:19.09 2025/12/20 16:55:19 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 12/20/25 16:55:19.115 2025/12/20 16:55:19 native-datamover 2025/12/20 16:55:19 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "cdf03285-6647-4a1c-8fc3-9b8f86f11deb", "resourceVersion": "90256", "generation": 1, "creationTimestamp": "2025-12-20T16:55:19Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:55:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 12/20/25 16:55:19.27 2025/12/20 16:55:19 Waiting for velero pod to be running 2025/12/20 16:55:19 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2025/12/20 16:55:19 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "cdf03285-6647-4a1c-8fc3-9b8f86f11deb", "resourceVersion": "90256", "generation": 1, "creationTimestamp": "2025-12-20T16:55:19Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2025-12-20T16:55:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {}, "f:disableFsBackup": {} } }, "f:logFormat": {}, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-2jmljgmw-interopoadp", "prefix": "velero-e2e-6f355927-ddc3-11f0-8171-0a580a804e0c" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "hypershift", "csi" ], "disableFsBackup": false }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null, "logFormat": "text" }, "status": {} } 2025/12/20 16:55:24 DPA successfully reconciled STEP: Prepare backup resources, depending on the volumes backup type @ 12/20/25 16:55:24.325 2025/12/20 16:55:24 Snapclass 'example-snapclass' doesn't exist, creating 2025/12/20 16:55:24 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:55:24 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:55:24 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 12/20/25 16:55:24.639 2025/12/20 16:55:24 🔄 [Attempt 1/3] Executing operation: Deploy application ocp-mysql 2025/12/20 16:55:24 Using admin kubeconfig for with_deploy operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:55:27 2025-12-20 16:55:26,302 p=33179 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:55:26,302 p=33179 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:26,605 p=33179 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:55:26,605 p=33179 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:26,910 p=33179 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:55:26,911 p=33179 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:27,218 p=33179 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:55:27,218 p=33179 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:27,236 p=33179 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:55:27,236 p=33179 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:55:27,236 p=33179 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:27,255 p=33179 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:55:27,256 p=33179 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:27,270 p=33179 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:55:27,271 p=33179 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:55:27,655 p=33179 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:55:27,656 p=33179 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:55:27,656 p=33179 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:55:27,656 p=33179 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:27,678 p=33179 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:55:27,679 p=33179 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:55:27,679 p=33179 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:55:27,680 p=33179 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:55:27,680 p=33179 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-439 2025/12/20 16:55:27 2025/12/20 16:55:27 ❌ Operation 'Deploy application ocp-mysql' failed with non-retryable error: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-439","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:55:27.884 < Exit [It] [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application @ 12/20/25 16:55:27.884 (8.794s) > Enter [JustAfterEach] TOP-LEVEL @ 12/20/25 16:55:27.885 2025/12/20 16:55:27 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3 STEP: Get the failed spec name @ 12/20/25 16:55:27.885 2025/12/20 16:55:27 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application STEP: Create a folder for all must-gather files if it doesn't exists already @ 12/20/25 16:55:27.885 STEP: Create a folder for the failed spec if it doesn't exists already @ 12/20/25 16:55:27.885 STEP: Run must-gather because the spec failed @ 12/20/25 16:55:27.885 2025/12/20 16:55:27 Log the present working directory path:- /alabama/cspi/e2e 2025/12/20 16:55:27 [adm must-gather --dest-dir /alabama/cspi/e2e/logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.5.3] 2025/12/20 16:55:39 Log all the files present in /alabama/cspi/e2e/logs directory 2025/12/20 16:55:39 It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 2025/12/20 16:55:39 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check 2025/12/20 16:55:39 It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update 2025/12/20 16:55:39 It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application 2025/12/20 16:55:39 It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia STEP: Find must-gather folder and rename it to a shorter more readable name @ 12/20/25 16:55:39.432 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application/must-gather already exists, skipping renaming the folder Secret deletion flag: true Deleting cloud credentials secret cloud-credentials in namespace openshift-adp < Exit [JustAfterEach] TOP-LEVEL @ 12/20/25 16:55:39.457 (11.572s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:39.457 2025/12/20 16:55:39 Cleaning app 2025/12/20 16:55:39 Using admin kubeconfig for with_cleanup operation: /home/jenkins/.kube/config [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name 'namespace'. Origin:   namespace  PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Print admin kubeconfig path] ********************************************* ok: [localhost] => {  "msg": "Admin KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Print user kubeconfig path] ********************************************** ok: [localhost] => {  "msg": "User KUBECONFIG path: /home/jenkins/.kube/config" } TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint (from admin kubeconfig)] **************************** changed: [localhost] TASK [Get admin token] ********************************************************* changed: [localhost] TASK [Get user token] ********************************************************** changed: [localhost] [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9  35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}"  ^ column 9  TASK [Set core facts (admin + user token)] ************************************* ok: [localhost] TASK [Choose token based on non_admin flag] ************************************ ok: [localhost] TASK [Print token] ************************************************************* ok: [localhost] => {  "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. TASK [Extract Kubernetes minor version from cluster] *************************** ok: [localhost] TASK [Map Kubernetes minor to OCP release] ************************************* [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage.  Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7  53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release  ^ column 7  <<< caused by >>>  Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7  54 55 - name: Map Kubernetes minor to OCP release 56 set_fact:  ^ column 7  <<< caused by >>>  Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26  55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >-  ^ column 26  <<< caused by >>>  Type 'Undefined' is unsupported for variable storage. Origin:   fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} PLAY RECAP ********************************************************************* localhost : ok=12  changed=4  unreachable=0 failed=1  skipped=0 rescued=0 ignored=0 2025/12/20 16:55:42 2025-12-20 16:55:41,108 p=33404 u=1001620000 n=ansible INFO| TASK [Remove all the contents from the file] *********************************** 2025-12-20 16:55:41,108 p=33404 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:41,414 p=33404 u=1001620000 n=ansible INFO| TASK [Get cluster endpoint (from admin kubeconfig)] **************************** 2025-12-20 16:55:41,414 p=33404 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:41,716 p=33404 u=1001620000 n=ansible INFO| TASK [Get admin token] ********************************************************* 2025-12-20 16:55:41,717 p=33404 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:42,020 p=33404 u=1001620000 n=ansible INFO| TASK [Get user token] ********************************************************** 2025-12-20 16:55:42,020 p=33404 u=1001620000 n=ansible INFO| changed: [localhost] 2025-12-20 16:55:42,037 p=33404 u=1001620000 n=ansible WARNING| [WARNING]: Found variable using reserved name 'namespace'. Origin: /alabama/cspi/sample-applications/ansible/main.yml:37:9 35 set_fact: 36 oc_binary: "oc" 37 namespace: "{{ use_role }}" ^ column 9 2025-12-20 16:55:42,038 p=33404 u=1001620000 n=ansible INFO| TASK [Set core facts (admin + user token)] ************************************* 2025-12-20 16:55:42,038 p=33404 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:42,057 p=33404 u=1001620000 n=ansible INFO| TASK [Choose token based on non_admin flag] ************************************ 2025-12-20 16:55:42,057 p=33404 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:42,071 p=33404 u=1001620000 n=ansible INFO| TASK [Print token] ************************************************************* 2025-12-20 16:55:42,071 p=33404 u=1001620000 n=ansible INFO| ok: [localhost] => { "msg": "Token: sha256~gvgMQ6jLIYGrTDoiJUjmTk58QR5zZm-RIPDuGWTGHuQ" } 2025-12-20 16:55:42,448 p=33404 u=1001620000 n=ansible INFO| TASK [Extract Kubernetes minor version from cluster] *************************** 2025-12-20 16:55:42,448 p=33404 u=1001620000 n=ansible WARNING| [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg. 2025-12-20 16:55:42,448 p=33404 u=1001620000 n=ansible INFO| [DEPRECATION WARNING]: The `ansible.module_utils.common._collections_compat` module is deprecated. This feature will be removed from ansible-core version 2.24. Use `collections.abc` from the Python standard library instead. 2025-12-20 16:55:42,448 p=33404 u=1001620000 n=ansible INFO| ok: [localhost] 2025-12-20 16:55:42,470 p=33404 u=1001620000 n=ansible INFO| TASK [Map Kubernetes minor to OCP release] ************************************* 2025-12-20 16:55:42,471 p=33404 u=1001620000 n=ansible ERROR| [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage. Task failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:55:7 53 {{ lookup('k8s', cluster_info='version', validate_certs=false, host=url, api_key=token).kubernetes.minor... 54 55 - name: Map Kubernetes minor to OCP release ^ column 7 <<< caused by >>> Finalization of task args for 'ansible.builtin.set_fact' failed. Origin: /alabama/cspi/sample-applications/ansible/main.yml:56:7 54 55 - name: Map Kubernetes minor to OCP release 56 set_fact: ^ column 7 <<< caused by >>> Error while resolving value for 'cluster_version': Error rendering template. Origin: /alabama/cspi/sample-applications/ansible/main.yml:57:26 55 - name: Map Kubernetes minor to OCP release 56 set_fact: 57 cluster_version: >- ^ column 26 <<< caused by >>> Type 'Undefined' is unsupported for variable storage. Origin: 2025-12-20 16:55:42,471 p=33404 u=1001620000 n=ansible INFO| fatal: [localhost]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'cluster_version': Error rendering template: Type 'Undefined' is unsupported for variable storage."} 2025-12-20 16:55:42,472 p=33404 u=1001620000 n=ansible INFO| PLAY RECAP ********************************************************************* 2025-12-20 16:55:42,472 p=33404 u=1001620000 n=ansible INFO| localhost : ok=12 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-439 2025/12/20 16:55:42 < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:42.678 (3.221s) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:42.678 2025/12/20 16:55:42 Cleaning setup resources for the backup 2025/12/20 16:55:42 Setting new default StorageClass 'odf-operator-ceph-rbd' 2025/12/20 16:55:42 Checking default storage class count Skipping creation of StorageClass The current StorageClass: odf-operator-ceph-rbd matches the new StorageClass: odf-operator-ceph-rbd 2025/12/20 16:55:42 Deleting VolumeSnapshotClass 'example-snapclass' < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:42.738 (60ms) > Enter [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:42.738 2025/12/20 16:55:42 Deleting DPA CR < Exit [DeferCleanup (Each)] TOP-LEVEL @ 12/20/25 16:55:42.765 (27ms) • [FAILED] [70.595 seconds] [datamover] DataMover: Backup/Restore stateful application with CSI  [It] [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:34 [FAILED] Unexpected error: <*errors.Error | 0xc0013a70c0>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"admin_kubeconfig":"/home/jenkins/.kube/config","namespace":"test-oadp-439","non_admin_user":false,"use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql","user_kubeconfig":"/home/jenkins/.kube/config","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"admin_kubeconfig\":\"/home/jenkins/.kube/config\",\"namespace\":\"test-oadp-439\",\"non_admin_user\":false,\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql\",\"user_kubeconfig\":\"/home/jenkins/.kube/config\",\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:49 @ 12/20/25 16:55:27.884 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ [SynchronizedAfterSuite]  /alabama/cspi/e2e/e2e_suite_test.go:229 > Enter [SynchronizedAfterSuite] TOP-LEVEL @ 12/20/25 16:55:42.766 < Exit [SynchronizedAfterSuite] TOP-LEVEL @ 12/20/25 16:55:42.766 (0s) > Enter [SynchronizedAfterSuite] TOP-LEVEL @ 12/20/25 16:55:42.766 < Exit [SynchronizedAfterSuite] TOP-LEVEL @ 12/20/25 16:55:42.766 (0s) [SynchronizedAfterSuite] PASSED [0.000 seconds] ------------------------------ [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report autogenerated by Ginkgo > Enter [ReportAfterSuite] TOP-LEVEL @ 12/20/25 16:55:42.766 < Exit [ReportAfterSuite] TOP-LEVEL @ 12/20/25 16:55:42.776 (10ms) [ReportAfterSuite] PASSED [0.010 seconds] ------------------------------ Summarizing 5 Failures: [FAIL] [skip-disconnected] Restore hooks tests Successful Init hook [It] [tc-id:OADP-164][interop][smoke] [Tier0] MySQL app with Kopia /alabama/cspi/test_common/backup_restore_app_case.go:49 [FAIL] Backup restore tests Application backup [It] [tc-id:OADP-437][interop][smoke] [Tier0] [vf:fsb-kopia] MySQL application with filesystem, Kopia [mr-check] /alabama/cspi/test_common/backup_restore_app_case.go:49 [FAIL] Backup restore tests Application backup [It] [tc-id:OADP-122] [interop] [skip-disconnected] [Tier0] [vf:csi-snapshots] Django application with BSL&CSI [exclude_aro-4] /alabama/cspi/test_common/backup_restore_app_case.go:49 [FAIL] Incremental backup restore tests Incremental restore pod count [It] [tc-id:OADP-165][interop] [Tier0] [vf:restore-policies] Todolist app with CSI - policy: update /alabama/cspi/test_common/backup_restore_app_case.go:49 [FAIL] [datamover] DataMover: Backup/Restore stateful application with CSI  [It] [tc-id:OADP-439][interop] [Tier0] [vf:data-mover] MySQL application /alabama/cspi/test_common/backup_restore_app_case.go:49 Ran 5 of 193 Specs in 552.549 seconds FAIL! -- 0 Passed | 5 Failed | 0 Pending | 188 Skipped --- FAIL: TestOADPE2E (552.68s) FAIL Ginkgo ran 1 suite in 9m16.900219546s Test Suite Failed [must-gather ] OUT 2025-12-20T16:56:12.863907251Z Using must-gather plug-in image: registry.redhat.io/oadp/oadp-mustgather-rhel9:v1.4 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: ce271184-7500-4576-8f03-f3940db7522e ClientVersion: 4.20.8 ClusterVersion: Stable at "4.21.0-0.nightly-2025-12-18-020818" ClusterOperators: clusteroperator/operator-lifecycle-manager is not upgradeable because ClusterServiceVersions blocking minor version upgrades to 4.21.0 or higher: - maximum supported OCP version for openshift-storage/odf-dependencies.v4.19.9-rhodf is 4.20 - maximum supported OCP version for openshift-storage/odf-operator.v4.19.9-rhodf is 4.20 [must-gather ] OUT 2025-12-20T16:56:12.987907953Z namespace/openshift-must-gather-npjpp created [must-gather ] OUT 2025-12-20T16:56:13.018542173Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-js58m created [must-gather ] OUT 2025-12-20T16:56:13.079903504Z pod for plug-in image registry.redhat.io/oadp/oadp-mustgather-rhel9:v1.4 created [must-gather-xp8nl] POD 2025-12-20T16:56:25.338541951Z [disk usage checker] Started [must-gather-xp8nl] POD 2025-12-20T16:56:25.342965857Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather-xp8nl] POD 2025-12-20T16:56:26.155808320Z W1220 16:56:26.155752 4 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather-xp8nl] POD 2025-12-20T16:56:26.193902064Z W1220 16:56:26.193864 4 warnings.go:70] kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2. [must-gather-xp8nl] POD 2025-12-20T16:56:26.653766717Z W1220 16:56:26.653709 4 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather-xp8nl] POD 2025-12-20T16:56:28.574856447Z W1220 16:56:28.574793 4 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather-xp8nl] POD 2025-12-20T16:56:28.618793214Z W1220 16:56:28.618742 4 warnings.go:70] kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2. [must-gather-xp8nl] POD 2025-12-20T16:56:29.214850975Z W1220 16:56:29.214799 4 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather-xp8nl] POD 2025-12-20T16:56:30.348577429Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] POD 2025-12-20T16:56:32.634176659Z Caches written to disk [must-gather-xp8nl] OUT 2025-12-20T16:56:33.696086011Z waiting for gather to complete [must-gather-xp8nl] OUT 2025-12-20T16:56:33.895885006Z downloading gather output [must-gather-xp8nl] OUT 2025-12-20T16:56:34.664461645Z receiving incremental file list [must-gather-xp8nl] OUT 2025-12-20T16:56:34.700252033Z ./ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.700349555Z version [must-gather-xp8nl] OUT 2025-12-20T16:56:34.726558914Z clusters/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.726579045Z clusters/ce271184/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.726634506Z clusters/ce271184/event-filter.html [must-gather-xp8nl] OUT 2025-12-20T16:56:34.729171944Z clusters/ce271184/oadp-must-gather-summary.md [must-gather-xp8nl] OUT 2025-12-20T16:56:34.729314947Z clusters/ce271184/timestamp [must-gather-xp8nl] OUT 2025-12-20T16:56:34.729383678Z clusters/ce271184/cluster-scoped-resources/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.729392859Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.729397329Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.72945379Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/backuprepositories.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.729618884Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/backups.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.7298724Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/backupstoragelocations.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.730028973Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudstorages.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.730183137Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.73733106Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datadownloads.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.737539135Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dataprotectionapplications.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.73819536Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datauploads.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.738379314Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/deletebackuprequests.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.738513437Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/downloadrequests.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.738675341Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podvolumebackups.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.738841465Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podvolumerestores.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.739010618Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/restores.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.739217943Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedules.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.739466169Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serverstatusrequests.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.739596122Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.740436251Z clusters/ce271184/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotlocations.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.740522673Z clusters/ce271184/cluster-scoped-resources/config.openshift.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.740577654Z clusters/ce271184/cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.740674587Z clusters/ce271184/cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.740684037Z clusters/ce271184/cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.740745178Z clusters/ce271184/cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/volumesnapshotclasses.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.74082696Z clusters/ce271184/cluster-scoped-resources/storage.k8s.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.74083616Z clusters/ce271184/cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.740889181Z clusters/ce271184/cluster-scoped-resources/storage.k8s.io/csidrivers/csidrivers.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.740974943Z clusters/ce271184/cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741045735Z clusters/ce271184/cluster-scoped-resources/storage.k8s.io/storageclasses/storageclasses.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741139077Z clusters/ce271184/namespaces/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741147657Z clusters/ce271184/namespaces/openshift-adp/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741219619Z clusters/ce271184/namespaces/openshift-adp/openshift-adp.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741318931Z clusters/ce271184/namespaces/openshift-adp/apps.openshift.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741390323Z clusters/ce271184/namespaces/openshift-adp/apps.openshift.io/deploymentconfigs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741452734Z clusters/ce271184/namespaces/openshift-adp/apps/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741532646Z clusters/ce271184/namespaces/openshift-adp/apps/daemonsets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741646699Z clusters/ce271184/namespaces/openshift-adp/apps/deployments.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.741868714Z clusters/ce271184/namespaces/openshift-adp/apps/replicasets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742056968Z clusters/ce271184/namespaces/openshift-adp/apps/statefulsets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.74211872Z clusters/ce271184/namespaces/openshift-adp/autoscaling/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742187541Z clusters/ce271184/namespaces/openshift-adp/autoscaling/horizontalpodautoscalers.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742255583Z clusters/ce271184/namespaces/openshift-adp/batch/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742346145Z clusters/ce271184/namespaces/openshift-adp/batch/cronjobs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742463657Z clusters/ce271184/namespaces/openshift-adp/batch/jobs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742531499Z clusters/ce271184/namespaces/openshift-adp/build.openshift.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.74259675Z clusters/ce271184/namespaces/openshift-adp/build.openshift.io/buildconfigs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742714143Z clusters/ce271184/namespaces/openshift-adp/build.openshift.io/builds.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742775444Z clusters/ce271184/namespaces/openshift-adp/cdi.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742851036Z clusters/ce271184/namespaces/openshift-adp/cdi.kubevirt.io/dataimportcrons.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.742982479Z clusters/ce271184/namespaces/openshift-adp/cdi.kubevirt.io/datasources.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.743097582Z clusters/ce271184/namespaces/openshift-adp/cdi.kubevirt.io/datavolumes.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.743172593Z clusters/ce271184/namespaces/openshift-adp/clone.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.743240105Z clusters/ce271184/namespaces/openshift-adp/clone.kubevirt.io/virtualmachineclones.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.743314497Z clusters/ce271184/namespaces/openshift-adp/core/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.743393359Z clusters/ce271184/namespaces/openshift-adp/core/configmaps.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.743548212Z clusters/ce271184/namespaces/openshift-adp/core/endpoints.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.743663965Z clusters/ce271184/namespaces/openshift-adp/core/events.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.747183705Z clusters/ce271184/namespaces/openshift-adp/core/persistentvolumeclaims.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.747318708Z clusters/ce271184/namespaces/openshift-adp/core/pods.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.747528413Z clusters/ce271184/namespaces/openshift-adp/core/replicationcontrollers.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.747671386Z clusters/ce271184/namespaces/openshift-adp/core/secrets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.747870361Z clusters/ce271184/namespaces/openshift-adp/core/services.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.747944413Z clusters/ce271184/namespaces/openshift-adp/discovery.k8s.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748016294Z clusters/ce271184/namespaces/openshift-adp/discovery.k8s.io/endpointslices.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748094486Z clusters/ce271184/namespaces/openshift-adp/export.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748154418Z clusters/ce271184/namespaces/openshift-adp/export.kubevirt.io/virtualmachineexports.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748225729Z clusters/ce271184/namespaces/openshift-adp/hco.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.74828655Z clusters/ce271184/namespaces/openshift-adp/hco.kubevirt.io/hyperconvergeds.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748380653Z clusters/ce271184/namespaces/openshift-adp/image.openshift.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748448974Z clusters/ce271184/namespaces/openshift-adp/image.openshift.io/imagestreams.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748532676Z clusters/ce271184/namespaces/openshift-adp/instancetype.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748575907Z clusters/ce271184/namespaces/openshift-adp/instancetype.kubevirt.io/virtualmachineinstancetypes.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.74870278Z clusters/ce271184/namespaces/openshift-adp/instancetype.kubevirt.io/virtualmachinepreferences.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748763071Z clusters/ce271184/namespaces/openshift-adp/k8s.ovn.org/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748829753Z clusters/ce271184/namespaces/openshift-adp/k8s.ovn.org/egressfirewalls.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.748949205Z clusters/ce271184/namespaces/openshift-adp/k8s.ovn.org/egressqoses.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749012097Z clusters/ce271184/namespaces/openshift-adp/kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749080519Z clusters/ce271184/namespaces/openshift-adp/kubevirt.io/kubevirts.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749197531Z clusters/ce271184/namespaces/openshift-adp/kubevirt.io/virtualmachineinstancemigrations.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749339834Z clusters/ce271184/namespaces/openshift-adp/kubevirt.io/virtualmachineinstancepresets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749458037Z clusters/ce271184/namespaces/openshift-adp/kubevirt.io/virtualmachineinstancereplicasets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749626341Z clusters/ce271184/namespaces/openshift-adp/kubevirt.io/virtualmachineinstances.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749750704Z clusters/ce271184/namespaces/openshift-adp/kubevirt.io/virtualmachines.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749827786Z clusters/ce271184/namespaces/openshift-adp/monitoring.coreos.com/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749893577Z clusters/ce271184/namespaces/openshift-adp/monitoring.coreos.com/servicemonitors.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.749964789Z clusters/ce271184/namespaces/openshift-adp/networking.k8s.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.75002211Z clusters/ce271184/namespaces/openshift-adp/networking.k8s.io/networkpolicies.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750102942Z clusters/ce271184/namespaces/openshift-adp/operators.coreos.com/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750110332Z clusters/ce271184/namespaces/openshift-adp/operators.coreos.com/clusterserviceversions/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750172193Z clusters/ce271184/namespaces/openshift-adp/operators.coreos.com/clusterserviceversions/clusterserviceversions.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.7504613Z clusters/ce271184/namespaces/openshift-adp/operators.coreos.com/subscriptions/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750529082Z clusters/ce271184/namespaces/openshift-adp/operators.coreos.com/subscriptions/subscriptions.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750614484Z clusters/ce271184/namespaces/openshift-adp/pods/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750622264Z clusters/ce271184/namespaces/openshift-adp/pods/openshift-adp-controller-manager-fd84cfc68-b4rl6/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750688475Z clusters/ce271184/namespaces/openshift-adp/pods/openshift-adp-controller-manager-fd84cfc68-b4rl6/openshift-adp-controller-manager-fd84cfc68-b4rl6.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750825608Z clusters/ce271184/namespaces/openshift-adp/pods/openshift-adp-controller-manager-fd84cfc68-b4rl6/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750833259Z clusters/ce271184/namespaces/openshift-adp/pods/openshift-adp-controller-manager-fd84cfc68-b4rl6/manager/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.750837589Z clusters/ce271184/namespaces/openshift-adp/pods/openshift-adp-controller-manager-fd84cfc68-b4rl6/manager/manager/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.75091308Z clusters/ce271184/namespaces/openshift-adp/pods/openshift-adp-controller-manager-fd84cfc68-b4rl6/manager/manager/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.755786792Z clusters/ce271184/namespaces/openshift-adp/pods/openshift-adp-controller-manager-fd84cfc68-b4rl6/manager/manager/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.755907404Z clusters/ce271184/namespaces/openshift-adp/pods/openshift-adp-controller-manager-fd84cfc68-b4rl6/manager/manager/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.755956056Z clusters/ce271184/namespaces/openshift-adp/policy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756013717Z clusters/ce271184/namespaces/openshift-adp/policy/poddisruptionbudgets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756088589Z clusters/ce271184/namespaces/openshift-adp/pool.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.7561523Z clusters/ce271184/namespaces/openshift-adp/pool.kubevirt.io/virtualmachinepools.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756222172Z clusters/ce271184/namespaces/openshift-adp/route.openshift.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756283983Z clusters/ce271184/namespaces/openshift-adp/route.openshift.io/routes.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756367465Z clusters/ce271184/namespaces/openshift-adp/snapshot.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756430687Z clusters/ce271184/namespaces/openshift-adp/snapshot.kubevirt.io/virtualmachinerestores.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756555749Z clusters/ce271184/namespaces/openshift-adp/snapshot.kubevirt.io/virtualmachinesnapshotcontents.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756676812Z clusters/ce271184/namespaces/openshift-adp/snapshot.kubevirt.io/virtualmachinesnapshots.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756739834Z clusters/ce271184/namespaces/openshift-cnv/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756801495Z clusters/ce271184/namespaces/openshift-cnv/openshift-cnv.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756885587Z clusters/ce271184/namespaces/openshift-cnv/apps.openshift.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.756951508Z clusters/ce271184/namespaces/openshift-cnv/apps.openshift.io/deploymentconfigs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.75701826Z clusters/ce271184/namespaces/openshift-cnv/apps/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.757076721Z clusters/ce271184/namespaces/openshift-cnv/apps/daemonsets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.757285526Z clusters/ce271184/namespaces/openshift-cnv/apps/deployments.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.758214577Z clusters/ce271184/namespaces/openshift-cnv/apps/replicasets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759063867Z clusters/ce271184/namespaces/openshift-cnv/apps/statefulsets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759121928Z clusters/ce271184/namespaces/openshift-cnv/autoscaling/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.75919835Z clusters/ce271184/namespaces/openshift-cnv/autoscaling/horizontalpodautoscalers.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759259481Z clusters/ce271184/namespaces/openshift-cnv/batch/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759355203Z clusters/ce271184/namespaces/openshift-cnv/batch/cronjobs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759475876Z clusters/ce271184/namespaces/openshift-cnv/batch/jobs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759540747Z clusters/ce271184/namespaces/openshift-cnv/build.openshift.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759608589Z clusters/ce271184/namespaces/openshift-cnv/build.openshift.io/buildconfigs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759723102Z clusters/ce271184/namespaces/openshift-cnv/build.openshift.io/builds.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759784893Z clusters/ce271184/namespaces/openshift-cnv/cdi.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759860055Z clusters/ce271184/namespaces/openshift-cnv/cdi.kubevirt.io/dataimportcrons.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.759976208Z clusters/ce271184/namespaces/openshift-cnv/cdi.kubevirt.io/datasources.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.76009635Z clusters/ce271184/namespaces/openshift-cnv/cdi.kubevirt.io/datavolumes.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.760163772Z clusters/ce271184/namespaces/openshift-cnv/clone.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.760231863Z clusters/ce271184/namespaces/openshift-cnv/clone.kubevirt.io/virtualmachineclones.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.760312385Z clusters/ce271184/namespaces/openshift-cnv/core/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.760403597Z clusters/ce271184/namespaces/openshift-cnv/core/configmaps.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.765607566Z clusters/ce271184/namespaces/openshift-cnv/core/endpoints.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.765820011Z clusters/ce271184/namespaces/openshift-cnv/core/events.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.767275374Z clusters/ce271184/namespaces/openshift-cnv/core/persistentvolumeclaims.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.767433878Z clusters/ce271184/namespaces/openshift-cnv/core/pods.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.768953473Z clusters/ce271184/namespaces/openshift-cnv/core/replicationcontrollers.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.769103246Z clusters/ce271184/namespaces/openshift-cnv/core/secrets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.769628398Z clusters/ce271184/namespaces/openshift-cnv/core/services.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.769809482Z clusters/ce271184/namespaces/openshift-cnv/discovery.k8s.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.769875154Z clusters/ce271184/namespaces/openshift-cnv/discovery.k8s.io/endpointslices.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770074188Z clusters/ce271184/namespaces/openshift-cnv/export.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.77014044Z clusters/ce271184/namespaces/openshift-cnv/export.kubevirt.io/virtualmachineexports.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770212281Z clusters/ce271184/namespaces/openshift-cnv/hco.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770286413Z clusters/ce271184/namespaces/openshift-cnv/hco.kubevirt.io/hyperconvergeds.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770422256Z clusters/ce271184/namespaces/openshift-cnv/image.openshift.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770492868Z clusters/ce271184/namespaces/openshift-cnv/image.openshift.io/imagestreams.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770548799Z clusters/ce271184/namespaces/openshift-cnv/instancetype.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770634111Z clusters/ce271184/namespaces/openshift-cnv/instancetype.kubevirt.io/virtualmachineinstancetypes.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770744614Z clusters/ce271184/namespaces/openshift-cnv/instancetype.kubevirt.io/virtualmachinepreferences.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770804785Z clusters/ce271184/namespaces/openshift-cnv/k8s.ovn.org/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.770871066Z clusters/ce271184/namespaces/openshift-cnv/k8s.ovn.org/egressfirewalls.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.771000349Z clusters/ce271184/namespaces/openshift-cnv/k8s.ovn.org/egressqoses.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.771064141Z clusters/ce271184/namespaces/openshift-cnv/kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.771127932Z clusters/ce271184/namespaces/openshift-cnv/kubevirt.io/kubevirts.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.771324267Z clusters/ce271184/namespaces/openshift-cnv/kubevirt.io/virtualmachineinstancemigrations.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.77145419Z clusters/ce271184/namespaces/openshift-cnv/kubevirt.io/virtualmachineinstancepresets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.771578113Z clusters/ce271184/namespaces/openshift-cnv/kubevirt.io/virtualmachineinstancereplicasets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.771753337Z clusters/ce271184/namespaces/openshift-cnv/kubevirt.io/virtualmachineinstances.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.771872459Z clusters/ce271184/namespaces/openshift-cnv/kubevirt.io/virtualmachines.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.771943511Z clusters/ce271184/namespaces/openshift-cnv/monitoring.coreos.com/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.772008142Z clusters/ce271184/namespaces/openshift-cnv/monitoring.coreos.com/servicemonitors.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.772112285Z clusters/ce271184/namespaces/openshift-cnv/networking.k8s.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.772174986Z clusters/ce271184/namespaces/openshift-cnv/networking.k8s.io/networkpolicies.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.772277049Z clusters/ce271184/namespaces/openshift-cnv/operators.coreos.com/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.772284749Z clusters/ce271184/namespaces/openshift-cnv/operators.coreos.com/clusterserviceversions/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.772374721Z clusters/ce271184/namespaces/openshift-cnv/operators.coreos.com/clusterserviceversions/clusterserviceversions.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773063876Z clusters/ce271184/namespaces/openshift-cnv/operators.coreos.com/subscriptions/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773143248Z clusters/ce271184/namespaces/openshift-cnv/operators.coreos.com/subscriptions/subscriptions.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773228Z clusters/ce271184/namespaces/openshift-cnv/pods/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.7732365Z clusters/ce271184/namespaces/openshift-cnv/pods/aaq-operator-6bd9c9fdd4-6r6cs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773317892Z clusters/ce271184/namespaces/openshift-cnv/pods/aaq-operator-6bd9c9fdd4-6r6cs/aaq-operator-6bd9c9fdd4-6r6cs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773487566Z clusters/ce271184/namespaces/openshift-cnv/pods/aaq-operator-6bd9c9fdd4-6r6cs/aaq-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773495816Z clusters/ce271184/namespaces/openshift-cnv/pods/aaq-operator-6bd9c9fdd4-6r6cs/aaq-operator/aaq-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773499856Z clusters/ce271184/namespaces/openshift-cnv/pods/aaq-operator-6bd9c9fdd4-6r6cs/aaq-operator/aaq-operator/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773573908Z clusters/ce271184/namespaces/openshift-cnv/pods/aaq-operator-6bd9c9fdd4-6r6cs/aaq-operator/aaq-operator/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773734842Z clusters/ce271184/namespaces/openshift-cnv/pods/aaq-operator-6bd9c9fdd4-6r6cs/aaq-operator/aaq-operator/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773846354Z clusters/ce271184/namespaces/openshift-cnv/pods/aaq-operator-6bd9c9fdd4-6r6cs/aaq-operator/aaq-operator/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773898286Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-6qbq7/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.773971327Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-6qbq7/bridge-marker-6qbq7.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.77407379Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-6qbq7/bridge-marker/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.77408171Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-6qbq7/bridge-marker/bridge-marker/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.77408573Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-6qbq7/bridge-marker/bridge-marker/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.774154551Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-6qbq7/bridge-marker/bridge-marker/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.774273484Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-6qbq7/bridge-marker/bridge-marker/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.774408607Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-6qbq7/bridge-marker/bridge-marker/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.774471199Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-mt2q6/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.77453775Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-mt2q6/bridge-marker-mt2q6.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.774624272Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-mt2q6/bridge-marker/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.774632032Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-mt2q6/bridge-marker/bridge-marker/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.774636463Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-mt2q6/bridge-marker/bridge-marker/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.774717894Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-mt2q6/bridge-marker/bridge-marker/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.774830797Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-mt2q6/bridge-marker/bridge-marker/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.77496576Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-mt2q6/bridge-marker/bridge-marker/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775021951Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-vmjb6/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775089093Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-vmjb6/bridge-marker-vmjb6.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775176115Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-vmjb6/bridge-marker/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775186905Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-vmjb6/bridge-marker/bridge-marker/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775191135Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-vmjb6/bridge-marker/bridge-marker/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775265857Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-vmjb6/bridge-marker/bridge-marker/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.77540142Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-vmjb6/bridge-marker/bridge-marker/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775511752Z clusters/ce271184/namespaces/openshift-cnv/pods/bridge-marker-vmjb6/bridge-marker/bridge-marker/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775565354Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-apiserver-cf94ff697-v4ktk/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775633495Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-apiserver-cf94ff697-v4ktk/cdi-apiserver-cf94ff697-v4ktk.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775732368Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-apiserver-cf94ff697-v4ktk/cdi-apiserver/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775740958Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-apiserver-cf94ff697-v4ktk/cdi-apiserver/cdi-apiserver/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775744988Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-apiserver-cf94ff697-v4ktk/cdi-apiserver/cdi-apiserver/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775814939Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-apiserver-cf94ff697-v4ktk/cdi-apiserver/cdi-apiserver/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.775952062Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-apiserver-cf94ff697-v4ktk/cdi-apiserver/cdi-apiserver/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.776066625Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-apiserver-cf94ff697-v4ktk/cdi-apiserver/cdi-apiserver/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.776126537Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-deployment-59f688d645-847k4/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.776188898Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-deployment-59f688d645-847k4/cdi-deployment-59f688d645-847k4.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.776324181Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-deployment-59f688d645-847k4/cdi-deployment/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.776341111Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-deployment-59f688d645-847k4/cdi-deployment/cdi-deployment/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.776345491Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-deployment-59f688d645-847k4/cdi-deployment/cdi-deployment/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.776418633Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-deployment-59f688d645-847k4/cdi-deployment/cdi-deployment/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.790326281Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-deployment-59f688d645-847k4/cdi-deployment/cdi-deployment/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.790478284Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-deployment-59f688d645-847k4/cdi-deployment/cdi-deployment/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.790535836Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-operator-66b9bc75cf-7pp7d/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.790601337Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-operator-66b9bc75cf-7pp7d/cdi-operator-66b9bc75cf-7pp7d.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.79074352Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-operator-66b9bc75cf-7pp7d/cdi-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.790754611Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-operator-66b9bc75cf-7pp7d/cdi-operator/cdi-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.790761381Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-operator-66b9bc75cf-7pp7d/cdi-operator/cdi-operator/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.790824052Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-operator-66b9bc75cf-7pp7d/cdi-operator/cdi-operator/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.79158231Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-operator-66b9bc75cf-7pp7d/cdi-operator/cdi-operator/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.791680082Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-operator-66b9bc75cf-7pp7d/cdi-operator/cdi-operator/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.791724763Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-uploadproxy-9b99c47bb-q9mbj/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.791787064Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-uploadproxy-9b99c47bb-q9mbj/cdi-uploadproxy-9b99c47bb-q9mbj.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.791868936Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-uploadproxy-9b99c47bb-q9mbj/cdi-uploadproxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.791876316Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-uploadproxy-9b99c47bb-q9mbj/cdi-uploadproxy/cdi-uploadproxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.791880146Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-uploadproxy-9b99c47bb-q9mbj/cdi-uploadproxy/cdi-uploadproxy/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.791943778Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-uploadproxy-9b99c47bb-q9mbj/cdi-uploadproxy/cdi-uploadproxy/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.792074011Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-uploadproxy-9b99c47bb-q9mbj/cdi-uploadproxy/cdi-uploadproxy/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.792176103Z clusters/ce271184/namespaces/openshift-cnv/pods/cdi-uploadproxy-9b99c47bb-q9mbj/cdi-uploadproxy/cdi-uploadproxy/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.792220044Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.792275835Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/cluster-network-addons-operator-5c8c5db4fd-bnr8l.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.79245053Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/cluster-network-addons-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.79246257Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/cluster-network-addons-operator/cluster-network-addons-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.79246793Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/cluster-network-addons-operator/cluster-network-addons-operator/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.792531421Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/cluster-network-addons-operator/cluster-network-addons-operator/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.793675947Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/cluster-network-addons-operator/cluster-network-addons-operator/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.79378891Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/cluster-network-addons-operator/cluster-network-addons-operator/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.793811081Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/kube-rbac-proxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.793816961Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/kube-rbac-proxy/kube-rbac-proxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.793821861Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.793898023Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.794024425Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.794126528Z clusters/ce271184/namespaces/openshift-cnv/pods/cluster-network-addons-operator-5c8c5db4fd-bnr8l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.794141478Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-operator-5f4c9558b5-8lwqm/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.79422071Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-operator-5f4c9558b5-8lwqm/hco-operator-5f4c9558b5-8lwqm.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.794344103Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-operator-5f4c9558b5-8lwqm/hyperconverged-cluster-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.794357523Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-operator-5f4c9558b5-8lwqm/hyperconverged-cluster-operator/hyperconverged-cluster-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.794361793Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-operator-5f4c9558b5-8lwqm/hyperconverged-cluster-operator/hyperconverged-cluster-operator/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.794433705Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-operator-5f4c9558b5-8lwqm/hyperconverged-cluster-operator/hyperconverged-cluster-operator/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.809201302Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-operator-5f4c9558b5-8lwqm/hyperconverged-cluster-operator/hyperconverged-cluster-operator/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.809327415Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-operator-5f4c9558b5-8lwqm/hyperconverged-cluster-operator/hyperconverged-cluster-operator/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.809348296Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-webhook-b8d5f8ccb-srqs2/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.809429887Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-webhook-b8d5f8ccb-srqs2/hco-webhook-b8d5f8ccb-srqs2.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.80955154Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-webhook-b8d5f8ccb-srqs2/hyperconverged-cluster-webhook/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.80956596Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-webhook-b8d5f8ccb-srqs2/hyperconverged-cluster-webhook/hyperconverged-cluster-webhook/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.809571461Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-webhook-b8d5f8ccb-srqs2/hyperconverged-cluster-webhook/hyperconverged-cluster-webhook/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.809619502Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-webhook-b8d5f8ccb-srqs2/hyperconverged-cluster-webhook/hyperconverged-cluster-webhook/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.809876597Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-webhook-b8d5f8ccb-srqs2/hyperconverged-cluster-webhook/hyperconverged-cluster-webhook/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.80998607Z clusters/ce271184/namespaces/openshift-cnv/pods/hco-webhook-b8d5f8ccb-srqs2/hyperconverged-cluster-webhook/hyperconverged-cluster-webhook/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81000698Z clusters/ce271184/namespaces/openshift-cnv/pods/hostpath-provisioner-operator-55c45dddb8-xt6tg/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810075952Z clusters/ce271184/namespaces/openshift-cnv/pods/hostpath-provisioner-operator-55c45dddb8-xt6tg/hostpath-provisioner-operator-55c45dddb8-xt6tg.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810202965Z clusters/ce271184/namespaces/openshift-cnv/pods/hostpath-provisioner-operator-55c45dddb8-xt6tg/hostpath-provisioner-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810213185Z clusters/ce271184/namespaces/openshift-cnv/pods/hostpath-provisioner-operator-55c45dddb8-xt6tg/hostpath-provisioner-operator/hostpath-provisioner-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810217825Z clusters/ce271184/namespaces/openshift-cnv/pods/hostpath-provisioner-operator-55c45dddb8-xt6tg/hostpath-provisioner-operator/hostpath-provisioner-operator/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810276347Z clusters/ce271184/namespaces/openshift-cnv/pods/hostpath-provisioner-operator-55c45dddb8-xt6tg/hostpath-provisioner-operator/hostpath-provisioner-operator/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81043889Z clusters/ce271184/namespaces/openshift-cnv/pods/hostpath-provisioner-operator-55c45dddb8-xt6tg/hostpath-provisioner-operator/hostpath-provisioner-operator/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810542123Z clusters/ce271184/namespaces/openshift-cnv/pods/hostpath-provisioner-operator-55c45dddb8-xt6tg/hostpath-provisioner-operator/hostpath-provisioner-operator/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810581004Z clusters/ce271184/namespaces/openshift-cnv/pods/hyperconverged-cluster-cli-download-774579cf7b-jn4r8/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810653545Z clusters/ce271184/namespaces/openshift-cnv/pods/hyperconverged-cluster-cli-download-774579cf7b-jn4r8/hyperconverged-cluster-cli-download-774579cf7b-jn4r8.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810745557Z clusters/ce271184/namespaces/openshift-cnv/pods/hyperconverged-cluster-cli-download-774579cf7b-jn4r8/server/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810757238Z clusters/ce271184/namespaces/openshift-cnv/pods/hyperconverged-cluster-cli-download-774579cf7b-jn4r8/server/server/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810761258Z clusters/ce271184/namespaces/openshift-cnv/pods/hyperconverged-cluster-cli-download-774579cf7b-jn4r8/server/server/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810820639Z clusters/ce271184/namespaces/openshift-cnv/pods/hyperconverged-cluster-cli-download-774579cf7b-jn4r8/server/server/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.810965952Z clusters/ce271184/namespaces/openshift-cnv/pods/hyperconverged-cluster-cli-download-774579cf7b-jn4r8/server/server/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811056515Z clusters/ce271184/namespaces/openshift-cnv/pods/hyperconverged-cluster-cli-download-774579cf7b-jn4r8/server/server/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811102145Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-kwgc8/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811167277Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-kwgc8/kube-cni-linux-bridge-plugin-kwgc8.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811254739Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-kwgc8/cni-plugins/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811266639Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-kwgc8/cni-plugins/cni-plugins/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811276329Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-kwgc8/cni-plugins/cni-plugins/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811349711Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-kwgc8/cni-plugins/cni-plugins/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811470144Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-kwgc8/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811570656Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-kwgc8/cni-plugins/cni-plugins/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811608017Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-mgxsb/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811673698Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-mgxsb/kube-cni-linux-bridge-plugin-mgxsb.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81175762Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-mgxsb/cni-plugins/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811766941Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-mgxsb/cni-plugins/cni-plugins/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811770981Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-mgxsb/cni-plugins/cni-plugins/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811825352Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-mgxsb/cni-plugins/cni-plugins/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.811946525Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-mgxsb/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812047687Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-mgxsb/cni-plugins/cni-plugins/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812090818Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-rp4dc/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81215368Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-rp4dc/kube-cni-linux-bridge-plugin-rp4dc.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812233011Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-rp4dc/cni-plugins/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812240971Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-rp4dc/cni-plugins/cni-plugins/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812244972Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-rp4dc/cni-plugins/cni-plugins/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812334084Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-rp4dc/cni-plugins/cni-plugins/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812453567Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-rp4dc/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812552389Z clusters/ce271184/namespaces/openshift-cnv/pods/kube-cni-linux-bridge-plugin-rp4dc/cni-plugins/cni-plugins/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81259499Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-cert-manager-6d647f8d85-fwqmm/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812660921Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-cert-manager-6d647f8d85-fwqmm/kubemacpool-cert-manager-6d647f8d85-fwqmm.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812738773Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-cert-manager-6d647f8d85-fwqmm/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812746133Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-cert-manager-6d647f8d85-fwqmm/manager/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812759263Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-cert-manager-6d647f8d85-fwqmm/manager/manager/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.812813675Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-cert-manager-6d647f8d85-fwqmm/manager/manager/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813013999Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-cert-manager-6d647f8d85-fwqmm/manager/manager/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813116162Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-cert-manager-6d647f8d85-fwqmm/manager/manager/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813155843Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813225804Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813369437Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/kube-rbac-proxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813381438Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/kube-rbac-proxy/kube-rbac-proxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813385588Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813441579Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813571562Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813676784Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813708895Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813716435Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/manager/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813721345Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/manager/manager/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813789617Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/manager/manager/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81392591Z clusters/ce271184/namespaces/openshift-cnv/pods/kubemacpool-mac-controller-manager-55b59b68bd-dtcjg/manager/manager/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.813994252Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-8twb6/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814057823Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-8twb6/kubevirt-apiserver-proxy-5c79f8bb-8twb6.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814142835Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-8twb6/kubevirt-apiserver-proxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814152855Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-8twb6/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814157145Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-8twb6/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814217097Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-8twb6/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81436085Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-8twb6/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814467922Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-8twb6/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814505813Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-t6jbm/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814568475Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-t6jbm/kubevirt-apiserver-proxy-5c79f8bb-t6jbm.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814644237Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-t6jbm/kubevirt-apiserver-proxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814651877Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-t6jbm/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814655727Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-t6jbm/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814724308Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-t6jbm/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814841201Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-t6jbm/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814953873Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-apiserver-proxy-5c79f8bb-t6jbm/kubevirt-apiserver-proxy/kubevirt-apiserver-proxy/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.814986804Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-kxkvq/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815053176Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-kxkvq/kubevirt-console-plugin-857d9cf689-kxkvq.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815132558Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-kxkvq/kubevirt-console-plugin/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815140158Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-kxkvq/kubevirt-console-plugin/kubevirt-console-plugin/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815144238Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-kxkvq/kubevirt-console-plugin/kubevirt-console-plugin/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815205259Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-kxkvq/kubevirt-console-plugin/kubevirt-console-plugin/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815345382Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-kxkvq/kubevirt-console-plugin/kubevirt-console-plugin/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815457795Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-kxkvq/kubevirt-console-plugin/kubevirt-console-plugin/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815489316Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-rcq54/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815556467Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-rcq54/kubevirt-console-plugin-857d9cf689-rcq54.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815634219Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-rcq54/kubevirt-console-plugin/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815646649Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-rcq54/kubevirt-console-plugin/kubevirt-console-plugin/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815650749Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-rcq54/kubevirt-console-plugin/kubevirt-console-plugin/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815709201Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-rcq54/kubevirt-console-plugin/kubevirt-console-plugin/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815825784Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-rcq54/kubevirt-console-plugin/kubevirt-console-plugin/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815929416Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-console-plugin-857d9cf689-rcq54/kubevirt-console-plugin/kubevirt-console-plugin/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.815967597Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-ipam-controller-manager-6ccf4c4b7c-m4vxp/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816030418Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-ipam-controller-manager-6ccf4c4b7c-m4vxp/kubevirt-ipam-controller-manager-6ccf4c4b7c-m4vxp.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81611055Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-ipam-controller-manager-6ccf4c4b7c-m4vxp/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81612182Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-ipam-controller-manager-6ccf4c4b7c-m4vxp/manager/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816126Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-ipam-controller-manager-6ccf4c4b7c-m4vxp/manager/manager/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816182821Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-ipam-controller-manager-6ccf4c4b7c-m4vxp/manager/manager/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816354116Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-ipam-controller-manager-6ccf4c4b7c-m4vxp/manager/manager/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816463318Z clusters/ce271184/namespaces/openshift-cnv/pods/kubevirt-ipam-controller-manager-6ccf4c4b7c-m4vxp/manager/manager/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816503949Z clusters/ce271184/namespaces/openshift-cnv/pods/ssp-operator-74dc99648f-65kwh/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.81656295Z clusters/ce271184/namespaces/openshift-cnv/pods/ssp-operator-74dc99648f-65kwh/ssp-operator-74dc99648f-65kwh.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816668773Z clusters/ce271184/namespaces/openshift-cnv/pods/ssp-operator-74dc99648f-65kwh/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816676593Z clusters/ce271184/namespaces/openshift-cnv/pods/ssp-operator-74dc99648f-65kwh/manager/manager/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816680683Z clusters/ce271184/namespaces/openshift-cnv/pods/ssp-operator-74dc99648f-65kwh/manager/manager/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.816737804Z clusters/ce271184/namespaces/openshift-cnv/pods/ssp-operator-74dc99648f-65kwh/manager/manager/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.818903874Z clusters/ce271184/namespaces/openshift-cnv/pods/ssp-operator-74dc99648f-65kwh/manager/manager/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.819049017Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-2zglp/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.819106839Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-2zglp/virt-api-6fdb9b587f-2zglp.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.819202231Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-2zglp/virt-api/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.819213201Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-2zglp/virt-api/virt-api/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.819223071Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-2zglp/virt-api/virt-api/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.819279342Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-2zglp/virt-api/virt-api/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.821149375Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-2zglp/virt-api/virt-api/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.821247867Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-2zglp/virt-api/virt-api/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.821288688Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-47dng/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.82137521Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-47dng/virt-api-6fdb9b587f-47dng.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.821457202Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-47dng/virt-api/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.821465782Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-47dng/virt-api/virt-api/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.821469842Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-47dng/virt-api/virt-api/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.821528334Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-47dng/virt-api/virt-api/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.823408127Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-47dng/virt-api/virt-api/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.823499669Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-api-6fdb9b587f-47dng/virt-api/virt-api/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.823532869Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-g8ff9/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.823607331Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-g8ff9/virt-controller-74b44f6dc4-g8ff9.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.823692263Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-g8ff9/virt-controller/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.823700183Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-g8ff9/virt-controller/virt-controller/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.823704253Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-g8ff9/virt-controller/virt-controller/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.823769565Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-g8ff9/virt-controller/virt-controller/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.823939009Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-g8ff9/virt-controller/virt-controller/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824041481Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-g8ff9/virt-controller/virt-controller/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824088222Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-gzx5h/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824150164Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-gzx5h/virt-controller-74b44f6dc4-gzx5h.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824238116Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-gzx5h/virt-controller/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824246376Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-gzx5h/virt-controller/virt-controller/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824250706Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-gzx5h/virt-controller/virt-controller/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824347918Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-gzx5h/virt-controller/virt-controller/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824487161Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-gzx5h/virt-controller/virt-controller/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824582664Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-controller-74b44f6dc4-gzx5h/virt-controller/virt-controller/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824622734Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-6xf8r/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824689786Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-6xf8r/virt-exportproxy-5bc78744b8-6xf8r.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824767958Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-6xf8r/exportproxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824776798Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-6xf8r/exportproxy/exportproxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824780688Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-6xf8r/exportproxy/exportproxy/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824837109Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-6xf8r/exportproxy/exportproxy/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.824962052Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-6xf8r/exportproxy/exportproxy/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825065604Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-6xf8r/exportproxy/exportproxy/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825104056Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-fttrg/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825172247Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-fttrg/virt-exportproxy-5bc78744b8-fttrg.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825250599Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-fttrg/exportproxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825258429Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-fttrg/exportproxy/exportproxy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825262899Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-fttrg/exportproxy/exportproxy/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825351201Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-fttrg/exportproxy/exportproxy/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825470084Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-fttrg/exportproxy/exportproxy/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825578276Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-exportproxy-5bc78744b8-fttrg/exportproxy/exportproxy/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825596587Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825660528Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-handler-5ghpx.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825770181Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-handler/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825778951Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-handler/virt-handler/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825782891Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-handler/virt-handler/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.825849333Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-handler/virt-handler/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826050167Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-handler/virt-handler/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.8261525Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-handler/virt-handler/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.82619334Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-launcher/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.82620073Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-launcher/virt-launcher/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826204711Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-launcher/virt-launcher/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826265152Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-launcher/virt-launcher/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826408915Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-launcher/virt-launcher/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826518578Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-5ghpx/virt-launcher/virt-launcher/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826552098Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.82661846Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-handler-fxj9n.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826713922Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-handler/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826722142Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-handler/virt-handler/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826726213Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-handler/virt-handler/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826785064Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-handler/virt-handler/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.826996779Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-handler/virt-handler/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827106881Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-handler/virt-handler/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827125391Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-launcher/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827146412Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-launcher/virt-launcher/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827155292Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-launcher/virt-launcher/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827213714Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-launcher/virt-launcher/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827356877Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-launcher/virt-launcher/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827462449Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-fxj9n/virt-launcher/virt-launcher/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.82748599Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827554691Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-handler-mvfxs.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827654014Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-handler/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827666244Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-handler/virt-handler/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827675424Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-handler/virt-handler/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.827729035Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-handler/virt-handler/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.82792329Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-handler/virt-handler/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828025762Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-handler/virt-handler/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828069543Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-launcher/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828080193Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-launcher/virt-launcher/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828086453Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-launcher/virt-launcher/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828145325Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-launcher/virt-launcher/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828261858Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-launcher/virt-launcher/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828489263Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-handler-mvfxs/virt-launcher/virt-launcher/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828533214Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-44zxd/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828596025Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-44zxd/virt-operator-6f4f54ccb7-44zxd.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828735898Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-44zxd/virt-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828745439Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-44zxd/virt-operator/virt-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.828749379Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-44zxd/virt-operator/virt-operator/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.82880671Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-44zxd/virt-operator/virt-operator/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.832268709Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-44zxd/virt-operator/virt-operator/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.832433163Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-44zxd/virt-operator/virt-operator/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.832481134Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-rvjxg/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.832542925Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-rvjxg/virt-operator-6f4f54ccb7-rvjxg.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.832650968Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-rvjxg/virt-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.832662258Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-rvjxg/virt-operator/virt-operator/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.832667158Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-rvjxg/virt-operator/virt-operator/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.83272047Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-rvjxg/virt-operator/virt-operator/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.832902764Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-rvjxg/virt-operator/virt-operator/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833008696Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-operator-6f4f54ccb7-rvjxg/virt-operator/virt-operator/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833028637Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-6kd9p/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833097588Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-6kd9p/virt-template-validator-74d57dc87b-6kd9p.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.83318808Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-6kd9p/webhook/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.83320003Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-6kd9p/webhook/webhook/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833205121Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-6kd9p/webhook/webhook/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833258462Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-6kd9p/webhook/webhook/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833404095Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-6kd9p/webhook/webhook/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833505057Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-6kd9p/webhook/webhook/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833546208Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-lgl6j/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.83361174Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-lgl6j/virt-template-validator-74d57dc87b-lgl6j.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833688541Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-lgl6j/webhook/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833696652Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-lgl6j/webhook/webhook/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833701262Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-lgl6j/webhook/webhook/logs/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833760733Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-lgl6j/webhook/webhook/logs/current.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833884356Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-lgl6j/webhook/webhook/logs/previous.insecure.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.833986598Z clusters/ce271184/namespaces/openshift-cnv/pods/virt-template-validator-74d57dc87b-lgl6j/webhook/webhook/logs/previous.log [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834027099Z clusters/ce271184/namespaces/openshift-cnv/policy/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834085681Z clusters/ce271184/namespaces/openshift-cnv/policy/poddisruptionbudgets.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834164023Z clusters/ce271184/namespaces/openshift-cnv/pool.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834226914Z clusters/ce271184/namespaces/openshift-cnv/pool.kubevirt.io/virtualmachinepools.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834276395Z clusters/ce271184/namespaces/openshift-cnv/route.openshift.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834360717Z clusters/ce271184/namespaces/openshift-cnv/route.openshift.io/routes.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834441789Z clusters/ce271184/namespaces/openshift-cnv/snapshot.kubevirt.io/ [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834533871Z clusters/ce271184/namespaces/openshift-cnv/snapshot.kubevirt.io/virtualmachinerestores.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834643583Z clusters/ce271184/namespaces/openshift-cnv/snapshot.kubevirt.io/virtualmachinesnapshotcontents.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.834746486Z clusters/ce271184/namespaces/openshift-cnv/snapshot.kubevirt.io/virtualmachinesnapshots.yaml [must-gather-xp8nl] OUT 2025-12-20T16:56:34.856229816Z [must-gather-xp8nl] OUT 2025-12-20T16:56:34.856261537Z sent 6,423 bytes received 1,744,118 bytes 3,501,082.00 bytes/sec [must-gather-xp8nl] OUT 2025-12-20T16:56:34.856271408Z total size is 25,636,255 speedup is 14.64 [must-gather ] OUT 2025-12-20T16:56:35.062149881Z namespace/openshift-must-gather-npjpp deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: ce271184-7500-4576-8f03-f3940db7522e ClientVersion: 4.20.8 ClusterVersion: Stable at "4.21.0-0.nightly-2025-12-18-020818" ClusterOperators: clusteroperator/operator-lifecycle-manager is not upgradeable because ClusterServiceVersions blocking minor version upgrades to 4.21.0 or higher: - maximum supported OCP version for openshift-storage/odf-dependencies.v4.19.9-rhodf is 4.20 - maximum supported OCP version for openshift-storage/odf-operator.v4.19.9-rhodf is 4.20 Checking for additional logs in /alabama/cspi/e2e/logs Copying /alabama/cspi/e2e/logs to /logs/artifacts... It_Backup_restore_tests_Application_backup_tc-id_OADP-122_interop_skip-disconnected_Tier0_vf_csi-snapshots_Django_application_with_BSL_CSI_exclude_aro-4 It_Backup_restore_tests_Application_backup_tc-id_OADP-437_interop_smoke_Tier0_vf_fsb-kopia_MySQL_application_with_filesystem_Kopia_mr-check It_Incremental_backup_restore_tests_Incremental_restore_pod_count_tc-id_OADP-165_interop_Tier0_vf_restore-policies_Todolist_app_with_CSI_-_policy_update It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_Tier0_vf_data-mover_MySQL_application It_skip-disconnected_Restore_hooks_tests_Successful_Init_hook_tc-id_OADP-164_interop_smoke_Tier0_MySQL_app_with_Kopia artifacts Checking if yq exists Installing yq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 12.9M 100 12.9M 0 0 34.0M 0 --:--:-- --:--:-- --:--:-- 34.0M Collecting original results in /logs/artifacts/original_results/ Patching Tests Result File: /logs/artifacts/junit_oadp_cnv_results.xml Mapping Test Suite Name To: OADP-lp-interop Patching Tests Result File: /logs/artifacts/junit_oadp_interop_results.xml Mapping Test Suite Name To: OADP-lp-interop Patching Tests Result File: /logs/artifacts/original_results/junit_oadp_cnv_results.xml Mapping Test Suite Name To: OADP-lp-interop Patching Tests Result File: /logs/artifacts/original_results/junit_oadp_interop_results.xml Mapping Test Suite Name To: OADP-lp-interop