Extract /home/jenkins/oadp-e2e-qe.tar.gz to /alabama/cspi Extract /home/jenkins/oadp-apps-deployer.tar.gz to /alabama/oadpApps Extract /home/jenkins/mtc-python-client.tar.gz to /alabama/pyclient Create and populate /tmp/test-settings... Login as Kubeadmin to the test cluster at https://api.ci-op-3mvpvvyj-07f22.cspilp.interop.ccitredhat.com:6443... WARNING: Using insecure TLS client config. Setting this option is not supported! Login successful. You have access to 75 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". Create virtual environment and install required packages... Collecting ansible_runner Downloading ansible_runner-2.4.0-py3-none-any.whl (79 kB) Collecting pyyaml Downloading PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB) Collecting pexpect>=4.5 Downloading pexpect-4.9.0-py2.py3-none-any.whl (63 kB) Collecting packaging Downloading packaging-24.1-py3-none-any.whl (53 kB) Collecting python-daemon Downloading python_daemon-3.0.1-py3-none-any.whl (31 kB) Collecting ptyprocess>=0.5 Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB) Collecting setuptools>=62.4.0 Downloading setuptools-75.0.0-py3-none-any.whl (1.2 MB) Collecting lockfile>=0.10 Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB) Collecting docutils Downloading docutils-0.21.2-py3-none-any.whl (587 kB) Installing collected packages: setuptools, ptyprocess, lockfile, docutils, pyyaml, python-daemon, pexpect, packaging, ansible-runner Attempting uninstall: setuptools Found existing installation: setuptools 57.4.0 Uninstalling setuptools-57.4.0: Successfully uninstalled setuptools-57.4.0 Successfully installed ansible-runner-2.4.0 docutils-0.21.2 lockfile-0.12.2 packaging-24.1 pexpect-4.9.0 ptyprocess-0.7.0 python-daemon-3.0.1 pyyaml-6.0.2 setuptools-75.0.0 WARNING: You are using pip version 21.2.3; however, version 24.2 is available. You should consider upgrading via the '/alabama/venv/bin/python3 -m pip install --upgrade pip' command. Processing /alabama/oadpApps DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing wheel metadata: started Preparing wheel metadata: finished with status 'done' Building wheels for collected packages: ocpdeployer Building wheel for ocpdeployer (PEP 517): started Building wheel for ocpdeployer (PEP 517): finished with status 'done' Created wheel for ocpdeployer: filename=ocpdeployer-0.0.1-py2.py3-none-any.whl size=224959 sha256=839bec4983ce264bcad2e29f4351c16be176559b486da1b839ef0e04883a991e Stored in directory: /tmp/pip-ephem-wheel-cache-_ovji8f1/wheels/ea/a6/87/9d98fc51cc395d30bd2147a7e53e3f5a2e80044d9dd9e64977 Successfully built ocpdeployer Installing collected packages: ocpdeployer WARNING: Value for scheme.platlib does not match. Please report this to distutils: /tmp/pip-target-j9xgl19a/lib64/python sysconfig: /tmp/pip-target-j9xgl19a/lib/python WARNING: Additional context: user = False home = '/tmp/pip-target-j9xgl19a' root = None prefix = None Successfully installed ocpdeployer-0.0.1 WARNING: You are using pip version 21.2.3; however, version 24.2 is available. You should consider upgrading via the '/alabama/venv/bin/python3 -m pip install --upgrade pip' command. Processing /alabama/pyclient DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. Collecting suds-py3 Downloading suds_py3-1.4.5.0-py3-none-any.whl (298 kB) Collecting requests Downloading requests-2.32.3-py3-none-any.whl (64 kB) Collecting jinja2 Downloading jinja2-3.1.4-py3-none-any.whl (133 kB) Collecting kubernetes==11.0.0 Downloading kubernetes-11.0.0-py3-none-any.whl (1.5 MB) Collecting openshift==0.11.2 Downloading openshift-0.11.2.tar.gz (19 kB) Collecting urllib3>=1.24.2 Downloading urllib3-2.2.3-py3-none-any.whl (126 kB) Collecting google-auth>=1.0.1 Downloading google_auth-2.34.0-py2.py3-none-any.whl (200 kB) Collecting requests-oauthlib Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl (24 kB) Requirement already satisfied: setuptools>=21.0.0 in /alabama/venv/lib/python3.10/site-packages (from kubernetes==11.0.0->mtc==0.0.1) (75.0.0) Requirement already satisfied: pyyaml>=3.12 in /alabama/venv/lib/python3.10/site-packages (from kubernetes==11.0.0->mtc==0.0.1) (6.0.2) Collecting certifi>=14.05.14 Downloading certifi-2024.8.30-py3-none-any.whl (167 kB) Collecting python-dateutil>=2.5.3 Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 Downloading websocket_client-1.8.0-py3-none-any.whl (58 kB) Collecting six>=1.9.0 Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting python-string-utils Downloading python_string_utils-1.0.0-py3-none-any.whl (26 kB) Collecting ruamel.yaml>=0.15 Downloading ruamel.yaml-0.18.6-py3-none-any.whl (117 kB) Collecting pyasn1-modules>=0.2.1 Downloading pyasn1_modules-0.4.1-py3-none-any.whl (181 kB) Collecting rsa<5,>=3.1.4 Downloading rsa-4.9-py3-none-any.whl (34 kB) Collecting cachetools<6.0,>=2.0.0 Downloading cachetools-5.5.0-py3-none-any.whl (9.5 kB) Collecting pyasn1<0.7.0,>=0.4.6 Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB) Collecting ruamel.yaml.clib>=0.2.7 Downloading ruamel.yaml.clib-0.2.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (526 kB) Collecting MarkupSafe>=2.0 Downloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB) Collecting charset-normalizer<4,>=2 Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB) Collecting idna<4,>=2.5 Downloading idna-3.10-py3-none-any.whl (70 kB) Collecting oauthlib>=3.0.0 Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB) Using legacy 'setup.py install' for mtc, since package 'wheel' is not installed. Using legacy 'setup.py install' for openshift, since package 'wheel' is not installed. Installing collected packages: urllib3, pyasn1, idna, charset-normalizer, certifi, six, rsa, requests, pyasn1-modules, oauthlib, cachetools, websocket-client, ruamel.yaml.clib, requests-oauthlib, python-dateutil, MarkupSafe, google-auth, ruamel.yaml, python-string-utils, kubernetes, jinja2, suds-py3, openshift, mtc Running setup.py install for openshift: started Running setup.py install for openshift: finished with status 'done' Running setup.py install for mtc: started Running setup.py install for mtc: finished with status 'done' Successfully installed MarkupSafe-2.1.5 cachetools-5.5.0 certifi-2024.8.30 charset-normalizer-3.3.2 google-auth-2.34.0 idna-3.10 jinja2-3.1.4 kubernetes-11.0.0 mtc-0.0.1 oauthlib-3.2.2 openshift-0.11.2 pyasn1-0.6.1 pyasn1-modules-0.4.1 python-dateutil-2.9.0.post0 python-string-utils-1.0.0 requests-2.32.3 requests-oauthlib-2.0.0 rsa-4.9 ruamel.yaml-0.18.6 ruamel.yaml.clib-0.2.8 six-1.16.0 suds-py3-1.4.5.0 urllib3-2.2.3 websocket-client-1.8.0 WARNING: You are using pip version 21.2.3; however, version 24.2 is available. You should consider upgrading via the '/alabama/venv/bin/python3 -m pip install --upgrade pip' command. go: downloading github.com/onsi/ginkgo/v2 v2.11.0 go: downloading github.com/vmware-tanzu/velero v1.12.0 go: downloading k8s.io/apimachinery v0.25.6 go: downloading sigs.k8s.io/controller-runtime v0.12.2 go: downloading github.com/onsi/gomega v1.27.8 go: downloading github.com/openshift/oadp-operator v1.0.2-0.20231003152846-925e5cc16370 go: downloading k8s.io/client-go v0.25.6 go: downloading k8s.io/api v0.25.6 go: downloading github.com/kubernetes-csi/external-snapshotter/client/v4 v4.2.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading k8s.io/utils v0.0.0-20230115233650-391b47cb4029 go: downloading github.com/apenella/go-ansible v1.1.5 go: downloading github.com/backube/volsync v0.7.0 go: downloading github.com/konveyor/volume-snapshot-mover v0.0.0-20230510202920-14b0e79dbaaa go: downloading github.com/openshift/api v0.0.0-20230414143018-3367bc7e6ac7 go: downloading github.com/andygrunwald/go-jira v1.16.0 go: downloading github.com/google/uuid v1.3.0 go: downloading github.com/operator-framework/api v0.14.1-0.20220413143725-33310d6154f3 go: downloading github.com/openshift/client-go v0.0.0-20211209144617-7385dd6338e3 go: downloading sigs.k8s.io/yaml v1.3.0 go: downloading k8s.io/kubectl v0.25.6 go: downloading github.com/go-logr/logr v1.2.4 go: downloading github.com/google/go-cmp v0.5.9 go: downloading github.com/apenella/go-common-utils/error v0.0.0-20210528133155-34ba915e28c8 go: downloading github.com/apenella/go-common-utils/data v0.0.0-20210528133155-34ba915e28c8 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/gofuzz v1.2.0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/spf13/pflag v1.0.5 go: downloading k8s.io/klog/v2 v2.90.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.3 go: downloading github.com/evanphx/json-patch v5.6.0+incompatible go: downloading github.com/fatih/structs v1.1.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading github.com/google/go-querystring v1.1.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/trivago/tgo v1.0.7 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading github.com/sirupsen/logrus v1.9.0 go: downloading golang.org/x/net v0.12.0 go: downloading github.com/imdario/mergo v0.3.13 go: downloading golang.org/x/term v0.10.0 go: downloading golang.org/x/sys v0.10.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/go-logr/zapr v1.2.3 go: downloading go.uber.org/zap v1.24.0 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/kube-openapi v0.0.0-20230118215034-64b6bb138190 go: downloading golang.org/x/time v0.3.0 go: downloading k8s.io/apiextensions-apiserver v0.26.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/spf13/cobra v1.6.1 go: downloading k8s.io/cli-runtime v0.25.6 go: downloading k8s.io/component-base v0.26.1 go: downloading github.com/moby/spdystream v0.2.0 go: downloading github.com/golang/protobuf v1.5.3 go: downloading github.com/google/gnostic v0.6.9 go: downloading google.golang.org/protobuf v1.30.0 go: downloading golang.org/x/oauth2 v0.7.0 go: downloading github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 go: downloading golang.org/x/tools v0.11.0 go: downloading golang.org/x/text v0.11.0 go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c go: downloading github.com/onsi/ginkgo v1.16.5 go: downloading github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 go: downloading github.com/stretchr/testify v1.8.2 go: downloading github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153 go: downloading go.uber.org/atomic v1.10.0 go: downloading gomodules.xyz/jsonpatch/v2 v2.2.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/bombsimon/logrusr/v3 v3.0.0 go: downloading github.com/prometheus/client_golang v1.15.0 go: downloading k8s.io/kube-aggregator v0.19.12 go: downloading github.com/jonboulle/clockwork v0.2.2 go: downloading k8s.io/component-helpers v0.26.1 go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de go: downloading github.com/daviddengcn/go-colortext v1.0.0 go: downloading github.com/docker/distribution v2.8.1+incompatible go: downloading github.com/moby/term v0.0.0-20220808134915-39b0c02b01ae go: downloading sigs.k8s.io/kustomize/kustomize/v4 v4.5.7 go: downloading sigs.k8s.io/kustomize/kyaml v0.13.9 go: downloading github.com/fvbommel/sortorder v1.0.1 go: downloading github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d go: downloading k8s.io/metrics v0.26.1 go: downloading github.com/lithammer/dedent v1.1.0 go: downloading github.com/chai2010/gettext-go v1.0.2 go: downloading github.com/MakeNowJust/heredoc v1.0.0 go: downloading github.com/mitchellh/go-wordwrap v1.0.0 go: downloading github.com/russross/blackfriday v1.5.2 go: downloading github.com/inconshreveable/mousetrap v1.1.0 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/go-openapi/swag v0.22.3 go: downloading github.com/google/pprof v0.0.0-20230705174524-200ffdc848b8 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/pmezard/go-difflib v1.0.0 go: downloading github.com/fsnotify/fsnotify v1.6.0 go: downloading github.com/robfig/cron v1.1.0 go: downloading github.com/hashicorp/go-hclog v1.2.0 go: downloading github.com/hashicorp/go-plugin v1.4.3 go: downloading go.uber.org/goleak v1.1.12 go: downloading github.com/stretchr/objx v0.5.0 go: downloading github.com/spf13/afero v1.9.3 go: downloading google.golang.org/grpc v1.54.0 go: downloading github.com/gobwas/glob v0.2.3 go: downloading sigs.k8s.io/kustomize/api v0.12.1 go: downloading github.com/fatih/camelcase v1.0.0 go: downloading github.com/golangplus/testing v1.0.0 go: downloading github.com/prometheus/client_model v0.3.0 go: downloading github.com/prometheus/common v0.42.0 go: downloading github.com/creack/pty v1.1.11 go: downloading gotest.tools/v3 v3.0.3 go: downloading github.com/emicklei/go-restful/v3 v3.10.1 go: downloading gotest.tools v2.2.0+incompatible go: downloading github.com/go-openapi/jsonreference v0.20.2 go: downloading google.golang.org/appengine v1.6.7 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/opencontainers/go-digest v1.0.0 go: downloading github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 go: downloading github.com/peterbourgon/diskv v2.0.1+incompatible go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.10.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.2.0 go: downloading github.com/prometheus/procfs v0.9.0 go: downloading github.com/Azure/azure-sdk-for-go v67.2.0+incompatible go: downloading github.com/Azure/go-autorest/autorest v0.11.27 go: downloading github.com/Azure/go-autorest/autorest/azure/auth v0.5.8 go: downloading github.com/aws/aws-sdk-go v1.44.253 go: downloading github.com/joho/godotenv v1.3.0 go: downloading github.com/Azure/go-autorest/autorest/to v0.3.0 go: downloading github.com/kopia/kopia v0.13.0 go: downloading github.com/fatih/color v1.15.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/Azure/go-autorest v14.2.0+incompatible go: downloading github.com/mattn/go-isatty v0.0.17 go: downloading github.com/mitchellh/go-testing-interface v1.0.0 go: downloading github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb go: downloading github.com/oklog/run v1.0.0 go: downloading github.com/jhump/protoreflect v1.6.0 go: downloading github.com/nxadm/tail v1.4.8 go: downloading github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f go: downloading github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4 go: downloading github.com/go-openapi/jsonpointer v0.19.6 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/google/btree v1.0.1 go: downloading google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 go: downloading github.com/go-errors/errors v1.0.1 go: downloading github.com/Azure/go-autorest/logger v0.2.1 go: downloading github.com/Azure/go-autorest/tracing v0.6.0 go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.2 go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.20 go: downloading github.com/Azure/go-autorest/autorest/azure/cli v0.4.2 go: downloading github.com/dimchansky/utfbom v1.1.1 go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 go: downloading github.com/xlab/treeprint v1.1.0 go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0 go: downloading golang.org/x/crypto v0.11.0 go: downloading golang.org/x/mod v0.12.0 go: downloading github.com/mitchellh/go-homedir v1.1.0 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading go.opentelemetry.io/otel v1.14.0 go: downloading go.opentelemetry.io/otel/trace v1.14.0 go: downloading golang.org/x/sync v0.3.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/edsrzf/mmap-go v1.1.0 go: downloading github.com/klauspost/compress v1.16.5 go: downloading github.com/klauspost/pgzip v1.2.5 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/gofrs/flock v0.8.1 go: downloading github.com/natefinch/atomic v1.0.1 go: downloading golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb go: downloading github.com/klauspost/reedsolomon v1.11.7 go: downloading github.com/zeebo/blake3 v0.2.3 go: downloading github.com/chmduquesne/rollinghash v4.0.0+incompatible go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.3.0 go: downloading github.com/Azure/azure-storage-blob-go v0.15.0 go: downloading cloud.google.com/go/storage v1.30.1 go: downloading google.golang.org/api v0.120.0 go: downloading cloud.google.com/go v0.110.0 go: downloading github.com/minio/minio-go/v7 v7.0.52 go: downloading go.starlark.net v0.0.0-20201006213952-227f4aabceb5 go: downloading github.com/sergi/go-diff v1.1.0 go: downloading github.com/zeebo/assert v1.1.0 go: downloading github.com/kopia/htmluibuild v0.0.0-20230326183719-f482ef17e2c9 go: downloading github.com/tg123/go-htpasswd v1.2.1 go: downloading github.com/zalando/go-keyring v0.2.2 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/Azure/azure-pipeline-go v0.2.3 go: downloading cloud.google.com/go/compute v1.19.0 go: downloading cloud.google.com/go/iam v0.13.0 go: downloading github.com/googleapis/gax-go/v2 v2.8.0 go: downloading golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading gopkg.in/ini.v1 v1.67.0 go: downloading github.com/minio/md5-simd v1.1.2 go: downloading github.com/minio/sha256-simd v1.0.0 go: downloading github.com/hanwen/go-fuse/v2 v2.3.0 go: downloading github.com/GehirnInc/crypt v0.0.0-20200316065508-bb7000b8a962 go: downloading github.com/mattn/go-ieproxy v0.0.1 go: downloading github.com/alessio/shellescape v1.4.1 go: downloading github.com/danieljoos/wincred v1.1.2 go: downloading github.com/godbus/dbus/v5 v5.1.0 go: downloading go.opencensus.io v0.24.0 go: downloading github.com/rs/xid v1.4.0 go: downloading cloud.google.com/go/longrunning v0.4.1 go: downloading github.com/frankban/quicktest v1.14.3 go: downloading github.com/google/martian/v3 v3.3.2 go: downloading github.com/google/martian v2.1.0+incompatible go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/zeebo/pcg v1.0.1 go: downloading github.com/jmespath/go-jmespath/internal/testify v1.5.1 go: downloading github.com/google/s2a-go v0.1.2 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.3 go: downloading github.com/Azure/go-autorest/autorest/validation v0.2.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v0.21.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v0.8.3 go: downloading github.com/dnaeon/go-vcr v1.2.0 storageclass.storage.k8s.io/gp2-csi annotated storageclass.storage.k8s.io/gp3-csi annotated storageclass.storage.k8s.io/ocs-storagecluster-ceph-rbd annotated storageclass.storage.k8s.io/ocs-storagecluster-ceph-rbd-virtualization annotated storageclass.storage.k8s.io/ocs-storagecluster-cephfs annotated storageclass.storage.k8s.io/openshift-storage.noobaa.io annotated storageclass.storage.k8s.io/ocs-storagecluster-ceph-rbd annotated + readonly 'RED=\e[31m' + RED='\e[31m' + readonly 'BLUE=\033[34m' + BLUE='\033[34m' + readonly 'CLEAR=\e[39m' + CLEAR='\e[39m' ++ oc get infrastructures cluster -o 'jsonpath={.status.platform}' ++ awk '{print tolower($0)}' + CLOUD_PROVIDER=aws + [[ aws == *-arm* ]] + [[ aws == *-fips* ]] + E2E_TIMEOUT_MULTIPLIER=2 + export NAMESPACE=openshift-adp + NAMESPACE=openshift-adp + export PROVIDER=aws + PROVIDER=aws ++ echo aws ++ awk '{print tolower($0)}' + BACKUP_LOCATION=aws + export BACKUP_LOCATION=aws + BACKUP_LOCATION=aws + export BUCKET=ci-op-3mvpvvyj-interopoadp + BUCKET=ci-op-3mvpvvyj-interopoadp + OADP_CREDS_FILE=/tmp/test-settings/credentials + OADP_VSL_CREDS_FILE=/tmp/test-settings/aws_vsl_creds +++ readlink -f /alabama/cspi/test_settings/scripts/test_runner.sh ++ dirname /alabama/cspi/test_settings/scripts/test_runner.sh + readonly SCRIPT_DIR=/alabama/cspi/test_settings/scripts + SCRIPT_DIR=/alabama/cspi/test_settings/scripts ++ cd /alabama/cspi/test_settings/scripts ++ git rev-parse --show-toplevel + readonly TOP_DIR=/alabama/cspi + TOP_DIR=/alabama/cspi + echo /alabama/cspi /alabama/cspi + TESTS_FOLDER=/alabama/cspi/e2e/kubevirt-plugin ++ oc get nodes -o 'jsonpath={.items[*].metadata.labels.kubernetes\.io/arch}' ++ tr ' ' '\n' ++ sort -u ++ xargs + export NODES_ARCHITECTURE=amd64 + NODES_ARCHITECTURE=amd64 + export OADP_REPOSITORY=redhat + OADP_REPOSITORY=redhat + SKIP_DPA_CREATION=false ++ oc get ns openshift-storage ++ echo true + OPENSHIFT_STORAGE=true + '[' redhat == upstream-velero ']' + '[' true == true ']' ++ oc get sc ++ awk '$1 ~ /^.+ceph-rbd$/ {print $1}' ++ tail -1 + CEPH_RBD_STORAGE_CLASS=ocs-storagecluster-ceph-rbd + '[' -n ocs-storagecluster-ceph-rbd ']' + export CEPH_RBD_STORAGE_CLASS + echo 'ceph-rbd StorageClass found: ocs-storagecluster-ceph-rbd' ceph-rbd StorageClass found: ocs-storagecluster-ceph-rbd ++ oc get storageclass -o 'jsonpath={range .items[*]}{@.metadata.name}{" "}{@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}{end}' ++ awk '$2=="true"{print $1}' ++ wc -l + NUM_DEFAULT_STORAGE_CLASS=1 + '[' 1 -ne 1 ']' ++ oc get storageclass -o 'jsonpath={.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=='\''true'\'')].metadata.name}' + DEFAULT_SC=ocs-storagecluster-ceph-rbd + export STORAGE_CLASS=ocs-storagecluster-ceph-rbd + STORAGE_CLASS=ocs-storagecluster-ceph-rbd + '[' -n ocs-storagecluster-ceph-rbd ']' + '[' ocs-storagecluster-ceph-rbd '!=' ocs-storagecluster-ceph-rbd ']' + export STORAGE_CLASS_OPENSHIFT_STORAGE=ocs-storagecluster-ceph-rbd + STORAGE_CLASS_OPENSHIFT_STORAGE=ocs-storagecluster-ceph-rbd + echo 'Using the StorageClass for openshift-storage: ocs-storagecluster-ceph-rbd' Using the StorageClass for openshift-storage: ocs-storagecluster-ceph-rbd + [[ amd64 != \a\m\d\6\4 ]] + TEST_FILTER='!// || (// && !exclude_aws && (!/target/ || target_aws) ) ' + [[ aws =~ ^osp ]] + [[ aws =~ ^vsphere ]] + [[ aws =~ ^gcp-wif ]] + [[ aws =~ ^ibmcloud ]] ++ oc config current-context ++ awk -F / '{print $2}' + SETTINGS_TMP=/alabama/cspi/output_files/api-ci-op-3mvpvvyj-07f22-cspilp-interop-ccitredhat-com:6443 + mkdir -p /alabama/cspi/output_files/api-ci-op-3mvpvvyj-07f22-cspilp-interop-ccitredhat-com:6443 ++ oc get authentication cluster -o 'jsonpath={.spec.serviceAccountIssuer}' + IS_OIDC= + '[' '!' -z ']' + [[ aws == \a\w\s ]] + export PROVIDER=aws + PROVIDER=aws + export CREDS_SECRET_REF=cloud-credentials + CREDS_SECRET_REF=cloud-credentials ++ oc get infrastructures cluster -o 'jsonpath={.status.platformStatus.aws.region}' --allow-missing-template-keys=false + export REGION=us-east-1 + REGION=us-east-1 + settings_script=aws_settings.sh + '[' aws == aws-sts ']' + BUCKET=ci-op-3mvpvvyj-interopoadp + TMP_DIR=/alabama/cspi/output_files/api-ci-op-3mvpvvyj-07f22-cspilp-interop-ccitredhat-com:6443 + source /alabama/cspi/test_settings/scripts/aws_settings.sh ++ cat ++ [[ aws == *aws* ]] ++ cat ++ echo -e '\n }\n}' +++ cat /alabama/cspi/output_files/api-ci-op-3mvpvvyj-07f22-cspilp-interop-ccitredhat-com:6443/settings.json ++ x='{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-1" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-3mvpvvyj-interopoadp" } } } ] , "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-1" } } } ] } }' ++ echo '{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-1" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-3mvpvvyj-interopoadp" } } } ] , "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-1" } } } ] } }' ++ grep -o '^[^#]*' + FILE_SETTINGS_NAME=settings.json + printf '\033[34mGenerated settings file under /alabama/cspi/output_files/api-ci-op-3mvpvvyj-07f22-cspilp-interop-ccitredhat-com:6443/settings.json\e[39m\n' Generated settings file under /alabama/cspi/output_files/api-ci-op-3mvpvvyj-07f22-cspilp-interop-ccitredhat-com:6443/settings.json + cat /alabama/cspi/output_files/api-ci-op-3mvpvvyj-07f22-cspilp-interop-ccitredhat-com:6443/settings.json ++ oc get volumesnapshotclass -o name + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc annotated + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/ocs-storagecluster-cephfsplugin-snapclass snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/ocs-storagecluster-cephfsplugin-snapclass annotated + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/ocs-storagecluster-rbdplugin-snapclass snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/ocs-storagecluster-rbdplugin-snapclass annotated ++ ./e2e/must-gather/get-latest-build.sh + oc get configmaps -n default must-gather-image + UPSTREAM_VERSION=99.0.0 ++ oc get OperatorCondition -n openshift-adp -o 'jsonpath={.items[*].metadata.name}' ++ awk -F v '{print $2}' + OADP_VERSION=1.4.0 + '[' -z 1.4.0 ']' + '[' 1.4.0 == 99.0.0 ']' ++ oc get sub redhat-oadp-operator -n openshift-adp -o 'jsonpath={.spec.source}' + OADP_REPO=redhat-operators + '[' -z redhat-operators ']' + '[' redhat-operators == redhat-operators ']' + REGISTRY_PATH=registry.redhat.io/oadp/oadp-mustgather-rhel9: + TAG=1.4.0 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + echo registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + '[' -z registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 ']' + export NUM_OF_OADP_INSTANCES=1 + NUM_OF_OADP_INSTANCES=1 + ginkgo run -p --nodes=1 -mod=mod /alabama/cspi/e2e/kubevirt-plugin/ --show-node-events -- -credentials_file=/tmp/test-settings/credentials -vsl_credentials_file=/tmp/test-settings/aws_vsl_creds -oadp_namespace=openshift-adp -settings=/alabama/cspi/output_files/api-ci-op-3mvpvvyj-07f22-cspilp-interop-ccitredhat-com:6443/settings.json -must_gather_image=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 -timeout_multiplier=2 -skip_dpa_creation=false --ginkgo.junit-report=/logs/artifacts/junit_oadp_cnv_results.xml '--ginkgo.label-filter=!// || (// && !exclude_aws && (!/target/ || target_aws) ) ' --ginkgo.skip=tc-id:OADP-555 Ginkgo detected a version mismatch between the Ginkgo CLI and the version of Ginkgo imported by your packages: Ginkgo CLI Version: 2.20.2 Mismatched package versions found: 2.11.0 used by kubevirt-plugin Ginkgo will continue to attempt to run but you may see errors (including flag parsing errors) and should either update your go.mod or your version of the Ginkgo CLI to match. To install the matching version of the CLI run go install github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file. Alternatively you can use go run github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file to invoke the matching version of the Ginkgo CLI. If you are attempting to test multiple packages that each have a different version of the Ginkgo library with a single Ginkgo CLI that is currently unsupported.  2024/09/16 07:20:16 Setting up clients 2024/09/16 07:20:16 Getting default StorageClass... 2024/09/16 07:20:16 Checking default storage class count Run the command: oc get sc 2024/09/16 07:20:16 Got default StorageClass ocs-storagecluster-ceph-rbd 2024/09/16 07:20:16 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 50m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 50m ocs-storagecluster-ceph-rbd (default) openshift-storage.rbd.csi.ceph.com Delete Immediate true 4m10s ocs-storagecluster-ceph-rbd-virtualization openshift-storage.rbd.csi.ceph.com Delete Immediate true 4m10s ocs-storagecluster-cephfs openshift-storage.cephfs.csi.ceph.com Delete Immediate true 3m59s openshift-storage.noobaa.io openshift-storage.noobaa.io/obc Delete Immediate false 3m5s 2024/09/16 07:20:16 Using velero prefix: velero-e2e-kubevirt-1f45bd4b-73fc-11ef-abd2-0a580a83d092 Running Suite: OADP E2E Virtualization Workloads Suite - /alabama/cspi/e2e/kubevirt-plugin ========================================================================================== Random Seed: 1726471170 Will run 4 of 5 specs ------------------------------ [BeforeSuite]  /alabama/cspi/e2e/kubevirt-plugin/kubevirt_suite_test.go:62 [BeforeSuite] PASSED [0.019 seconds] ------------------------------ S ------------------------------ Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads  [tc-id:OADP-401] [kubevirt] Started VM should over ceph filesytem mode /alabama/cspi/e2e/kubevirt-plugin/backup_restore_datamover.go:129 2024/09/16 07:20:16 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 09/16/24 07:20:16.562 2024/09/16 07:20:16 native-datamover 2024/09/16 07:20:16 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "5d34b397-0dfc-4dbc-a665-d1d6c95b2fd0", "resourceVersion": "56984", "generation": 1, "creationTimestamp": "2024-09-16T07:20:16Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-09-16T07:20:16Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-1" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-3mvpvvyj-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 09/16/24 07:20:16.583 2024/09/16 07:20:16 Waiting for velero pod to be running 2024/09/16 07:20:21 pod: velero-6985485bdb-ns6g5 is not yet running with status: {Pending [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:16 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:16 +0000 UTC ContainersNotInitialized containers with incomplete status: [openshift-velero-plugin velero-plugin-for-aws kubevirt-velero-plugin]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:16 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:16 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:16 +0000 UTC }] 10.0.33.175 [] 2024-09-16 07:20:16 +0000 UTC [{openshift-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:1eea6e649be9406b56a02061c70eff02fb61f1be20afc116ec85f90d2ef74d15 0xc000c06fab} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:c328548fd34a8d790e002fac1e9e7327231fd207a83612cfa2aa49991ea0bf47 0xc000c06fac} {kubevirt-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:cf0d86eeb16076ed4c520fdda6b74b0588c36d6a18f3911b069dc4e32b13501c 0xc000c06fad}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:777ec93a24cafd94022804da68d02da6f64041716ec4a39a85017d7289846d72 0xc000c06fe6}] Burstable []} 2024/09/16 07:20:26 pod: velero-6985485bdb-ns6g5 is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:22 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:16 +0000 UTC ContainersNotInitialized containers with incomplete status: [kubevirt-velero-plugin]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:16 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:16 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-09-16 07:20:16 +0000 UTC }] 10.0.33.175 10.128.2.46 [{10.128.2.46}] 2024-09-16 07:20:16 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-09-16 07:20:22 +0000 UTC,FinishedAt:2024-09-16 07:20:22 +0000 UTC,ContainerID:cri-o://9de5aa427b9942fee2d0261eaa664ae4695e2146511557016b6c62a0140919a9,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:1eea6e649be9406b56a02061c70eff02fb61f1be20afc116ec85f90d2ef74d15 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:1eea6e649be9406b56a02061c70eff02fb61f1be20afc116ec85f90d2ef74d15 cri-o://9de5aa427b9942fee2d0261eaa664ae4695e2146511557016b6c62a0140919a9 0xc001019ca9} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-09-16 07:20:25 +0000 UTC,FinishedAt:2024-09-16 07:20:25 +0000 UTC,ContainerID:cri-o://c557a8e55c9ebd77438bb3f58cbf4a1341daff434b1772a32a2e03f627c5d4dc,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:c328548fd34a8d790e002fac1e9e7327231fd207a83612cfa2aa49991ea0bf47 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:64902371978127f649f7ceedc569ee5bc1e2b2f7ed876fce2d5404a96dbef1e4 cri-o://c557a8e55c9ebd77438bb3f58cbf4a1341daff434b1772a32a2e03f627c5d4dc 0xc001019cb9} {kubevirt-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:cf0d86eeb16076ed4c520fdda6b74b0588c36d6a18f3911b069dc4e32b13501c 0xc001019cba}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:777ec93a24cafd94022804da68d02da6f64041716ec4a39a85017d7289846d72 0xc001019cd6}] Burstable []} 2024/09/16 07:20:31 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 09/16/24 07:20:31.616 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/09/16 07:20:31 The 'openshift-storage' namespace exists 2024/09/16 07:20:31 Checking default storage class count 2024/09/16 07:20:31 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2024/09/16 07:20:31 Snapclass 'example-snapclass' doesn't exist, creating 2024/09/16 07:20:31 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2024/09/16 07:20:31 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd 2024/09/16 07:20:31 Checking for correct number of running NodeAgent pods... STEP: Installing application for case ocp-kubevirt @ 09/16/24 07:20:31.763 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] /usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ci-op-3mvpvvyj-07f22.cspilp.interop.ccitredhat.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Deploy vm test-vm] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (60 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (59 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (58 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (57 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (56 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (55 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (54 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (53 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (52 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (51 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (50 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (49 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (48 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (47 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (46 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (45 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (44 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (43 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (42 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (41 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (40 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (39 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (38 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (37 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (36 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (35 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (34 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (33 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (32 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (31 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (30 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (29 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (28 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (27 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (26 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (25 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (24 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (23 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (22 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (21 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (20 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (19 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (18 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (17 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (16 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (15 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (14 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (13 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (12 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (11 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (10 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (9 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (8 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (7 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (6 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (5 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (4 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (3 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (2 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (1 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** fatal: [localhost]: FAILED! => {"api_found": true, "attempts": 60, "changed": false, "resources": [{"apiVersion": "kubevirt.io/v1", "kind": "VirtualMachine", "metadata": {"annotations": {"kubevirt.io/latest-observed-api-version": "v1", "kubevirt.io/storage-observed-api-version": "v1"}, "creationTimestamp": "2024-09-16T07:20:35Z", "finalizers": ["kubevirt.io/virtualMachineControllerFinalize"], "generation": 1, "managedFields": [{"apiVersion": "kubevirt.io/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:dataVolumeTemplates": {}, "f:running": {}, "f:template": {".": {}, "f:metadata": {".": {}, "f:creationTimestamp": {}, "f:name": {}}, "f:spec": {".": {}, "f:domain": {".": {}, "f:devices": {".": {}, "f:disks": {}, "f:interfaces": {}, "f:rng": {}}, "f:machine": {".": {}, "f:type": {}}, "f:resources": {".": {}, "f:requests": {".": {}, "f:memory": {}}}}, "f:networks": {}, "f:terminationGracePeriodSeconds": {}, "f:volumes": {}}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2024-09-16T07:20:35Z"}, {"apiVersion": "kubevirt.io/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:metadata": {"f:annotations": {"f:kubevirt.io/latest-observed-api-version": {}, "f:kubevirt.io/storage-observed-api-version": {}}, "f:finalizers": {".": {}, "v:\"kubevirt.io/virtualMachineControllerFinalize\"": {}}}}, "manager": "virt-controller", "operation": "Update", "time": "2024-09-16T07:20:35Z"}, {"apiVersion": "kubevirt.io/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {".": {}, "f:conditions": {}, "f:printableStatus": {}, "f:volumeSnapshotStatuses": {}}}, "manager": "virt-controller", "operation": "Update", "subresource": "status", "time": "2024-09-16T07:20:35Z"}], "name": "test-vm", "namespace": "test-oadp-401", "resourceVersion": "57444", "uid": "4f594c65-b1cc-42d5-ae1a-b201870f19e6"}, "spec": {"dataVolumeTemplates": [{"metadata": {"annotations": {"cdi.kubevirt.io/storage.deleteAfterCompletion": "false"}, "creationTimestamp": null, "name": "test-vm-dv"}, "spec": {"pvc": {"accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "5Gi"}}, "storageClassName": "odf-operator-cephfs", "volumeMode": "Filesystem"}, "source": {"registry": {"pullMethod": "node", "url": "docker://quay.io/kubevirt/fedora-with-test-tooling-container-disk"}}}}], "running": true, "template": {"metadata": {"creationTimestamp": null, "name": "test-vm"}, "spec": {"architecture": "amd64", "domain": {"devices": {"disks": [{"disk": {"bus": "virtio"}, "name": "volume0"}, {"disk": {"bus": "virtio"}, "name": "volume1"}], "interfaces": [{"macAddress": "02:a5:a2:00:00:00", "masquerade": {}, "name": "default"}], "rng": {}}, "machine": {"type": "q35"}, "resources": {"requests": {"memory": "256M"}}}, "networks": [{"name": "default", "pod": {}}], "terminationGracePeriodSeconds": 0, "volumes": [{"dataVolume": {"name": "test-vm-dv"}, "name": "volume0"}, {"cloudInitNoCloud": {"networkData": "ethernets:\n eth0:\n addresses:\n - fd10:0:2::2/120\n dhcp4: true\n gateway6: fd10:0:2::1\n match: {}\n nameservers:\n addresses:\n - 10.96.0.10\n search:\n - default.svc.cluster.local\n - svc.cluster.local\n - cluster.local\nversion: 2"}, "name": "volume1"}]}}}, "status": {"conditions": [{"lastProbeTime": "2024-09-16T07:20:35Z", "lastTransitionTime": "2024-09-16T07:20:35Z", "message": "VMI does not exist", "reason": "VMINotExists", "status": "False", "type": "Ready"}], "printableStatus": "Provisioning", "volumeSnapshotStatuses": [{"enabled": false, "name": "volume0", "reason": "No VolumeSnapshotClass: Volume snapshots are not configured for this StorageClass [odf-operator-cephfs] [volume0]"}, {"enabled": false, "name": "volume1", "reason": "Snapshot is not supported for this volumeSource type [volume1]"}]}}]} PLAY RECAP ********************************************************************* localhost : ok=12  changed=5  unreachable=0 failed=1  skipped=1  rescued=0 ignored=0 2024/09/16 07:26:08 2024-09-16 07:20:33,161 p=21295 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:20:33,161 p=21295 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:20:33,390 p=21295 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:20:33,390 p=21295 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:20:33,616 p=21295 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:20:33,616 p=21295 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:20:33,630 p=21295 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:20:33,631 p=21295 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:20:33,962 p=21295 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:20:33,963 p=21295 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:20:33,989 p=21295 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:20:33,990 p=21295 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:20:34,007 p=21295 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:20:34,008 p=21295 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:20:34,017 p=21295 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:20:34,545 p=21295 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:20:34,546 p=21295 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:20:35,283 p=21295 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Create namespace] *** 2024-09-16 07:20:35,284 p=21295 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:20:35,284 p=21295 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:20:35,906 p=21295 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Deploy vm test-vm] *** 2024-09-16 07:20:35,906 p=21295 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:26:08,110 p=21295 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** 2024-09-16 07:26:08,111 p=21295 u=1006570000 n=ansible | fatal: [localhost]: FAILED! => {"api_found": true, "attempts": 60, "changed": false, "resources": [{"apiVersion": "kubevirt.io/v1", "kind": "VirtualMachine", "metadata": {"annotations": {"kubevirt.io/latest-observed-api-version": "v1", "kubevirt.io/storage-observed-api-version": "v1"}, "creationTimestamp": "2024-09-16T07:20:35Z", "finalizers": ["kubevirt.io/virtualMachineControllerFinalize"], "generation": 1, "managedFields": [{"apiVersion": "kubevirt.io/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:dataVolumeTemplates": {}, "f:running": {}, "f:template": {".": {}, "f:metadata": {".": {}, "f:creationTimestamp": {}, "f:name": {}}, "f:spec": {".": {}, "f:domain": {".": {}, "f:devices": {".": {}, "f:disks": {}, "f:interfaces": {}, "f:rng": {}}, "f:machine": {".": {}, "f:type": {}}, "f:resources": {".": {}, "f:requests": {".": {}, "f:memory": {}}}}, "f:networks": {}, "f:terminationGracePeriodSeconds": {}, "f:volumes": {}}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2024-09-16T07:20:35Z"}, {"apiVersion": "kubevirt.io/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:metadata": {"f:annotations": {"f:kubevirt.io/latest-observed-api-version": {}, "f:kubevirt.io/storage-observed-api-version": {}}, "f:finalizers": {".": {}, "v:\"kubevirt.io/virtualMachineControllerFinalize\"": {}}}}, "manager": "virt-controller", "operation": "Update", "time": "2024-09-16T07:20:35Z"}, {"apiVersion": "kubevirt.io/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {".": {}, "f:conditions": {}, "f:printableStatus": {}, "f:volumeSnapshotStatuses": {}}}, "manager": "virt-controller", "operation": "Update", "subresource": "status", "time": "2024-09-16T07:20:35Z"}], "name": "test-vm", "namespace": "test-oadp-401", "resourceVersion": "57444", "uid": "4f594c65-b1cc-42d5-ae1a-b201870f19e6"}, "spec": {"dataVolumeTemplates": [{"metadata": {"annotations": {"cdi.kubevirt.io/storage.deleteAfterCompletion": "false"}, "creationTimestamp": null, "name": "test-vm-dv"}, "spec": {"pvc": {"accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "5Gi"}}, "storageClassName": "odf-operator-cephfs", "volumeMode": "Filesystem"}, "source": {"registry": {"pullMethod": "node", "url": "docker://quay.io/kubevirt/fedora-with-test-tooling-container-disk"}}}}], "running": true, "template": {"metadata": {"creationTimestamp": null, "name": "test-vm"}, "spec": {"architecture": "amd64", "domain": {"devices": {"disks": [{"disk": {"bus": "virtio"}, "name": "volume0"}, {"disk": {"bus": "virtio"}, "name": "volume1"}], "interfaces": [{"macAddress": "02:a5:a2:00:00:00", "masquerade": {}, "name": "default"}], "rng": {}}, "machine": {"type": "q35"}, "resources": {"requests": {"memory": "256M"}}}, "networks": [{"name": "default", "pod": {}}], "terminationGracePeriodSeconds": 0, "volumes": [{"dataVolume": {"name": "test-vm-dv"}, "name": "volume0"}, {"cloudInitNoCloud": {"networkData": "ethernets:\n eth0:\n addresses:\n - fd10:0:2::2/120\n dhcp4: true\n gateway6: fd10:0:2::1\n match: {}\n nameservers:\n addresses:\n - 10.96.0.10\n search:\n - default.svc.cluster.local\n - svc.cluster.local\n - cluster.local\nversion: 2"}, "name": "volume1"}]}}}, "status": {"conditions": [{"lastProbeTime": "2024-09-16T07:20:35Z", "lastTransitionTime": "2024-09-16T07:20:35Z", "message": "VMI does not exist", "reason": "VMINotExists", "status": "False", "type": "Ready"}], "printableStatus": "Provisioning", "volumeSnapshotStatuses": [{"enabled": false, "name": "volume0", "reason": "No VolumeSnapshotClass: Volume snapshots are not configured for this StorageClass [odf-operator-cephfs] [volume0]"}, {"enabled": false, "name": "volume1", "reason": "Snapshot is not supported for this volumeSource type [volume1]"}]}}]} 2024-09-16 07:26:08,112 p=21295 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:26:08,112 p=21295 u=1006570000 n=ansible | localhost : ok=12 changed=5 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-401 2024/09/16 07:26:08 LAST SEEN TYPE REASON OBJECT MESSAGE 4s Warning ProvisioningFailed persistentvolumeclaim/test-vm-dv storageclass.storage.k8s.io "odf-operator-cephfs" not found 5m33s Normal NotFound datavolume/test-vm-dv No PVC found 5m33s Normal Pending datavolume/test-vm-dv PVC test-vm-dv Pending 5m33s Normal SuccessfulDataVolumeCreate virtualmachine/test-vm Created DataVolume test-vm-dv [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:46 @ 09/16/24 07:26:08.266 2024/09/16 07:26:08 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 STEP: Get the failed spec name @ 09/16/24 07:26:08.266 2024/09/16 07:26:08 The failed spec name is: Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads [tc-id:OADP-401] [kubevirt] Started VM should over ceph filesytem mode STEP: Create a folder for all must-gather files if it doesn't exists already @ 09/16/24 07:26:08.266 2024/09/16 07:26:08 The folder logs does not exists, creating new folder with the name: logs STEP: Create a folder for the failed spec if it doesn't exists already @ 09/16/24 07:26:08.266 2024/09/16 07:26:08 The folder logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Started_VM_should_over_ceph_filesytem_mode does not exists, creating new folder with the name: logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Started_VM_should_over_ceph_filesytem_mode STEP: Run must-gather because the spec failed @ 09/16/24 07:26:08.266 2024/09/16 07:26:08 Log the present working directory path:- /alabama/cspi/e2e/kubevirt-plugin 2024/09/16 07:26:08 [adm must-gather --dest-dir /alabama/cspi/e2e/kubevirt-plugin/logs/It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Started_VM_should_over_ceph_filesytem_mode --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0] 2024/09/16 07:26:37 Log all the files present in /alabama/cspi/e2e/kubevirt-plugin/logs directory 2024/09/16 07:26:37 It_Native_CSI_Data_Mover_Backup_Restore_Openshift_Virtualization_Workloads_tc-id_OADP-401_kubevirt_Started_VM_should_over_ceph_filesytem_mode STEP: Find must-gather folder and rename it to a shorter more readable name @ 09/16/24 07:26:37.142 2024/09/16 07:26:37 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-401] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=8  rescued=0 ignored=0 2024/09/16 07:26:55 2024-09-16 07:26:38,581 p=22362 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:26:38,581 p=22362 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:26:38,821 p=22362 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:26:38,821 p=22362 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:26:39,059 p=22362 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:26:39,060 p=22362 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:26:39,075 p=22362 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:26:39,075 p=22362 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:26:39,361 p=22362 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:26:39,361 p=22362 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:26:39,389 p=22362 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:26:39,389 p=22362 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:26:39,407 p=22362 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:26:39,408 p=22362 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:26:39,418 p=22362 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:26:39,961 p=22362 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:26:39,961 p=22362 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:26:55,753 p=22362 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-401] *** 2024-09-16 07:26:55,753 p=22362 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:26:55,753 p=22362 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:26:55,924 p=22362 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:26:55,925 p=22362 u=1006570000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0 2024/09/16 07:26:55 Cleaning setup resources for the backup 2024/09/16 07:26:55 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2024/09/16 07:26:55 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd 2024/09/16 07:26:55 Deleting VolumeSnapshotClass 'example-snapclass' • [FAILED] [399.448 seconds] Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-401] [kubevirt] Started VM should over ceph filesytem mode /alabama/cspi/e2e/kubevirt-plugin/backup_restore_datamover.go:129 [FAILED] Unexpected error: <*errors.Error | 0xc000752040>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"namespace":"test-oadp-401","storage_class":"odf-operator-cephfs","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt","vm_disk_pvc_volume_mode":"Filesystem","with_deploy":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"namespace\":\"test-oadp-401\",\"storage_class\":\"odf-operator-cephfs\",\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt\",\"vm_disk_pvc_volume_mode\":\"Filesystem\",\"with_deploy\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:46 @ 09/16/24 07:26:08.266 ------------------------------ CSI: Backup/Restore Openshift Virtualization Workloads  [tc-id:OADP-185] [kubevirt] Backing up started VM should succeed /alabama/cspi/e2e/kubevirt-plugin/backup_restore_csi.go:34 2024/09/16 07:26:56 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 09/16/24 07:26:56.03 2024/09/16 07:26:56 csi 2024/09/16 07:26:56 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "cc674239-de0b-4e96-8e86-0359aad008f1", "resourceVersion": "62861", "generation": 1, "creationTimestamp": "2024-09-16T07:26:56Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-09-16T07:26:56Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-1" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-3mvpvvyj-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 09/16/24 07:26:56.111 2024/09/16 07:26:56 Waiting for velero pod to be running 2024/09/16 07:26:56 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/09/16 07:26:56 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "cc674239-de0b-4e96-8e86-0359aad008f1", "resourceVersion": "62861", "generation": 1, "creationTimestamp": "2024-09-16T07:26:56Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-09-16T07:26:56Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-1" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-3mvpvvyj-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 09/16/24 07:27:01.129 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/09/16 07:27:01 The 'openshift-storage' namespace exists 2024/09/16 07:27:01 Checking default storage class count 2024/09/16 07:27:01 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2024/09/16 07:27:01 Snapclass 'example-snapclass' doesn't exist, creating 2024/09/16 07:27:01 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2024/09/16 07:27:01 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd STEP: Installing application for case ocp-kubevirt @ 09/16/24 07:27:01.275 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Deploy vm test-vm] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (60 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (59 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (58 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (57 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (56 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (55 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (54 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (53 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (52 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=13  changed=5  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/09/16 07:27:57 2024-09-16 07:27:03,688 p=22587 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:27:03,688 p=22587 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:27:03,926 p=22587 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:27:03,926 p=22587 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:27:04,164 p=22587 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:27:04,164 p=22587 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:27:04,180 p=22587 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:27:04,180 p=22587 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:04,476 p=22587 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:27:04,476 p=22587 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:04,504 p=22587 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:27:04,504 p=22587 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:04,524 p=22587 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:27:04,525 p=22587 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:04,535 p=22587 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:27:05,077 p=22587 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:27:05,077 p=22587 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:05,843 p=22587 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Create namespace] *** 2024-09-16 07:27:05,843 p=22587 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:27:05,843 p=22587 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:27:06,498 p=22587 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Deploy vm test-vm] *** 2024-09-16 07:27:06,498 p=22587 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:27:57,269 p=22587 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** 2024-09-16 07:27:57,270 p=22587 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:57,381 p=22587 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:27:57,381 p=22587 u=1006570000 n=ansible | localhost : ok=13 changed=5 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 09/16/24 07:27:57.425 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** ok: [localhost] FAILED - RETRYING: [localhost]: Wait for VM to have AgentConnected status True indicating the guest agent is running (60 retries left). FAILED - RETRYING: [localhost]: Wait for VM to have AgentConnected status True indicating the guest agent is running (59 retries left). FAILED - RETRYING: [localhost]: Wait for VM to have AgentConnected status True indicating the guest agent is running (58 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to have AgentConnected status True indicating the guest agent is running] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=12  changed=3  unreachable=0 failed=0 skipped=7  rescued=0 ignored=0 2024/09/16 07:28:18 2024-09-16 07:27:58,835 p=22933 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:27:58,835 p=22933 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:27:59,067 p=22933 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:27:59,067 p=22933 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:27:59,310 p=22933 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:27:59,310 p=22933 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:27:59,325 p=22933 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:27:59,325 p=22933 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:59,618 p=22933 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:27:59,618 p=22933 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:59,646 p=22933 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:27:59,646 p=22933 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:59,664 p=22933 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:27:59,665 p=22933 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:27:59,676 p=22933 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:28:00,230 p=22933 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:28:00,230 p=22933 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:28:01,098 p=22933 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** 2024-09-16 07:28:01,098 p=22933 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:28:01,098 p=22933 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:28:18,352 p=22933 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to have AgentConnected status True indicating the guest agent is running] *** 2024-09-16 07:28:18,353 p=22933 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:28:18,366 p=22933 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:28:18,366 p=22933 u=1006570000 n=ansible | localhost : ok=12 changed=3 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0 2024/09/16 07:28:18 {{ } { } [{{ } {test-vm-dv test-oadp-185 443d7654-5a35-4321-8001-b7d870a43238 63876 0 2024-09-16 07:27:06 +0000 UTC map[app:containerized-data-importer app.kubernetes.io/component:storage app.kubernetes.io/managed-by:cdi-controller app.kubernetes.io/part-of:hyperconverged-cluster app.kubernetes.io/version:4.16.1 kubevirt.io/created-by:fa469ad4-c0f6-41db-bcc7-a2cc909ac82b] map[cdi.kubevirt.io/allowClaimAdoption:true cdi.kubevirt.io/createdForDataVolume:9ac161ff-97e8-4727-b99a-b4d25e01ab77 cdi.kubevirt.io/storage.condition.running:false cdi.kubevirt.io/storage.condition.running.message:Import Complete cdi.kubevirt.io/storage.condition.running.reason:Completed cdi.kubevirt.io/storage.contentType:kubevirt cdi.kubevirt.io/storage.deleteAfterCompletion:false cdi.kubevirt.io/storage.pod.phase:Succeeded cdi.kubevirt.io/storage.pod.restarts:0 cdi.kubevirt.io/storage.populator.progress:100.0% cdi.kubevirt.io/storage.preallocation.requested:false cdi.kubevirt.io/storage.usePopulator:true pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com volume.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com] [{cdi.kubevirt.io/v1beta1 DataVolume test-vm-dv 9ac161ff-97e8-4727-b99a-b4d25e01ab77 0xc0005c518a 0xc0005c518b}] [kubernetes.io/pvc-protection] [{kube-controller-manager Update v1 2024-09-16 07:27:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-09-16 07:27:42 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status} {virt-cdi-controller Update v1 2024-09-16 07:27:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cdi.kubevirt.io/allowClaimAdoption":{},"f:cdi.kubevirt.io/createdForDataVolume":{},"f:cdi.kubevirt.io/storage.condition.running":{},"f:cdi.kubevirt.io/storage.condition.running.message":{},"f:cdi.kubevirt.io/storage.condition.running.reason":{},"f:cdi.kubevirt.io/storage.contentType":{},"f:cdi.kubevirt.io/storage.deleteAfterCompletion":{},"f:cdi.kubevirt.io/storage.pod.phase":{},"f:cdi.kubevirt.io/storage.pod.restarts":{},"f:cdi.kubevirt.io/storage.populator.progress":{},"f:cdi.kubevirt.io/storage.preallocation.requested":{},"f:cdi.kubevirt.io/storage.usePopulator":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:kubevirt.io/created-by":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"9ac161ff-97e8-4727-b99a-b4d25e01ab77\"}":{}}},"f:spec":{"f:accessModes":{},"f:dataSourceRef":{".":{},"f:apiGroup":{},"f:kind":{},"f:name":{}},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} }]} {[ReadWriteOnce] nil {map[] map[storage:{{5368709120 0} {} 5Gi BinarySI}]} pvc-7b139c74-9670-4101-92e1-1f85a8eac208 0xc000f9e7f0 0xc000f9e800 &TypedLocalObjectReference{APIGroup:*cdi.kubevirt.io,Kind:VolumeImportSource,Name:volume-import-source-9ac161ff-97e8-4727-b99a-b4d25e01ab77,} &TypedLocalObjectReference{APIGroup:*cdi.kubevirt.io,Kind:VolumeImportSource,Name:volume-import-source-9ac161ff-97e8-4727-b99a-b4d25e01ab77,}} {Bound [ReadWriteOnce] map[storage:{{5368709120 0} {} 5Gi BinarySI}] [] map[] }}]} STEP: Creating backup ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 @ 09/16/24 07:28:18.417 2024/09/16 07:28:18 Wait until backup ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 is completed backup phase: Completed 2024/09/16 07:28:38 Verify the PVC and VSC count is matching 2024/09/16 07:28:38 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2024/09/16 07:28:38 Run velero describe on the backup 2024/09/16 07:28:38 [./velero describe backup ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 -n openshift-adp --details --insecure-skip-tls-verify] 2024/09/16 07:28:38 Exec stderr: "" 2024/09/16 07:28:38 Name: ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.30.4 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=30 Phase: Completed Namespaces: Included: test-oadp-185 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-09-16 07:28:18 +0000 UTC Completed: 2024-09-16 07:28:27 +0000 UTC Expiration: 2024-10-16 07:28:18 +0000 UTC Total items to be backed up: 80 Items backed up: 80 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io test-oadp-185/velero-test-vm-dv-2qd4p: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: test-oadp-185/velero-test-vm-dv-2qd4p/2024-09-16T07:28:25Z Items to Update: volumesnapshots.snapshot.storage.k8s.io test-oadp-185/velero-test-vm-dv-2qd4p volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-8732ddd3-ef69-42a1-9bc3-501fbae6ca1a Phase: Completed Created: 2024-09-16 07:28:25 +0000 UTC Started: 2024-09-16 07:28:25 +0000 UTC Updated: 2024-09-16 07:28:26 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com - datavolumes.cdi.kubevirt.io - virtualmachineinstances.kubevirt.io - virtualmachines.kubevirt.io apps/v1/ControllerRevision: - test-oadp-185/revision-start-vm-fa469ad4-c0f6-41db-bcc7-a2cc909ac82b-1 authorization.openshift.io/v1/RoleBinding: - test-oadp-185/system:deployers - test-oadp-185/system:image-builders - test-oadp-185/system:image-pullers cdi.kubevirt.io/v1beta1/DataVolume: - test-oadp-185/test-vm-dv kubevirt.io/v1/VirtualMachine: - test-oadp-185/test-vm kubevirt.io/v1/VirtualMachineInstance: - test-oadp-185/test-vm operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-185/volsync-product.v0.10.0 policy/v1/PodDisruptionBudget: - test-oadp-185/kubevirt-disruption-budget-vblcs rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-185/system:deployers - test-oadp-185/system:image-builders - test-oadp-185/system:image-pullers snapshot.storage.k8s.io/v1/VolumeSnapshot: - test-oadp-185/velero-test-vm-dv-2qd4p snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-8732ddd3-ef69-42a1-9bc3-501fbae6ca1a v1/ConfigMap: - test-oadp-185/kube-root-ca.crt - test-oadp-185/openshift-service-ca.crt v1/Event: - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cc13f2127b - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cc6ae63051 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cc8fff16a0 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cce968b210 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cce968e358 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cd0d7f0b67 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cd0ef522ee - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cd17ec7077 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cd18d1ebb3 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cd472ad890 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cd4f8aca36 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cd506d9b5e - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cd50776944 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8d069a90dc7 - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8d071406ccc - test-oadp-185/importer-prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8d07219acef - test-oadp-185/kubevirt-disruption-budget-vblcs.17f5a8d4582bb725 - test-oadp-185/prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cc118f344c - test-oadp-185/prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cc119984ee - test-oadp-185/prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8cc17f73e2f - test-oadp-185/prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8d453834a90 - test-oadp-185/prime-443d7654-5a35-4321-8001-b7d870a43238.17f5a8d455af0b17 - test-oadp-185/test-vm-dv.17f5a8cc1134fb4f - test-oadp-185/test-vm-dv.17f5a8cc1154902a - test-oadp-185/test-vm-dv.17f5a8cc119d4b11 - test-oadp-185/test-vm-dv.17f5a8cc119d8e34 - test-oadp-185/test-vm-dv.17f5a8cc11bc4ecb - test-oadp-185/test-vm-dv.17f5a8cc6f0863cb - test-oadp-185/test-vm-dv.17f5a8d08fef474b - test-oadp-185/test-vm-dv.17f5a8d410dd86b3 - test-oadp-185/test-vm-dv.17f5a8d454753393 - test-oadp-185/test-vm-dv.17f5a8d4550b2b80 - test-oadp-185/test-vm-dv.17f5a8d4583c27e6 - test-oadp-185/test-vm.17f5a8cc0e684ec3 - test-oadp-185/test-vm.17f5a8d457981de3 - test-oadp-185/test-vm.17f5a8d45832b9d2 - test-oadp-185/test-vm.17f5a8d459e77e91 - test-oadp-185/test-vm.17f5a8d70b2bf82c - test-oadp-185/test-vm.17f5a8d767ebf507 - test-oadp-185/test-vm.17f5a8d76b6e0922 - test-oadp-185/virt-launcher-test-vm-8nzsq.17f5a8d45a5acb9a - test-oadp-185/virt-launcher-test-vm-8nzsq.17f5a8d45d8f7802 - test-oadp-185/virt-launcher-test-vm-8nzsq.17f5a8d6a86179a6 - test-oadp-185/virt-launcher-test-vm-8nzsq.17f5a8d6c85df0e9 - test-oadp-185/virt-launcher-test-vm-8nzsq.17f5a8d6c85e2478 - test-oadp-185/virt-launcher-test-vm-8nzsq.17f5a8d6dfeb832d - test-oadp-185/virt-launcher-test-vm-8nzsq.17f5a8d6e140d3ff - test-oadp-185/virt-launcher-test-vm-8nzsq.17f5a8d6ebf91b22 - test-oadp-185/virt-launcher-test-vm-8nzsq.17f5a8d6ecc7dac0 v1/Namespace: - test-oadp-185 v1/PersistentVolume: - pvc-7b139c74-9670-4101-92e1-1f85a8eac208 v1/PersistentVolumeClaim: - test-oadp-185/test-vm-dv v1/Pod: - test-oadp-185/virt-launcher-test-vm-8nzsq v1/Secret: - test-oadp-185/builder-dockercfg-7vk7s - test-oadp-185/default-dockercfg-dfrt5 - test-oadp-185/deployer-dockercfg-vg5tb v1/ServiceAccount: - test-oadp-185/builder - test-oadp-185/default - test-oadp-185/deployer Backup Volumes: Velero-Native Snapshots: CSI Snapshots: test-oadp-185/test-vm-dv: Snapshot: Operation ID: test-oadp-185/velero-test-vm-dv-2qd4p/2024-09-16T07:28:25Z Snapshot Content Name: snapcontent-8732ddd3-ef69-42a1-9bc3-501fbae6ca1a Storage Snapshot ID: 0001-0011-openshift-storage-0000000000000001-b4aaba65-45e5-40ad-b906-0f2a633072ae Snapshot Size (bytes): 5368709120 CSI Driver: openshift-storage.rbd.csi.ceph.com Pod Volume Backups: HooksAttempted: 2 HooksFailed: 0 STEP: Verify backup ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 has completed successfully @ 09/16/24 07:28:38.86 2024/09/16 07:28:38 Backup for case ocp-kubevirt succeeded STEP: Delete the appplication resources ocp-kubevirt @ 09/16/24 07:28:38.904 STEP: Cleanup Application for case ocp-kubevirt @ 09/16/24 07:28:38.904 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-185] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=8  rescued=0 ignored=0 2024/09/16 07:28:57 2024-09-16 07:28:40,323 p=23190 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:28:40,323 p=23190 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:28:40,562 p=23190 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:28:40,562 p=23190 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:28:40,797 p=23190 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:28:40,797 p=23190 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:28:40,813 p=23190 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:28:40,813 p=23190 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:28:41,111 p=23190 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:28:41,111 p=23190 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:28:41,138 p=23190 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:28:41,139 p=23190 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:28:41,157 p=23190 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:28:41,157 p=23190 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:28:41,168 p=23190 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:28:41,715 p=23190 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:28:41,715 p=23190 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:28:57,513 p=23190 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-185] *** 2024-09-16 07:28:57,513 p=23190 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:28:57,513 p=23190 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:28:57,681 p=23190 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:28:57,682 p=23190 u=1006570000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0 2024/09/16 07:28:57 Creating restore ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 for case ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 STEP: Create restore ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 from backup ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 @ 09/16/24 07:28:57.726 2024/09/16 07:28:57 Wait until restore ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092 is complete restore phase: Finalizing restore phase: Completed STEP: Verify restore ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092has completed successfully @ 09/16/24 07:29:17.748 STEP: Verify Application restore @ 09/16/24 07:29:17.751 STEP: Verify Application deployment for case ocp-kubevirt @ 09/16/24 07:29:17.751 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** ok: [localhost] FAILED - RETRYING: [localhost]: Wait for VM to have AgentConnected status True indicating the guest agent is running (60 retries left). FAILED - RETRYING: [localhost]: Wait for VM to have AgentConnected status True indicating the guest agent is running (59 retries left). FAILED - RETRYING: [localhost]: Wait for VM to have AgentConnected status True indicating the guest agent is running (58 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to have AgentConnected status True indicating the guest agent is running] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=12  changed=3  unreachable=0 failed=0 skipped=7  rescued=0 ignored=0 2024/09/16 07:29:38 2024-09-16 07:29:19,167 p=23394 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:29:19,167 p=23394 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:29:19,403 p=23394 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:29:19,403 p=23394 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:29:19,637 p=23394 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:29:19,637 p=23394 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:29:19,653 p=23394 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:29:19,653 p=23394 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:19,952 p=23394 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:29:19,952 p=23394 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:19,980 p=23394 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:29:19,981 p=23394 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:20,000 p=23394 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:29:20,000 p=23394 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:20,011 p=23394 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:29:20,550 p=23394 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:29:20,550 p=23394 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:21,409 p=23394 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** 2024-09-16 07:29:21,409 p=23394 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:29:21,409 p=23394 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:38,650 p=23394 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to have AgentConnected status True indicating the guest agent is running] *** 2024-09-16 07:29:38,650 p=23394 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:38,663 p=23394 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:29:38,664 p=23394 u=1006570000 n=ansible | localhost : ok=12 changed=3 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0 2024/09/16 07:29:38 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/09/16 07:29:38 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-185] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=8  rescued=0 ignored=0 2024/09/16 07:29:57 2024-09-16 07:29:40,150 p=23649 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:29:40,151 p=23649 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:29:40,388 p=23649 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:29:40,389 p=23649 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:29:40,623 p=23649 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:29:40,623 p=23649 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:29:40,639 p=23649 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:29:40,639 p=23649 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:40,928 p=23649 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:29:40,928 p=23649 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:40,955 p=23649 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:29:40,956 p=23649 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:40,975 p=23649 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:29:40,975 p=23649 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:40,985 p=23649 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:29:41,527 p=23649 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:29:41,527 p=23649 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:29:57,314 p=23649 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-185] *** 2024-09-16 07:29:57,315 p=23649 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:29:57,315 p=23649 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:29:57,481 p=23649 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:29:57,481 p=23649 u=1006570000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0 2024/09/16 07:29:57 Cleaning setup resources for the backup 2024/09/16 07:29:57 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2024/09/16 07:29:57 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd 2024/09/16 07:29:57 Deleting VolumeSnapshotClass 'example-snapclass' • [181.545 seconds] ------------------------------ CSI: Backup/Restore Openshift Virtualization Workloads  [tc-id:OADP-186] [kubevirt] Stopped VM should be restored /alabama/cspi/e2e/kubevirt-plugin/backup_restore_csi.go:51 2024/09/16 07:29:57 Delete all downloadrequest ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092-136e40b6-0a20-4c8f-99f1-9445068400e6 ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092-d3f318ee-131f-4af3-a78c-60dfc08c1207 ocp-kubevirt-0d614d42-73fd-11ef-abd2-0a580a83d092-d497059c-e60a-4b10-92a8-0a0a7a107cc4 STEP: Create DPA CR @ 09/16/24 07:29:57.662 2024/09/16 07:29:57 csi 2024/09/16 07:29:57 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "73176baf-1952-4e49-9d51-9c7b1eef4c2c", "resourceVersion": "66267", "generation": 1, "creationTimestamp": "2024-09-16T07:29:57Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-09-16T07:29:57Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-1" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-3mvpvvyj-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 09/16/24 07:29:57.704 2024/09/16 07:29:57 Waiting for velero pod to be running 2024/09/16 07:29:57 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/09/16 07:29:57 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "73176baf-1952-4e49-9d51-9c7b1eef4c2c", "resourceVersion": "66267", "generation": 1, "creationTimestamp": "2024-09-16T07:29:57Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-09-16T07:29:57Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-1" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-3mvpvvyj-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 09/16/24 07:30:02.733 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/09/16 07:30:02 The 'openshift-storage' namespace exists 2024/09/16 07:30:02 Checking default storage class count 2024/09/16 07:30:02 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2024/09/16 07:30:02 Snapclass 'example-snapclass' doesn't exist, creating 2024/09/16 07:30:02 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2024/09/16 07:30:02 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd STEP: Installing application for case ocp-kubevirt @ 09/16/24 07:30:02.93 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Deploy vm test-vm] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (60 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (59 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (58 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (57 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (56 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (55 retries left). FAILED - RETRYING: [localhost]: Wait for VM to be Running & Ready (54 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Shutdown the VM if required] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM status to become 'Stopped'] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=15  changed=6  unreachable=0 failed=0 skipped=4  rescued=0 ignored=0 2024/09/16 07:30:47 2024-09-16 07:30:04,371 p=23874 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:30:04,372 p=23874 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:30:04,606 p=23874 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:30:04,607 p=23874 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:30:04,840 p=23874 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:30:04,840 p=23874 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:30:04,856 p=23874 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:30:04,856 p=23874 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:05,154 p=23874 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:30:05,154 p=23874 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:05,184 p=23874 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:30:05,184 p=23874 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:05,203 p=23874 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:30:05,204 p=23874 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:05,215 p=23874 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:30:05,746 p=23874 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:30:05,747 p=23874 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:06,499 p=23874 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Create namespace] *** 2024-09-16 07:30:06,500 p=23874 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:30:06,500 p=23874 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:30:07,100 p=23874 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Deploy vm test-vm] *** 2024-09-16 07:30:07,100 p=23874 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:30:46,693 p=23874 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM to be Running & Ready] *** 2024-09-16 07:30:46,693 p=23874 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:47,298 p=23874 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Shutdown the VM if required] *** 2024-09-16 07:30:47,299 p=23874 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:30:47,860 p=23874 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Wait for VM status to become 'Stopped'] *** 2024-09-16 07:30:47,861 p=23874 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:47,933 p=23874 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:30:47,933 p=23874 u=1006570000 n=ansible | localhost : ok=15 changed=6 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0 STEP: Verify Application deployment @ 09/16/24 07:30:47.979 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Verify VM is not in running state] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=3  unreachable=0 failed=0 skipped=8  rescued=0 ignored=0 2024/09/16 07:30:51 2024-09-16 07:30:49,403 p=24221 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:30:49,403 p=24221 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:30:49,632 p=24221 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:30:49,632 p=24221 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:30:49,862 p=24221 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:30:49,863 p=24221 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:30:49,878 p=24221 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:30:49,878 p=24221 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:50,169 p=24221 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:30:50,169 p=24221 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:50,196 p=24221 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:30:50,197 p=24221 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:50,215 p=24221 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:30:50,216 p=24221 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:50,226 p=24221 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:30:50,775 p=24221 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:30:50,775 p=24221 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:51,626 p=24221 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Verify VM is not in running state] *** 2024-09-16 07:30:51,626 p=24221 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:30:51,627 p=24221 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:30:51,683 p=24221 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:30:51,683 p=24221 u=1006570000 n=ansible | localhost : ok=11 changed=3 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0 2024/09/16 07:30:51 {{ } { } [{{ } {test-vm-dv test-oadp-186 d84511b9-009f-4ed5-bfd2-2764baeabc72 67105 0 2024-09-16 07:30:07 +0000 UTC map[app:containerized-data-importer app.kubernetes.io/component:storage app.kubernetes.io/managed-by:cdi-controller app.kubernetes.io/part-of:hyperconverged-cluster app.kubernetes.io/version:4.16.1 kubevirt.io/created-by:0b9b62eb-989e-4955-b0e8-5046a1cd171c] map[cdi.kubevirt.io/allowClaimAdoption:true cdi.kubevirt.io/createdForDataVolume:93d578ed-845d-417c-bbf4-2740d874b3eb cdi.kubevirt.io/storage.condition.running:false cdi.kubevirt.io/storage.condition.running.message:Import Complete cdi.kubevirt.io/storage.condition.running.reason:Completed cdi.kubevirt.io/storage.contentType:kubevirt cdi.kubevirt.io/storage.deleteAfterCompletion:false cdi.kubevirt.io/storage.pod.phase:Succeeded cdi.kubevirt.io/storage.pod.restarts:0 cdi.kubevirt.io/storage.populator.progress:100.0% cdi.kubevirt.io/storage.preallocation.requested:false cdi.kubevirt.io/storage.usePopulator:true pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com volume.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com] [{cdi.kubevirt.io/v1beta1 DataVolume test-vm-dv 93d578ed-845d-417c-bbf4-2740d874b3eb 0xc00090922a 0xc00090922b}] [kubernetes.io/pvc-protection] [{kube-controller-manager Update v1 2024-09-16 07:30:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-09-16 07:30:32 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status} {virt-cdi-controller Update v1 2024-09-16 07:30:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cdi.kubevirt.io/allowClaimAdoption":{},"f:cdi.kubevirt.io/createdForDataVolume":{},"f:cdi.kubevirt.io/storage.condition.running":{},"f:cdi.kubevirt.io/storage.condition.running.message":{},"f:cdi.kubevirt.io/storage.condition.running.reason":{},"f:cdi.kubevirt.io/storage.contentType":{},"f:cdi.kubevirt.io/storage.deleteAfterCompletion":{},"f:cdi.kubevirt.io/storage.pod.phase":{},"f:cdi.kubevirt.io/storage.pod.restarts":{},"f:cdi.kubevirt.io/storage.populator.progress":{},"f:cdi.kubevirt.io/storage.preallocation.requested":{},"f:cdi.kubevirt.io/storage.usePopulator":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:kubevirt.io/created-by":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"93d578ed-845d-417c-bbf4-2740d874b3eb\"}":{}}},"f:spec":{"f:accessModes":{},"f:dataSourceRef":{".":{},"f:apiGroup":{},"f:kind":{},"f:name":{}},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} }]} {[ReadWriteOnce] nil {map[] map[storage:{{5368709120 0} {} 5Gi BinarySI}]} pvc-5083e4fe-0f2c-4724-a1ac-55936ac10e9d 0xc00072aed0 0xc00072aee0 &TypedLocalObjectReference{APIGroup:*cdi.kubevirt.io,Kind:VolumeImportSource,Name:volume-import-source-93d578ed-845d-417c-bbf4-2740d874b3eb,} &TypedLocalObjectReference{APIGroup:*cdi.kubevirt.io,Kind:VolumeImportSource,Name:volume-import-source-93d578ed-845d-417c-bbf4-2740d874b3eb,}} {Bound [ReadWriteOnce] map[storage:{{5368709120 0} {} 5Gi BinarySI}] [] map[] }}]} STEP: Creating backup ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 @ 09/16/24 07:30:51.732 2024/09/16 07:30:51 Wait until backup ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 is completed backup phase: Completed 2024/09/16 07:31:11 Verify the PVC and VSC count is matching 2024/09/16 07:31:11 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2024/09/16 07:31:11 Run velero describe on the backup 2024/09/16 07:31:11 [./velero describe backup ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 -n openshift-adp --details --insecure-skip-tls-verify] 2024/09/16 07:31:12 Exec stderr: "" 2024/09/16 07:31:12 Name: ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.30.4 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=30 Phase: Completed Namespaces: Included: test-oadp-186 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-09-16 07:30:51 +0000 UTC Completed: 2024-09-16 07:30:59 +0000 UTC Expiration: 2024-10-16 07:30:51 +0000 UTC Total items to be backed up: 82 Items backed up: 82 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io test-oadp-186/velero-test-vm-dv-lzvmt: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: test-oadp-186/velero-test-vm-dv-lzvmt/2024-09-16T07:30:58Z Items to Update: volumesnapshots.snapshot.storage.k8s.io test-oadp-186/velero-test-vm-dv-lzvmt volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-47d614a6-ed10-4538-816a-c86210f22204 Phase: Completed Created: 2024-09-16 07:30:58 +0000 UTC Started: 2024-09-16 07:30:58 +0000 UTC Updated: 2024-09-16 07:30:58 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com - datavolumes.cdi.kubevirt.io - virtualmachines.kubevirt.io authorization.openshift.io/v1/RoleBinding: - test-oadp-186/system:deployers - test-oadp-186/system:image-builders - test-oadp-186/system:image-pullers cdi.kubevirt.io/v1beta1/DataVolume: - test-oadp-186/test-vm-dv kubevirt.io/v1/VirtualMachine: - test-oadp-186/test-vm operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-186/volsync-product.v0.10.0 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-186/system:deployers - test-oadp-186/system:image-builders - test-oadp-186/system:image-pullers snapshot.storage.k8s.io/v1/VolumeSnapshot: - test-oadp-186/velero-test-vm-dv-lzvmt snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-47d614a6-ed10-4538-816a-c86210f22204 v1/ConfigMap: - test-oadp-186/kube-root-ca.crt - test-oadp-186/openshift-service-ca.crt v1/Event: - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f61ea63fc6 - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f65b077d28 - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f68093bfe1 - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f79bf1635b - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f79bf1bcfd - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7b0d7509d - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7b238c81c - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7b8e9ff47 - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7b9cc3036 - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7cfd015d8 - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7d68f2edc - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7d77248e4 - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7d77d9f82 - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7dfe19ad0 - test-oadp-186/importer-prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f7e0c5973d - test-oadp-186/kubevirt-disruption-budget-csjbg.17f5a8fc13e4c8b5 - test-oadp-186/prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f61cc18c3d - test-oadp-186/prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f61cdc469f - test-oadp-186/prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8f6221c1f04 - test-oadp-186/prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8fc10228157 - test-oadp-186/prime-d84511b9-009f-4ed5-bfd2-2764baeabc72.17f5a8fc128eecf0 - test-oadp-186/test-vm-dv.17f5a8f61c41625b - test-oadp-186/test-vm-dv.17f5a8f61c51f43a - test-oadp-186/test-vm-dv.17f5a8f61cc2ce87 - test-oadp-186/test-vm-dv.17f5a8f61cc3525f - test-oadp-186/test-vm-dv.17f5a8f61cdc6e3d - test-oadp-186/test-vm-dv.17f5a8f61db9a524 - test-oadp-186/test-vm-dv.17f5a8f65e9a2e90 - test-oadp-186/test-vm-dv.17f5a8f8118ae8c0 - test-oadp-186/test-vm-dv.17f5a8fb902ff504 - test-oadp-186/test-vm-dv.17f5a8fc1103dca0 - test-oadp-186/test-vm-dv.17f5a8fc11b19cc7 - test-oadp-186/test-vm-dv.17f5a8fc132546a7 - test-oadp-186/test-vm.17f5a8f6195c92cd - test-oadp-186/test-vm.17f5a8fc137ac8cf - test-oadp-186/test-vm.17f5a8fc13e9fdaf - test-oadp-186/test-vm.17f5a8fc155a36ec - test-oadp-186/test-vm.17f5a8ff00911f42 - test-oadp-186/test-vm.17f5a8ff01c797f2 - test-oadp-186/test-vm.17f5a8ff04ddb08a - test-oadp-186/test-vm.17f5a8ff7357dafd - test-oadp-186/test-vm.17f5a8ff7363535e - test-oadp-186/test-vm.17f5a8ff7375d624 - test-oadp-186/test-vm.17f5a8ff80783dde - test-oadp-186/test-vm.17f5a8ff81a4984f - test-oadp-186/test-vm.17f5a8ff83b96945 - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8fc15bea4a1 - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8fc194774fb - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8fc963c3187 - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8fe4a9e2f0f - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8fe4a9e69f7 - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8fe6741cabb - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8fe68786eb4 - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8fe72f2293e - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8fe73d4ae7a - test-oadp-186/virt-launcher-test-vm-8n24k.17f5a8ff73c38ffe v1/Namespace: - test-oadp-186 v1/PersistentVolume: - pvc-5083e4fe-0f2c-4724-a1ac-55936ac10e9d v1/PersistentVolumeClaim: - test-oadp-186/test-vm-dv v1/Secret: - test-oadp-186/builder-dockercfg-r78nf - test-oadp-186/default-dockercfg-pbjp7 - test-oadp-186/deployer-dockercfg-489ph v1/ServiceAccount: - test-oadp-186/builder - test-oadp-186/default - test-oadp-186/deployer Backup Volumes: Velero-Native Snapshots: CSI Snapshots: test-oadp-186/test-vm-dv: Snapshot: Operation ID: test-oadp-186/velero-test-vm-dv-lzvmt/2024-09-16T07:30:58Z Snapshot Content Name: snapcontent-47d614a6-ed10-4538-816a-c86210f22204 Storage Snapshot ID: 0001-0011-openshift-storage-0000000000000001-fc7bd020-4614-4255-bc83-37559ab25ff2 Snapshot Size (bytes): 5368709120 CSI Driver: openshift-storage.rbd.csi.ceph.com Pod Volume Backups: HooksAttempted: 0 HooksFailed: 0 STEP: Verify backup ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 has completed successfully @ 09/16/24 07:31:12.152 2024/09/16 07:31:12 Backup for case ocp-kubevirt succeeded STEP: Delete the appplication resources ocp-kubevirt @ 09/16/24 07:31:12.193 STEP: Cleanup Application for case ocp-kubevirt @ 09/16/24 07:31:12.194 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-186] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=8  rescued=0 ignored=0 2024/09/16 07:31:30 2024-09-16 07:31:13,610 p=24425 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:31:13,610 p=24425 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:13,848 p=24425 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:31:13,849 p=24425 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:14,086 p=24425 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:31:14,086 p=24425 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:14,101 p=24425 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:31:14,102 p=24425 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:14,394 p=24425 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:31:14,394 p=24425 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:14,421 p=24425 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:31:14,422 p=24425 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:14,440 p=24425 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:31:14,440 p=24425 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:14,451 p=24425 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:31:14,995 p=24425 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:31:14,995 p=24425 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:30,759 p=24425 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-186] *** 2024-09-16 07:31:30,759 p=24425 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:31:30,759 p=24425 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:30,932 p=24425 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:31:30,932 p=24425 u=1006570000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0 2024/09/16 07:31:30 Creating restore ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 for case ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 STEP: Create restore ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 from backup ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 @ 09/16/24 07:31:30.979 2024/09/16 07:31:30 Wait until restore ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092 is complete restore phase: Completed STEP: Verify restore ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092has completed successfully @ 09/16/24 07:31:41.019 STEP: Verify Application restore @ 09/16/24 07:31:41.023 STEP: Verify Application deployment for case ocp-kubevirt @ 09/16/24 07:31:41.023 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Verify VM is not in running state] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=3  unreachable=0 failed=0 skipped=8  rescued=0 ignored=0 2024/09/16 07:31:44 2024-09-16 07:31:42,458 p=24629 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:31:42,459 p=24629 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:42,697 p=24629 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:31:42,697 p=24629 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:42,933 p=24629 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:31:42,933 p=24629 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:42,948 p=24629 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:31:42,948 p=24629 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:43,243 p=24629 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:31:43,243 p=24629 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:43,271 p=24629 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:31:43,272 p=24629 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:43,292 p=24629 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:31:43,292 p=24629 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:43,303 p=24629 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:31:43,852 p=24629 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:31:43,852 p=24629 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:44,706 p=24629 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Verify VM is not in running state] *** 2024-09-16 07:31:44,707 p=24629 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:31:44,707 p=24629 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:44,762 p=24629 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:31:44,762 p=24629 u=1006570000 n=ansible | localhost : ok=11 changed=3 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0 2024/09/16 07:31:44 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/09/16 07:31:44 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-186] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=8  rescued=0 ignored=0 2024/09/16 07:32:03 2024-09-16 07:31:46,245 p=24834 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:31:46,245 p=24834 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:46,483 p=24834 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:31:46,483 p=24834 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:46,719 p=24834 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:31:46,719 p=24834 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:31:46,734 p=24834 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:31:46,735 p=24834 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:47,032 p=24834 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:31:47,033 p=24834 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:47,061 p=24834 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:31:47,061 p=24834 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:47,081 p=24834 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:31:47,081 p=24834 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:31:47,091 p=24834 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:31:47,643 p=24834 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:31:47,643 p=24834 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:03,463 p=24834 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-kubevirt : Remove namespace test-oadp-186] *** 2024-09-16 07:32:03,464 p=24834 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:32:03,464 p=24834 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:32:03,630 p=24834 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:32:03,631 p=24834 u=1006570000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0 2024/09/16 07:32:03 Cleaning setup resources for the backup 2024/09/16 07:32:03 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2024/09/16 07:32:03 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd 2024/09/16 07:32:03 Deleting VolumeSnapshotClass 'example-snapclass' • [126.153 seconds] ------------------------------ CSI: Backup/Restore Openshift Virtualization Workloads  [tc-id:OADP-187] [kubevirt] Backup-restore data volume /alabama/cspi/e2e/kubevirt-plugin/backup_restore_csi.go:68 2024/09/16 07:32:03 Delete all downloadrequest ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092-2aa0b0b6-57f5-454f-a2ef-292c0b88e310 ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092-4914c548-5a27-4c06-913f-777b90864e55 ocp-kubevirt-799727ff-73fd-11ef-abd2-0a580a83d092-51312e84-fb01-415b-8ef4-6ebc71cc769e STEP: Create DPA CR @ 09/16/24 07:32:03.824 2024/09/16 07:32:03 csi 2024/09/16 07:32:03 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "dfae46d6-c4d0-4ec9-ad96-f73bc2813d27", "resourceVersion": "68757", "generation": 1, "creationTimestamp": "2024-09-16T07:32:03Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-09-16T07:32:03Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-1" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-3mvpvvyj-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 09/16/24 07:32:03.859 2024/09/16 07:32:03 Waiting for velero pod to be running 2024/09/16 07:32:03 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/09/16 07:32:03 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "dfae46d6-c4d0-4ec9-ad96-f73bc2813d27", "resourceVersion": "68757", "generation": 1, "creationTimestamp": "2024-09-16T07:32:03Z", "managedFields": [ { "manager": "kubevirt-plugin.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-09-16T07:32:03Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-1" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-3mvpvvyj-interopoadp", "prefix": "kubevirt" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 09/16/24 07:32:08.873 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/09/16 07:32:08 The 'openshift-storage' namespace exists 2024/09/16 07:32:08 Checking default storage class count 2024/09/16 07:32:08 Using the CSI driver: openshift-storage.rbd.csi.ceph.com 2024/09/16 07:32:08 Snapclass 'example-snapclass' doesn't exist, creating 2024/09/16 07:32:09 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2024/09/16 07:32:09 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd STEP: Installing application for case ocp-datavolume @ 09/16/24 07:32:09.014 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Deploy DataVolume test-dv] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=12  changed=5  unreachable=0 failed=0 skipped=2  rescued=0 ignored=0 2024/09/16 07:32:13 2024-09-16 07:32:10,408 p=25059 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:32:10,408 p=25059 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:32:10,644 p=25059 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:32:10,644 p=25059 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:32:10,878 p=25059 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:32:10,878 p=25059 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:32:10,894 p=25059 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:32:10,895 p=25059 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:11,180 p=25059 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:32:11,181 p=25059 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:11,207 p=25059 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:32:11,207 p=25059 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:11,229 p=25059 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:32:11,229 p=25059 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:11,239 p=25059 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:32:11,767 p=25059 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:32:11,767 p=25059 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:12,521 p=25059 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Create namespace] *** 2024-09-16 07:32:12,522 p=25059 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:32:12,522 p=25059 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:32:13,154 p=25059 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Deploy DataVolume test-dv] *** 2024-09-16 07:32:13,154 p=25059 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:32:13,185 p=25059 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:32:13,185 p=25059 u=1006570000 n=ansible | localhost : ok=12 changed=5 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 STEP: Verify Application deployment @ 09/16/24 07:32:13.227 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. FAILED - RETRYING: [localhost]: Wait for DataVolume to be in Succeeded phase (30 retries left). FAILED - RETRYING: [localhost]: Wait for DataVolume to be in Succeeded phase (29 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Wait for DataVolume to be in Succeeded phase] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=3  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/09/16 07:32:37 2024-09-16 07:32:14,608 p=25269 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:32:14,608 p=25269 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:32:14,843 p=25269 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:32:14,843 p=25269 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:32:15,073 p=25269 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:32:15,074 p=25269 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:32:15,088 p=25269 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:32:15,089 p=25269 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:15,380 p=25269 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:32:15,380 p=25269 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:15,407 p=25269 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:32:15,408 p=25269 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:15,428 p=25269 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:32:15,428 p=25269 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:15,438 p=25269 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:32:15,981 p=25269 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:32:15,981 p=25269 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:37,892 p=25269 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Wait for DataVolume to be in Succeeded phase] *** 2024-09-16 07:32:37,893 p=25269 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:32:37,893 p=25269 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:32:37,906 p=25269 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:32:37,906 p=25269 u=1006570000 n=ansible | localhost : ok=11 changed=3 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/09/16 07:32:37 {{ } { } [{{ } {test-dv test-oadp-187 8418775e-e31d-4edc-abf6-1a1fab9ce442 69315 0 2024-09-16 07:32:13 +0000 UTC map[alerts.k8s.io/KubePersistentVolumeFillingUp:disabled app:containerized-data-importer app.kubernetes.io/component:storage app.kubernetes.io/managed-by:cdi-controller app.kubernetes.io/part-of:hyperconverged-cluster app.kubernetes.io/version:4.16.1] map[cdi.kubevirt.io/createdForDataVolume:4aa7d7b7-5215-42d4-9a8d-9233373e4777 cdi.kubevirt.io/storage.bind.immediate.requested:true cdi.kubevirt.io/storage.condition.running:false cdi.kubevirt.io/storage.condition.running.message:Import Complete cdi.kubevirt.io/storage.condition.running.reason:Completed cdi.kubevirt.io/storage.contentType:kubevirt cdi.kubevirt.io/storage.deleteAfterCompletion:false cdi.kubevirt.io/storage.pod.phase:Succeeded cdi.kubevirt.io/storage.pod.restarts:0 cdi.kubevirt.io/storage.populator.progress:100.0% cdi.kubevirt.io/storage.preallocation.requested:false cdi.kubevirt.io/storage.usePopulator:true pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com volume.kubernetes.io/storage-provisioner:openshift-storage.rbd.csi.ceph.com] [{cdi.kubevirt.io/v1beta1 DataVolume test-dv 4aa7d7b7-5215-42d4-9a8d-9233373e4777 0xc000e6d527 0xc000e6d528}] [kubernetes.io/pvc-protection] [{kube-controller-manager Update v1 2024-09-16 07:32:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-09-16 07:32:28 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status} {virt-cdi-controller Update v1 2024-09-16 07:32:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cdi.kubevirt.io/createdForDataVolume":{},"f:cdi.kubevirt.io/storage.bind.immediate.requested":{},"f:cdi.kubevirt.io/storage.condition.running":{},"f:cdi.kubevirt.io/storage.condition.running.message":{},"f:cdi.kubevirt.io/storage.condition.running.reason":{},"f:cdi.kubevirt.io/storage.contentType":{},"f:cdi.kubevirt.io/storage.deleteAfterCompletion":{},"f:cdi.kubevirt.io/storage.pod.phase":{},"f:cdi.kubevirt.io/storage.pod.restarts":{},"f:cdi.kubevirt.io/storage.populator.progress":{},"f:cdi.kubevirt.io/storage.preallocation.requested":{},"f:cdi.kubevirt.io/storage.usePopulator":{}},"f:labels":{".":{},"f:alerts.k8s.io/KubePersistentVolumeFillingUp":{},"f:app":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"4aa7d7b7-5215-42d4-9a8d-9233373e4777\"}":{}}},"f:spec":{"f:accessModes":{},"f:dataSourceRef":{".":{},"f:apiGroup":{},"f:kind":{},"f:name":{}},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} }]} {[ReadWriteOnce] nil {map[] map[storage:{{104857600 0} {} 100Mi BinarySI}]} pvc-c3b123ed-bf39-4275-b2ce-91774be1a475 0xc00072b800 0xc00072b810 &TypedLocalObjectReference{APIGroup:*cdi.kubevirt.io,Kind:VolumeImportSource,Name:volume-import-source-4aa7d7b7-5215-42d4-9a8d-9233373e4777,} &TypedLocalObjectReference{APIGroup:*cdi.kubevirt.io,Kind:VolumeImportSource,Name:volume-import-source-4aa7d7b7-5215-42d4-9a8d-9233373e4777,}} {Bound [ReadWriteOnce] map[storage:{{104857600 0} {} 100Mi BinarySI}] [] map[] }}]} STEP: Creating backup ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 @ 09/16/24 07:32:37.957 2024/09/16 07:32:37 Wait until backup ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 is completed backup phase: Completed 2024/09/16 07:32:57 Verify the PVC and VSC count is matching 2024/09/16 07:32:57 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2024/09/16 07:32:58 Run velero describe on the backup 2024/09/16 07:32:58 [./velero describe backup ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 -n openshift-adp --details --insecure-skip-tls-verify] 2024/09/16 07:32:58 Exec stderr: "" 2024/09/16 07:32:58 Name: ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.30.4 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=30 Phase: Completed Namespaces: Included: test-oadp-187 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-09-16 07:32:38 +0000 UTC Completed: 2024-09-16 07:32:45 +0000 UTC Expiration: 2024-10-16 07:32:37 +0000 UTC Total items to be backed up: 48 Items backed up: 48 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io test-oadp-187/velero-test-dv-2d98k: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: test-oadp-187/velero-test-dv-2d98k/2024-09-16T07:32:44Z Items to Update: volumesnapshots.snapshot.storage.k8s.io test-oadp-187/velero-test-dv-2d98k volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-5101b904-7ba3-4f17-8644-a84d314fe836 Phase: Completed Created: 2024-09-16 07:32:44 +0000 UTC Started: 2024-09-16 07:32:44 +0000 UTC Updated: 2024-09-16 07:32:44 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com - datavolumes.cdi.kubevirt.io authorization.openshift.io/v1/RoleBinding: - test-oadp-187/system:deployers - test-oadp-187/system:image-builders - test-oadp-187/system:image-pullers cdi.kubevirt.io/v1beta1/DataVolume: - test-oadp-187/test-dv operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-187/volsync-product.v0.10.0 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-187/system:deployers - test-oadp-187/system:image-builders - test-oadp-187/system:image-pullers snapshot.storage.k8s.io/v1/VolumeSnapshot: - test-oadp-187/velero-test-dv-2d98k snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-5101b904-7ba3-4f17-8644-a84d314fe836 v1/ConfigMap: - test-oadp-187/kube-root-ca.crt - test-oadp-187/openshift-service-ca.crt v1/Event: - test-oadp-187/importer-prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a913749ac9cc - test-oadp-187/importer-prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a913b5c89565 - test-oadp-187/importer-prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a913d64480f5 - test-oadp-187/importer-prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a9164d2c1968 - test-oadp-187/importer-prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a9164e5de326 - test-oadp-187/importer-prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a9165568c4ef - test-oadp-187/importer-prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a9165654b217 - test-oadp-187/prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a9137131ac08 - test-oadp-187/prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a913714b00d9 - test-oadp-187/prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a9137564a905 - test-oadp-187/prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a916fa31d146 - test-oadp-187/prime-8418775e-e31d-4edc-abf6-1a1fab9ce442.17f5a916fd062f07 - test-oadp-187/test-dv.17f5a9137080bf74 - test-oadp-187/test-dv.17f5a91370b2692f - test-oadp-187/test-dv.17f5a91370f59d22 - test-oadp-187/test-dv.17f5a91370f5f237 - test-oadp-187/test-dv.17f5a91370f6c746 - test-oadp-187/test-dv.17f5a91371323def - test-oadp-187/test-dv.17f5a913bdbe5f03 - test-oadp-187/test-dv.17f5a9167ae561ea - test-oadp-187/test-dv.17f5a916b628795f - test-oadp-187/test-dv.17f5a916fbffa6cb - test-oadp-187/test-dv.17f5a916fc2a8885 - test-oadp-187/test-dv.17f5a916fceae56c v1/Namespace: - test-oadp-187 v1/PersistentVolume: - pvc-c3b123ed-bf39-4275-b2ce-91774be1a475 v1/PersistentVolumeClaim: - test-oadp-187/test-dv v1/Secret: - test-oadp-187/builder-dockercfg-2jwv9 - test-oadp-187/default-dockercfg-4qktp - test-oadp-187/deployer-dockercfg-mphbm v1/ServiceAccount: - test-oadp-187/builder - test-oadp-187/default - test-oadp-187/deployer Backup Volumes: Velero-Native Snapshots: CSI Snapshots: test-oadp-187/test-dv: Snapshot: Operation ID: test-oadp-187/velero-test-dv-2d98k/2024-09-16T07:32:44Z Snapshot Content Name: snapcontent-5101b904-7ba3-4f17-8644-a84d314fe836 Storage Snapshot ID: 0001-0011-openshift-storage-0000000000000001-a8423da1-d627-438d-92b6-d9202fa9bf71 Snapshot Size (bytes): 104857600 CSI Driver: openshift-storage.rbd.csi.ceph.com Pod Volume Backups: HooksAttempted: 0 HooksFailed: 0 STEP: Verify backup ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 has completed successfully @ 09/16/24 07:32:58.437 2024/09/16 07:32:58 Backup for case ocp-datavolume succeeded STEP: Delete the appplication resources ocp-datavolume @ 09/16/24 07:32:58.478 STEP: Cleanup Application for case ocp-datavolume @ 09/16/24 07:32:58.478 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Remove namespace test-oadp-187] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/09/16 07:33:17 2024-09-16 07:32:59,828 p=25494 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:32:59,829 p=25494 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:00,057 p=25494 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:33:00,057 p=25494 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:00,289 p=25494 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:33:00,289 p=25494 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:00,304 p=25494 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:33:00,304 p=25494 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:00,586 p=25494 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:33:00,586 p=25494 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:00,612 p=25494 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:33:00,613 p=25494 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:00,631 p=25494 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:33:00,631 p=25494 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:00,641 p=25494 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:33:01,154 p=25494 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:33:01,154 p=25494 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:16,911 p=25494 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Remove namespace test-oadp-187] *** 2024-09-16 07:33:16,911 p=25494 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:33:16,911 p=25494 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:16,976 p=25494 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:33:16,976 p=25494 u=1006570000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/09/16 07:33:17 Creating restore ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 for case ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 STEP: Create restore ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 from backup ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 @ 09/16/24 07:33:17.016 2024/09/16 07:33:17 Wait until restore ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092 is complete restore phase: Finalizing restore phase: Completed STEP: Verify restore ocp-datavolume-c4c88ce0-73fd-11ef-abd2-0a580a83d092has completed successfully @ 09/16/24 07:33:37.036 STEP: Verify Application restore @ 09/16/24 07:33:37.04 STEP: Verify Application deployment for case ocp-datavolume @ 09/16/24 07:33:37.04 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Wait for DataVolume to be in Succeeded phase] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=3  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/09/16 07:33:40 2024-09-16 07:33:38,430 p=25694 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:33:38,430 p=25694 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:38,661 p=25694 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:33:38,661 p=25694 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:38,894 p=25694 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:33:38,894 p=25694 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:38,910 p=25694 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:33:38,910 p=25694 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:39,200 p=25694 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:33:39,200 p=25694 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:39,227 p=25694 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:33:39,227 p=25694 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:39,249 p=25694 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:33:39,249 p=25694 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:39,260 p=25694 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:33:39,791 p=25694 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:33:39,791 p=25694 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:40,591 p=25694 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Wait for DataVolume to be in Succeeded phase] *** 2024-09-16 07:33:40,591 p=25694 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:33:40,592 p=25694 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:40,604 p=25694 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:33:40,604 p=25694 u=1006570000 n=ansible | localhost : ok=11 changed=3 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/09/16 07:33:40 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/09/16 07:33:40 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Remove namespace test-oadp-187] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/09/16 07:33:59 2024-09-16 07:33:42,043 p=25894 u=1006570000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-09-16 07:33:42,044 p=25894 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:42,282 p=25894 u=1006570000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-09-16 07:33:42,282 p=25894 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:42,512 p=25894 u=1006570000 n=ansible | TASK [Get current admin token] ************************************************* 2024-09-16 07:33:42,512 p=25894 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:42,527 p=25894 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:33:42,527 p=25894 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:42,813 p=25894 u=1006570000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-09-16 07:33:42,813 p=25894 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:42,840 p=25894 u=1006570000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-09-16 07:33:42,840 p=25894 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:42,858 p=25894 u=1006570000 n=ansible | TASK [set_fact] **************************************************************** 2024-09-16 07:33:42,858 p=25894 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:42,868 p=25894 u=1006570000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-09-16 07:33:43,406 p=25894 u=1006570000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-09-16 07:33:43,406 p=25894 u=1006570000 n=ansible | ok: [localhost] 2024-09-16 07:33:59,170 p=25894 u=1006570000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-datavolume : Remove namespace test-oadp-187] *** 2024-09-16 07:33:59,171 p=25894 u=1006570000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-09-16 07:33:59,171 p=25894 u=1006570000 n=ansible | changed: [localhost] 2024-09-16 07:33:59,237 p=25894 u=1006570000 n=ansible | PLAY RECAP ********************************************************************* 2024-09-16 07:33:59,237 p=25894 u=1006570000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/09/16 07:33:59 Cleaning setup resources for the backup 2024/09/16 07:33:59 Setting new default StorageClass 'ocs-storagecluster-ceph-rbd' 2024/09/16 07:33:59 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ocs-storagecluster-ceph-rbd matches the new StorageClass: ocs-storagecluster-ceph-rbd 2024/09/16 07:33:59 Deleting VolumeSnapshotClass 'example-snapclass' • [115.622 seconds] ------------------------------ [AfterSuite]  /alabama/cspi/e2e/kubevirt-plugin/kubevirt_suite_test.go:105 [AfterSuite] PASSED [0.007 seconds] ------------------------------ [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report autogenerated by Ginkgo [ReportAfterSuite] PASSED [0.002 seconds] ------------------------------ Summarizing 1 Failure: [FAIL] Native CSI Data Mover: Backup/Restore Openshift Virtualization Workloads  [It] [tc-id:OADP-401] [kubevirt] Started VM should over ceph filesytem mode /alabama/cspi/test_common/backup_restore_app_case.go:46 Ran 4 of 5 Specs in 822.794 seconds FAIL! -- 3 Passed | 1 Failed | 0 Pending | 1 Skipped --- FAIL: TestOADPE2E (822.80s) FAIL Ginkgo ran 1 suite in 14m28.797099556s Test Suite Failed