Extract /home/jenkins/oadp-e2e-qe.tar.gz to /alabama/cspi Extract /home/jenkins/oadp-apps-deployer.tar.gz to /alabama/oadpApps Extract /home/jenkins/mtc-python-client.tar.gz to /alabama/pyclient Create and populate /tmp/test-settings... Login as Kubeadmin to the test cluster at https://api.ci-op-1nitxr4v-2f653.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:6443... WARNING: Using insecure TLS client config. Setting this option is not supported! Login successful. You have access to 71 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". Create virtual environment and install required packages... Collecting ansible_runner Downloading ansible_runner-2.4.0-py3-none-any.whl (79 kB) Collecting packaging Downloading packaging-24.1-py3-none-any.whl (53 kB) Collecting pyyaml Downloading PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB) Collecting python-daemon Downloading python_daemon-3.0.1-py3-none-any.whl (31 kB) Collecting pexpect>=4.5 Downloading pexpect-4.9.0-py2.py3-none-any.whl (63 kB) Collecting ptyprocess>=0.5 Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB) Collecting lockfile>=0.10 Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB) Collecting docutils Downloading docutils-0.21.2-py3-none-any.whl (587 kB) Collecting setuptools>=62.4.0 Downloading setuptools-72.2.0-py3-none-any.whl (2.3 MB) Installing collected packages: setuptools, ptyprocess, lockfile, docutils, pyyaml, python-daemon, pexpect, packaging, ansible-runner Attempting uninstall: setuptools Found existing installation: setuptools 57.4.0 Uninstalling setuptools-57.4.0: Successfully uninstalled setuptools-57.4.0 Successfully installed ansible-runner-2.4.0 docutils-0.21.2 lockfile-0.12.2 packaging-24.1 pexpect-4.9.0 ptyprocess-0.7.0 python-daemon-3.0.1 pyyaml-6.0.2 setuptools-72.2.0 WARNING: You are using pip version 21.2.3; however, version 24.2 is available. You should consider upgrading via the '/alabama/venv/bin/python3 -m pip install --upgrade pip' command. Processing /alabama/oadpApps DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing wheel metadata: started Preparing wheel metadata: finished with status 'done' Building wheels for collected packages: ocpdeployer Building wheel for ocpdeployer (PEP 517): started Building wheel for ocpdeployer (PEP 517): finished with status 'done' Created wheel for ocpdeployer: filename=ocpdeployer-0.0.1-py2.py3-none-any.whl size=224959 sha256=78b61ece5f1cc70e1e704371638fa6acfa106149dc0cfc28d4e0ee01e5cc3a57 Stored in directory: /tmp/pip-ephem-wheel-cache-pdt7h30_/wheels/ea/a6/87/9d98fc51cc395d30bd2147a7e53e3f5a2e80044d9dd9e64977 Successfully built ocpdeployer Installing collected packages: ocpdeployer WARNING: Value for scheme.platlib does not match. Please report this to distutils: /tmp/pip-target-r0w6cmz2/lib64/python sysconfig: /tmp/pip-target-r0w6cmz2/lib/python WARNING: Additional context: user = False home = '/tmp/pip-target-r0w6cmz2' root = None prefix = None Successfully installed ocpdeployer-0.0.1 WARNING: You are using pip version 21.2.3; however, version 24.2 is available. You should consider upgrading via the '/alabama/venv/bin/python3 -m pip install --upgrade pip' command. Processing /alabama/pyclient DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. Collecting suds-py3 Downloading suds_py3-1.4.5.0-py3-none-any.whl (298 kB) Collecting requests Downloading requests-2.32.3-py3-none-any.whl (64 kB) Collecting jinja2 Downloading jinja2-3.1.4-py3-none-any.whl (133 kB) Collecting kubernetes==11.0.0 Downloading kubernetes-11.0.0-py3-none-any.whl (1.5 MB) Collecting openshift==0.11.2 Downloading openshift-0.11.2.tar.gz (19 kB) Collecting urllib3>=1.24.2 Downloading urllib3-2.2.2-py3-none-any.whl (121 kB) Collecting six>=1.9.0 Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting python-dateutil>=2.5.3 Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) Collecting requests-oauthlib Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl (24 kB) Collecting certifi>=14.05.14 Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) Requirement already satisfied: setuptools>=21.0.0 in /alabama/venv/lib/python3.10/site-packages (from kubernetes==11.0.0->mtc==0.0.1) (72.2.0) Requirement already satisfied: pyyaml>=3.12 in /alabama/venv/lib/python3.10/site-packages (from kubernetes==11.0.0->mtc==0.0.1) (6.0.2) Collecting google-auth>=1.0.1 Downloading google_auth-2.34.0-py2.py3-none-any.whl (200 kB) Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 Downloading websocket_client-1.8.0-py3-none-any.whl (58 kB) Collecting python-string-utils Downloading python_string_utils-1.0.0-py3-none-any.whl (26 kB) Collecting ruamel.yaml>=0.15 Downloading ruamel.yaml-0.18.6-py3-none-any.whl (117 kB) Collecting pyasn1-modules>=0.2.1 Downloading pyasn1_modules-0.4.0-py3-none-any.whl (181 kB) Collecting cachetools<6.0,>=2.0.0 Downloading cachetools-5.5.0-py3-none-any.whl (9.5 kB) Collecting rsa<5,>=3.1.4 Downloading rsa-4.9-py3-none-any.whl (34 kB) Collecting pyasn1<0.7.0,>=0.4.6 Downloading pyasn1-0.6.0-py2.py3-none-any.whl (85 kB) Collecting ruamel.yaml.clib>=0.2.7 Downloading ruamel.yaml.clib-0.2.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (526 kB) Collecting MarkupSafe>=2.0 Downloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB) Collecting idna<4,>=2.5 Downloading idna-3.7-py3-none-any.whl (66 kB) Collecting charset-normalizer<4,>=2 Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB) Collecting oauthlib>=3.0.0 Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB) Using legacy 'setup.py install' for mtc, since package 'wheel' is not installed. Using legacy 'setup.py install' for openshift, since package 'wheel' is not installed. Installing collected packages: urllib3, pyasn1, idna, charset-normalizer, certifi, six, rsa, requests, pyasn1-modules, oauthlib, cachetools, websocket-client, ruamel.yaml.clib, requests-oauthlib, python-dateutil, MarkupSafe, google-auth, ruamel.yaml, python-string-utils, kubernetes, jinja2, suds-py3, openshift, mtc Running setup.py install for openshift: started Running setup.py install for openshift: finished with status 'done' Running setup.py install for mtc: started Running setup.py install for mtc: finished with status 'done' Successfully installed MarkupSafe-2.1.5 cachetools-5.5.0 certifi-2024.7.4 charset-normalizer-3.3.2 google-auth-2.34.0 idna-3.7 jinja2-3.1.4 kubernetes-11.0.0 mtc-0.0.1 oauthlib-3.2.2 openshift-0.11.2 pyasn1-0.6.0 pyasn1-modules-0.4.0 python-dateutil-2.9.0.post0 python-string-utils-1.0.0 requests-2.32.3 requests-oauthlib-2.0.0 rsa-4.9 ruamel.yaml-0.18.6 ruamel.yaml.clib-0.2.8 six-1.16.0 suds-py3-1.4.5.0 urllib3-2.2.2 websocket-client-1.8.0 WARNING: You are using pip version 21.2.3; however, version 24.2 is available. You should consider upgrading via the '/alabama/venv/bin/python3 -m pip install --upgrade pip' command. go: downloading github.com/onsi/ginkgo/v2 v2.11.0 go: downloading github.com/onsi/gomega v1.27.8 go: downloading github.com/openshift/oadp-operator v1.0.2-0.20231003152846-925e5cc16370 go: downloading github.com/vmware-tanzu/velero v1.12.0 go: downloading k8s.io/apimachinery v0.25.6 go: downloading k8s.io/api v0.25.6 go: downloading k8s.io/client-go v0.25.6 go: downloading github.com/kubernetes-csi/external-snapshotter/client/v4 v4.2.0 go: downloading sigs.k8s.io/controller-runtime v0.12.2 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading k8s.io/utils v0.0.0-20230115233650-391b47cb4029 go: downloading github.com/apenella/go-ansible v1.1.5 go: downloading github.com/andygrunwald/go-jira v1.16.0 go: downloading github.com/backube/volsync v0.7.0 go: downloading github.com/konveyor/volume-snapshot-mover v0.0.0-20230510202920-14b0e79dbaaa go: downloading github.com/google/uuid v1.3.0 go: downloading github.com/openshift/api v0.0.0-20230414143018-3367bc7e6ac7 go: downloading github.com/openshift/client-go v0.0.0-20211209144617-7385dd6338e3 go: downloading github.com/operator-framework/api v0.14.1-0.20220413143725-33310d6154f3 go: downloading k8s.io/kubectl v0.25.6 go: downloading sigs.k8s.io/yaml v1.3.0 go: downloading github.com/google/go-cmp v0.5.9 go: downloading github.com/go-logr/logr v1.2.4 go: downloading github.com/evanphx/json-patch v5.6.0+incompatible go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/gofuzz v1.2.0 go: downloading github.com/apenella/go-common-utils/error v0.0.0-20210528133155-34ba915e28c8 go: downloading github.com/apenella/go-common-utils/data v0.0.0-20210528133155-34ba915e28c8 go: downloading k8s.io/klog/v2 v2.90.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.3 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/fatih/structs v1.1.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading github.com/google/go-querystring v1.1.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/trivago/tgo v1.0.7 go: downloading github.com/sirupsen/logrus v1.9.0 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading golang.org/x/net v0.12.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading golang.org/x/sys v0.10.0 go: downloading github.com/go-logr/zapr v1.2.3 go: downloading go.uber.org/zap v1.24.0 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/kube-openapi v0.0.0-20230118215034-64b6bb138190 go: downloading golang.org/x/time v0.3.0 go: downloading github.com/imdario/mergo v0.3.13 go: downloading golang.org/x/term v0.10.0 go: downloading k8s.io/apiextensions-apiserver v0.26.1 go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c go: downloading github.com/onsi/ginkgo v1.16.5 go: downloading github.com/stretchr/testify v1.8.2 go: downloading github.com/spf13/cobra v1.6.1 go: downloading k8s.io/cli-runtime v0.25.6 go: downloading k8s.io/component-base v0.26.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 go: downloading golang.org/x/tools v0.11.0 go: downloading github.com/golang/protobuf v1.5.3 go: downloading golang.org/x/text v0.11.0 go: downloading go.uber.org/atomic v1.10.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/google/gnostic v0.6.9 go: downloading google.golang.org/protobuf v1.30.0 go: downloading gomodules.xyz/jsonpatch/v2 v2.2.0 go: downloading k8s.io/kube-aggregator v0.19.12 go: downloading github.com/bombsimon/logrusr/v3 v3.0.0 go: downloading github.com/prometheus/client_golang v1.15.0 go: downloading golang.org/x/oauth2 v0.7.0 go: downloading github.com/moby/spdystream v0.2.0 go: downloading github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 go: downloading github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153 go: downloading github.com/jonboulle/clockwork v0.2.2 go: downloading k8s.io/component-helpers v0.26.1 go: downloading github.com/daviddengcn/go-colortext v1.0.0 go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de go: downloading github.com/docker/distribution v2.8.1+incompatible go: downloading github.com/moby/term v0.0.0-20220808134915-39b0c02b01ae go: downloading github.com/fvbommel/sortorder v1.0.1 go: downloading sigs.k8s.io/kustomize/kustomize/v4 v4.5.7 go: downloading github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d go: downloading github.com/lithammer/dedent v1.1.0 go: downloading sigs.k8s.io/kustomize/kyaml v0.13.9 go: downloading k8s.io/metrics v0.26.1 go: downloading github.com/chai2010/gettext-go v1.0.2 go: downloading github.com/MakeNowJust/heredoc v1.0.0 go: downloading github.com/mitchellh/go-wordwrap v1.0.0 go: downloading github.com/russross/blackfriday v1.5.2 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/pmezard/go-difflib v1.0.0 go: downloading github.com/inconshreveable/mousetrap v1.1.0 go: downloading github.com/google/pprof v0.0.0-20230705174524-200ffdc848b8 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/fsnotify/fsnotify v1.6.0 go: downloading google.golang.org/grpc v1.54.0 go: downloading github.com/hashicorp/go-plugin v1.4.3 go: downloading github.com/gobwas/glob v0.2.3 go: downloading github.com/stretchr/objx v0.5.0 go: downloading github.com/spf13/afero v1.9.3 go: downloading github.com/robfig/cron v1.1.0 go: downloading github.com/hashicorp/go-hclog v1.2.0 go: downloading go.uber.org/goleak v1.1.12 go: downloading github.com/prometheus/client_model v0.3.0 go: downloading github.com/prometheus/common v0.42.0 go: downloading sigs.k8s.io/kustomize/api v0.12.1 go: downloading github.com/fatih/camelcase v1.0.0 go: downloading github.com/golangplus/testing v1.0.0 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/go-openapi/swag v0.22.3 go: downloading github.com/creack/pty v1.1.11 go: downloading gotest.tools/v3 v3.0.3 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.10.0 go: downloading github.com/nxadm/tail v1.4.8 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.2.0 go: downloading github.com/prometheus/procfs v0.9.0 go: downloading github.com/mitchellh/go-testing-interface v1.0.0 go: downloading github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb go: downloading github.com/oklog/run v1.0.0 go: downloading github.com/Azure/go-autorest/autorest/to v0.3.0 go: downloading github.com/jhump/protoreflect v1.6.0 go: downloading github.com/Azure/go-autorest/autorest v0.11.27 go: downloading github.com/opencontainers/go-digest v1.0.0 go: downloading gotest.tools v2.2.0+incompatible go: downloading github.com/Azure/azure-sdk-for-go v67.2.0+incompatible go: downloading github.com/Azure/go-autorest v14.2.0+incompatible go: downloading github.com/Azure/go-autorest/autorest/azure/auth v0.5.8 go: downloading github.com/aws/aws-sdk-go v1.44.253 go: downloading github.com/joho/godotenv v1.3.0 go: downloading github.com/kopia/kopia v0.13.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 go: downloading github.com/fatih/color v1.15.0 go: downloading github.com/peterbourgon/diskv v2.0.1+incompatible go: downloading github.com/mattn/go-isatty v0.0.17 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4 go: downloading github.com/go-openapi/jsonreference v0.20.2 go: downloading google.golang.org/appengine v1.6.7 go: downloading github.com/emicklei/go-restful/v3 v3.10.1 go: downloading github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f go: downloading github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 go: downloading github.com/Azure/go-autorest/logger v0.2.1 go: downloading github.com/Azure/go-autorest/tracing v0.6.0 go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.2 go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.20 go: downloading github.com/Azure/go-autorest/autorest/azure/cli v0.4.2 go: downloading github.com/dimchansky/utfbom v1.1.1 go: downloading google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 go: downloading github.com/go-errors/errors v1.0.1 go: downloading github.com/google/btree v1.0.1 go: downloading github.com/go-openapi/jsonpointer v0.19.6 go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 go: downloading github.com/xlab/treeprint v1.1.0 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0 go: downloading golang.org/x/crypto v0.11.0 go: downloading github.com/mitchellh/go-homedir v1.1.0 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading go.opentelemetry.io/otel v1.14.0 go: downloading go.opentelemetry.io/otel/trace v1.14.0 go: downloading golang.org/x/sync v0.3.0 go: downloading golang.org/x/mod v0.12.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/edsrzf/mmap-go v1.1.0 go: downloading github.com/klauspost/compress v1.16.5 go: downloading github.com/gofrs/flock v0.8.1 go: downloading github.com/klauspost/pgzip v1.2.5 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/natefinch/atomic v1.0.1 go: downloading golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb go: downloading github.com/klauspost/reedsolomon v1.11.7 go: downloading github.com/zeebo/blake3 v0.2.3 go: downloading github.com/chmduquesne/rollinghash v4.0.0+incompatible go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.3.0 go: downloading github.com/Azure/azure-storage-blob-go v0.15.0 go: downloading cloud.google.com/go/storage v1.30.1 go: downloading google.golang.org/api v0.120.0 go: downloading github.com/minio/minio-go/v7 v7.0.52 go: downloading cloud.google.com/go v0.110.0 go: downloading go.starlark.net v0.0.0-20201006213952-227f4aabceb5 go: downloading github.com/sergi/go-diff v1.1.0 go: downloading github.com/zeebo/assert v1.1.0 go: downloading github.com/tg123/go-htpasswd v1.2.1 go: downloading github.com/kopia/htmluibuild v0.0.0-20230326183719-f482ef17e2c9 go: downloading github.com/zalando/go-keyring v0.2.2 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/Azure/azure-pipeline-go v0.2.3 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading cloud.google.com/go/compute v1.19.0 go: downloading github.com/minio/md5-simd v1.1.2 go: downloading github.com/minio/sha256-simd v1.0.0 go: downloading gopkg.in/ini.v1 v1.67.0 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 go: downloading cloud.google.com/go/iam v0.13.0 go: downloading github.com/googleapis/gax-go/v2 v2.8.0 go: downloading golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 go: downloading github.com/hanwen/go-fuse/v2 v2.3.0 go: downloading github.com/danieljoos/wincred v1.1.2 go: downloading github.com/alessio/shellescape v1.4.1 go: downloading github.com/godbus/dbus/v5 v5.1.0 go: downloading github.com/GehirnInc/crypt v0.0.0-20200316065508-bb7000b8a962 go: downloading github.com/mattn/go-ieproxy v0.0.1 go: downloading github.com/rs/xid v1.4.0 go: downloading go.opencensus.io v0.24.0 go: downloading cloud.google.com/go/longrunning v0.4.1 go: downloading github.com/frankban/quicktest v1.14.3 go: downloading github.com/google/martian/v3 v3.3.2 go: downloading github.com/google/martian v2.1.0+incompatible go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/zeebo/pcg v1.0.1 go: downloading github.com/jmespath/go-jmespath/internal/testify v1.5.1 go: downloading github.com/google/s2a-go v0.1.2 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.3 go: downloading github.com/Azure/go-autorest/autorest/validation v0.2.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v0.21.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v0.8.3 go: downloading github.com/dnaeon/go-vcr v1.2.0 + readonly 'RED=\e[31m' + RED='\e[31m' + readonly 'BLUE=\033[34m' + BLUE='\033[34m' + readonly 'CLEAR=\e[39m' + CLEAR='\e[39m' ++ oc get infrastructures cluster -o 'jsonpath={.status.platform}' ++ awk '{print tolower($0)}' + CLOUD_PROVIDER=ibmcloud + [[ ibmcloud == *-arm* ]] + [[ ibmcloud == *-fips* ]] + E2E_TIMEOUT_MULTIPLIER=2 + export NAMESPACE=openshift-adp + NAMESPACE=openshift-adp + export PROVIDER=ibmcloud + PROVIDER=ibmcloud ++ echo aws ++ awk '{print tolower($0)}' + BACKUP_LOCATION=aws + export BACKUP_LOCATION=aws + BACKUP_LOCATION=aws + export BUCKET=ci-op-1nitxr4v-interopoadp + BUCKET=ci-op-1nitxr4v-interopoadp + OADP_CREDS_FILE=/tmp/test-settings/credentials + OADP_VSL_CREDS_FILE=/tmp/test-settings/ibmcloud_vsl_creds +++ readlink -f /alabama/cspi/test_settings/scripts/test_runner.sh ++ dirname /alabama/cspi/test_settings/scripts/test_runner.sh + readonly SCRIPT_DIR=/alabama/cspi/test_settings/scripts + SCRIPT_DIR=/alabama/cspi/test_settings/scripts ++ cd /alabama/cspi/test_settings/scripts ++ git rev-parse --show-toplevel + readonly TOP_DIR=/alabama/cspi + TOP_DIR=/alabama/cspi + echo /alabama/cspi /alabama/cspi + TESTS_FOLDER=e2e ++ oc get nodes -o 'jsonpath={.items[*].metadata.labels.kubernetes\.io/arch}' ++ tr ' ' '\n' ++ sort -u ++ xargs + export NODES_ARCHITECTURE=amd64 + NODES_ARCHITECTURE=amd64 + export OADP_REPOSITORY=redhat + OADP_REPOSITORY=redhat + SKIP_DPA_CREATION=false ++ oc get ns openshift-storage ++ echo false + OPENSHIFT_STORAGE=false + '[' redhat == upstream-velero ']' + '[' false == true ']' ++ oc get storageclass -o 'jsonpath={range .items[*]}{@.metadata.name}{" "}{@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}{end}' ++ awk '$2=="true"{print $1}' ++ wc -l + NUM_DEFAULT_STORAGE_CLASS=1 + '[' 1 -ne 1 ']' ++ oc get storageclass -o 'jsonpath={.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=='\''true'\'')].metadata.name}' + DEFAULT_SC=ibmc-vpc-block-10iops-tier + export STORAGE_CLASS=ibmc-vpc-block-10iops-tier + STORAGE_CLASS=ibmc-vpc-block-10iops-tier + '[' -n ibmc-vpc-block-10iops-tier ']' + '[' ibmc-vpc-block-10iops-tier '!=' ibmc-vpc-block-10iops-tier ']' + export STORAGE_CLASS_OPENSHIFT_STORAGE=ibmc-vpc-block-10iops-tier + STORAGE_CLASS_OPENSHIFT_STORAGE=ibmc-vpc-block-10iops-tier + echo 'Using the StorageClass for openshift-storage: ibmc-vpc-block-10iops-tier' Using the StorageClass for openshift-storage: ibmc-vpc-block-10iops-tier + [[ amd64 != \a\m\d\6\4 ]] + TEST_FILTER='!// || (// && !exclude_ibmcloud && (!/target/ || target_ibmcloud) ) ' + [[ ibmcloud =~ ^osp ]] + [[ ibmcloud =~ ^vsphere ]] + [[ ibmcloud =~ ^gcp-wif ]] + [[ ibmcloud =~ ^ibmcloud ]] + TEST_FILTER='!// || (// && !exclude_ibmcloud && (!/target/ || target_ibmcloud) && !vsl ) ' ++ oc config current-context ++ awk -F / '{print $2}' + SETTINGS_TMP=/alabama/cspi/output_files/api-ci-op-1nitxr4v-2f653-zstream-interop-ccitredhat-com:6443 + mkdir -p /alabama/cspi/output_files/api-ci-op-1nitxr4v-2f653-zstream-interop-ccitredhat-com:6443 ++ oc get authentication cluster -o 'jsonpath={.spec.serviceAccountIssuer}' + IS_OIDC= + '[' '!' -z ']' + [[ aws == \a\w\s ]] + export PROVIDER=aws + PROVIDER=aws + export CREDS_SECRET_REF=cloud-credentials + CREDS_SECRET_REF=cloud-credentials ++ oc get infrastructures cluster -o 'jsonpath={.status.platformStatus.aws.region}' --allow-missing-template-keys=false ++ echo us-east-2 + export REGION=us-east-2 + REGION=us-east-2 + settings_script=aws_settings.sh + '[' ibmcloud == aws-sts ']' + BUCKET=ci-op-1nitxr4v-interopoadp + TMP_DIR=/alabama/cspi/output_files/api-ci-op-1nitxr4v-2f653-zstream-interop-ccitredhat-com:6443 + source /alabama/cspi/test_settings/scripts/aws_settings.sh ++ cat ++ [[ ibmcloud == *aws* ]] ++ echo -e '\n }\n}' +++ cat /alabama/cspi/output_files/api-ci-op-1nitxr4v-2f653-zstream-interop-ccitredhat-com:6443/settings.json ++ x='{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-2" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-1nitxr4v-interopoadp" } } } ] } }' ++ echo '{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-2" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-1nitxr4v-interopoadp" } } } ] } }' ++ grep -o '^[^#]*' + FILE_SETTINGS_NAME=settings.json + printf '\033[34mGenerated settings file under /alabama/cspi/output_files/api-ci-op-1nitxr4v-2f653-zstream-interop-ccitredhat-com:6443/settings.json\e[39m\n' Generated settings file under /alabama/cspi/output_files/api-ci-op-1nitxr4v-2f653-zstream-interop-ccitredhat-com:6443/settings.json + cat /alabama/cspi/output_files/api-ci-op-1nitxr4v-2f653-zstream-interop-ccitredhat-com:6443/settings.json ++ oc get volumesnapshotclass -o name + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/vpc-block-snapshot snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/vpc-block-snapshot annotated ++ ./e2e/must-gather/get-latest-build.sh + oc get configmaps -n default must-gather-image + UPSTREAM_VERSION=99.0.0 ++ oc get OperatorCondition -n openshift-adp -o 'jsonpath={.items[*].metadata.name}' ++ awk -F v '{print $2}' + OADP_VERSION=1.4.0 + '[' -z 1.4.0 ']' + '[' 1.4.0 == 99.0.0 ']' ++ oc get sub redhat-oadp-operator -n openshift-adp -o 'jsonpath={.spec.source}' + OADP_REPO=redhat-operators + '[' -z redhat-operators ']' + '[' redhat-operators == redhat-operators ']' + REGISTRY_PATH=registry.redhat.io/oadp/oadp-mustgather-rhel9: + TAG=1.4.0 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + echo registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 + '[' -z registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 ']' + export NUM_OF_OADP_INSTANCES=1 + NUM_OF_OADP_INSTANCES=1 + ginkgo run -p --nodes=1 -mod=mod e2e/ --show-node-events -- -credentials_file=/tmp/test-settings/credentials -vsl_credentials_file=/tmp/test-settings/ibmcloud_vsl_creds -oadp_namespace=openshift-adp -settings=/alabama/cspi/output_files/api-ci-op-1nitxr4v-2f653-zstream-interop-ccitredhat-com:6443/settings.json -must_gather_image=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 -timeout_multiplier=2 -skip_dpa_creation=false --ginkgo.junit-report=/logs/artifacts/junit_oadp_interop_results.xml '--ginkgo.label-filter=!// || (// && !exclude_ibmcloud && (!/target/ || target_ibmcloud) && !vsl ) ' --ginkgo.focus=interop Ginkgo detected a version mismatch between the Ginkgo CLI and the version of Ginkgo imported by your packages: Ginkgo CLI Version: 2.20.0 Mismatched package versions found: 2.11.0 used by e2e Ginkgo will continue to attempt to run but you may see errors (including flag parsing errors) and should either update your go.mod or your version of the Ginkgo CLI to match. To install the matching version of the CLI run go install github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file. Alternatively you can use go run github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file to invoke the matching version of the Ginkgo CLI. If you are attempting to test multiple packages that each have a different version of the Ginkgo library with a single Ginkgo CLI that is currently unsupported.  2024/08/19 11:30:28 Setting up clients 2024/08/19 11:30:28 Getting default StorageClass... 2024/08/19 11:30:28 Checking default storage class count Run the command: oc get sc 2024/08/19 11:30:28 Got default StorageClass ibmc-vpc-block-10iops-tier 2024/08/19 11:30:28 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE ibmc-vpc-block-10iops-tier (default) vpc.block.csi.ibm.io Delete WaitForFirstConsumer true 27m ibmc-vpc-block-5iops-tier vpc.block.csi.ibm.io Delete WaitForFirstConsumer true 27m ibmc-vpc-block-custom vpc.block.csi.ibm.io Delete WaitForFirstConsumer true 27m 2024/08/19 11:30:28 Using velero prefix: velero-e2e-6fcaada4-5e1e-11ef-906a-0a580a810aef 2024/08/19 11:30:28 Checking default storage class count Running Suite: OADP E2E Suite - /alabama/cspi/e2e ================================================= Random Seed: 1724066983 Will run 9 of 193 specs ------------------------------ [SynchronizedBeforeSuite]  /alabama/cspi/e2e/e2e_suite_test.go:76 2024/08/19 11:30:28 Error reading VSL credentials file: open /tmp/test-settings/ibmcloud_vsl_creds: no such file or directory [SynchronizedBeforeSuite] PASSED [0.045 seconds] ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ [datamover] DataMover: Backup/Restore stateful application with CSI  [tc-id:OADP-439][interop] MySQL application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:114 2024/08/19 11:30:29 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 08/19/24 11:30:29.024 2024/08/19 11:30:29 native-datamover 2024/08/19 11:30:29 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "349e2a4c-bbcd-4cfa-aae9-bb428097c83a", "resourceVersion": "32780", "generation": 1, "creationTimestamp": "2024-08-19T11:30:29Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T11:30:29Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 08/19/24 11:30:29.075 2024/08/19 11:30:29 Waiting for velero pod to be running 2024/08/19 11:30:34 pod: velero-7f76d9448c-kh7j5 is not yet running with status: {Pending [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC ContainersNotInitialized containers with incomplete status: [openshift-velero-plugin velero-plugin-for-aws]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC }] 10.241.65.5 [] 2024-08-19 11:30:29 +0000 UTC [{openshift-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:1eea6e649be9406b56a02061c70eff02fb61f1be20afc116ec85f90d2ef74d15 0xc00103b25b} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:c328548fd34a8d790e002fac1e9e7327231fd207a83612cfa2aa49991ea0bf47 0xc00103b25c}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:777ec93a24cafd94022804da68d02da6f64041716ec4a39a85017d7289846d72 0xc00103b286}] Burstable []} 2024/08/19 11:30:39 pod: velero-7f76d9448c-kh7j5 is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:37 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC ContainersNotInitialized containers with incomplete status: [velero-plugin-for-aws]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC }] 10.241.65.5 10.129.2.17 [{10.129.2.17}] 2024-08-19 11:30:29 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-08-19 11:30:37 +0000 UTC,FinishedAt:2024-08-19 11:30:37 +0000 UTC,ContainerID:cri-o://6b5812b8cedce53bbaf64ce2ab1dee8ec73aee5e7c020fa30445493bd8a0ac9e,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:1eea6e649be9406b56a02061c70eff02fb61f1be20afc116ec85f90d2ef74d15 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:1eea6e649be9406b56a02061c70eff02fb61f1be20afc116ec85f90d2ef74d15 cri-o://6b5812b8cedce53bbaf64ce2ab1dee8ec73aee5e7c020fa30445493bd8a0ac9e 0xc000fc2119} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:c328548fd34a8d790e002fac1e9e7327231fd207a83612cfa2aa49991ea0bf47 0xc000fc211a}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:777ec93a24cafd94022804da68d02da6f64041716ec4a39a85017d7289846d72 0xc000fc2136}] Burstable []} 2024/08/19 11:30:44 pod: velero-7f76d9448c-kh7j5 is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:37 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC ContainersNotInitialized containers with incomplete status: [velero-plugin-for-aws]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-08-19 11:30:29 +0000 UTC }] 10.241.65.5 10.129.2.17 [{10.129.2.17}] 2024-08-19 11:30:29 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-08-19 11:30:37 +0000 UTC,FinishedAt:2024-08-19 11:30:37 +0000 UTC,ContainerID:cri-o://6b5812b8cedce53bbaf64ce2ab1dee8ec73aee5e7c020fa30445493bd8a0ac9e,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:1eea6e649be9406b56a02061c70eff02fb61f1be20afc116ec85f90d2ef74d15 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:1eea6e649be9406b56a02061c70eff02fb61f1be20afc116ec85f90d2ef74d15 cri-o://6b5812b8cedce53bbaf64ce2ab1dee8ec73aee5e7c020fa30445493bd8a0ac9e 0xc000fc2869} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:c328548fd34a8d790e002fac1e9e7327231fd207a83612cfa2aa49991ea0bf47 0xc000fc286a}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:777ec93a24cafd94022804da68d02da6f64041716ec4a39a85017d7289846d72 0xc000fc2886}] Burstable []} 2024/08/19 11:30:49 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 11:30:49.149 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/08/19 11:30:49 The 'openshift-storage' namespace does not exist 2024/08/19 11:30:49 Using default CSI driver based on infrastructure: vpc.block.csi.ibm.io 2024/08/19 11:30:49 Snapclass 'example-snapclass' doesn't exist, creating 2024/08/19 11:30:49 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 11:30:49 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier 2024/08/19 11:30:49 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 08/19/24 11:30:49.34 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] /usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ci-op-1nitxr4v-2f653.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-439] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). FAILED - RETRYING: [localhost]: Check pod status (28 retries left). FAILED - RETRYING: [localhost]: Check pod status (27 retries left). FAILED - RETRYING: [localhost]: Check pod status (26 retries left). FAILED - RETRYING: [localhost]: Check pod status (25 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] Pausing for 30 seconds TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=20  changed=10  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/08/19 11:32:07 2024-08-19 11:30:51,093 p=21674 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:30:51,093 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:30:51,319 p=21674 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:30:51,319 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:30:51,547 p=21674 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:30:51,548 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:30:51,565 p=21674 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:30:51,565 p=21674 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:30:52,497 p=21674 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:30:52,498 p=21674 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:30:52,528 p=21674 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:30:52,529 p=21674 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:30:52,547 p=21674 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:30:52,547 p=21674 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:30:52,562 p=21674 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:30:53,110 p=21674 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:30:53,110 p=21674 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:30:53,864 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-439] *** 2024-08-19 11:30:53,864 p=21674 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:30:53,864 p=21674 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:30:54,343 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-08-19 11:30:54,343 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:30:55,230 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-08-19 11:30:55,231 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:31:29,363 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-08-19 11:31:29,363 p=21674 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:31:29,878 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-08-19 11:31:29,879 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:31:35,477 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 11:31:35,478 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:31:36,067 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-08-19 11:31:36,067 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:31:36,851 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-08-19 11:31:36,851 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:31:37,418 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-08-19 11:31:37,419 p=21674 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:31:37,436 p=21674 u=1015750000 n=ansible | Pausing for 30 seconds 2024-08-19 11:32:07,471 p=21674 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** 2024-08-19 11:32:07,471 p=21674 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:32:07,544 p=21674 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:32:07,544 p=21674 u=1015750000 n=ansible | localhost : ok=20 changed=10 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 08/19/24 11:32:07.583 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 11:32:12 2024-08-19 11:32:08,960 p=22222 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:32:08,960 p=22222 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:32:09,186 p=22222 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:32:09,186 p=22222 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:32:09,417 p=22222 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:32:09,417 p=22222 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:32:09,431 p=22222 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:32:09,432 p=22222 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:32:09,710 p=22222 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:32:09,711 p=22222 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:32:09,737 p=22222 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:32:09,737 p=22222 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:32:09,754 p=22222 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:32:09,754 p=22222 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:32:09,764 p=22222 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:32:10,287 p=22222 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:32:10,288 p=22222 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:32:11,234 p=22222 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-08-19 11:32:11,235 p=22222 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:32:11,235 p=22222 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:32:11,569 p=22222 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 11:32:11,569 p=22222 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:32:11,983 p=22222 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-08-19 11:32:11,983 p=22222 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:32:12,573 p=22222 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-08-19 11:32:12,574 p=22222 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:32:12,586 p=22222 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:32:12,586 p=22222 u=1015750000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 STEP: Creating backup mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef @ 08/19/24 11:32:12.627 2024/08/19 11:32:12 Wait until backup mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef is completed backup phase: WaitingForPluginOperations DataUpload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hzdmb phase: Accepted DataUpload Name: mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hzdmb and status: Accepted 2024/08/19 11:32:32 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hzdmb", "generateName": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "2f169284-b2aa-436e-9782-9579fc29eb13", "resourceVersion": "33558", "generation": 2, "creationTimestamp": "2024-08-19T11:32:19Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "du-82902024-a654-408c-a65e-6d9e873225f6.17782e33-4830-4ce781f38", "velero.io/backup-name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "velero.io/backup-uid": "82902024-a654-408c-a65e-6d9e873225f6", "velero.io/pvc-uid": "17782e33-4830-4cec-8759-5d4c893e06f4" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "uid": "82902024-a654-408c-a65e-6d9e873225f6", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"82902024-a654-408c-a65e-6d9e873225f6\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-data-7xzlk", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "mysql-data", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataUpload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-d78gt phase: Accepted DataUpload Name: mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-d78gt and status: Accepted 2024/08/19 11:32:32 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-d78gt", "generateName": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "ec6fc8ac-e9f3-4676-89ca-4fb575241118", "resourceVersion": "33617", "generation": 2, "creationTimestamp": "2024-08-19T11:32:25Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z", "velero.io/async-operation-id": "du-82902024-a654-408c-a65e-6d9e873225f6.70bdc4ca-b4f6-40e499738", "velero.io/backup-name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "velero.io/backup-uid": "82902024-a654-408c-a65e-6d9e873225f6", "velero.io/pvc-uid": "70bdc4ca-b4f6-40ec-8ad3-3544807fe470" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "uid": "82902024-a654-408c-a65e-6d9e873225f6", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:25Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:25Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"82902024-a654-408c-a65e-6d9e873225f6\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-data1-jsmvz", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "mysql-data1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hzdmb phase: Accepted DataUpload Name: mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hzdmb and status: Accepted 2024/08/19 11:32:52 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hzdmb", "generateName": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "2f169284-b2aa-436e-9782-9579fc29eb13", "resourceVersion": "33558", "generation": 2, "creationTimestamp": "2024-08-19T11:32:19Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "du-82902024-a654-408c-a65e-6d9e873225f6.17782e33-4830-4ce781f38", "velero.io/backup-name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "velero.io/backup-uid": "82902024-a654-408c-a65e-6d9e873225f6", "velero.io/pvc-uid": "17782e33-4830-4cec-8759-5d4c893e06f4" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "uid": "82902024-a654-408c-a65e-6d9e873225f6", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"82902024-a654-408c-a65e-6d9e873225f6\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-data-7xzlk", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "mysql-data", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataUpload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-d78gt phase: Accepted DataUpload Name: mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-d78gt and status: Accepted 2024/08/19 11:32:52 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-d78gt", "generateName": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "ec6fc8ac-e9f3-4676-89ca-4fb575241118", "resourceVersion": "33617", "generation": 2, "creationTimestamp": "2024-08-19T11:32:25Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z", "velero.io/async-operation-id": "du-82902024-a654-408c-a65e-6d9e873225f6.70bdc4ca-b4f6-40e499738", "velero.io/backup-name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "velero.io/backup-uid": "82902024-a654-408c-a65e-6d9e873225f6", "velero.io/pvc-uid": "70bdc4ca-b4f6-40ec-8ad3-3544807fe470" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "uid": "82902024-a654-408c-a65e-6d9e873225f6", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:25Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:25Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"82902024-a654-408c-a65e-6d9e873225f6\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-data1-jsmvz", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "mysql-data1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hzdmb phase: Completed DataUpload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-d78gt phase: Accepted DataUpload Name: mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-d78gt and status: Accepted 2024/08/19 11:33:12 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-d78gt", "generateName": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "ec6fc8ac-e9f3-4676-89ca-4fb575241118", "resourceVersion": "33617", "generation": 2, "creationTimestamp": "2024-08-19T11:32:25Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z", "velero.io/async-operation-id": "du-82902024-a654-408c-a65e-6d9e873225f6.70bdc4ca-b4f6-40e499738", "velero.io/backup-name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "velero.io/backup-uid": "82902024-a654-408c-a65e-6d9e873225f6", "velero.io/pvc-uid": "70bdc4ca-b4f6-40ec-8ad3-3544807fe470" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "uid": "82902024-a654-408c-a65e-6d9e873225f6", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:25Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:32:25Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"82902024-a654-408c-a65e-6d9e873225f6\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-data1-jsmvz", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "mysql-data1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } backup phase: Completed STEP: Verify backup mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef has completed successfully @ 08/19/24 11:33:32.751 2024/08/19 11:33:32 Backup for case mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef succeeded STEP: Delete the appplication resources mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef @ 08/19/24 11:33:32.759 STEP: Cleanup Application for case mysql @ 08/19/24 11:33:32.76 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2024/08/19 11:33:51 2024-08-19 11:33:34,226 p=22531 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:33:34,227 p=22531 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:33:34,476 p=22531 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:33:34,477 p=22531 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:33:34,722 p=22531 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:33:34,723 p=22531 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:33:34,739 p=22531 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:33:34,739 p=22531 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:33:35,042 p=22531 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:33:35,042 p=22531 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:33:35,070 p=22531 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:33:35,070 p=22531 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:33:35,089 p=22531 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:33:35,089 p=22531 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:33:35,099 p=22531 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:33:35,649 p=22531 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:33:35,649 p=22531 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:33:51,547 p=22531 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** 2024-08-19 11:33:51,548 p=22531 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:33:51,548 p=22531 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:33:51,808 p=22531 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:33:51,808 p=22531 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 STEP: Create restore mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef from backup mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef @ 08/19/24 11:33:51.852 2024/08/19 11:33:51 Wait until restore mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef completes restore phase: WaitingForPluginOperations DataDownload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hpf84 phase: Accepted DataDownload Name: mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hpf84 and status: Accepted 2024/08/19 11:34:11 { "kind": "DataDownload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hpf84", "generateName": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "89456797-6b06-4811-992e-010259144aee", "resourceVersion": "34417", "generation": 2, "creationTimestamp": "2024-08-19T11:33:54Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "dd-12e363d0-7cd7-472b-8e81-b1539f205c6d.17782e33-4830-4ce1cb2d5", "velero.io/restore-name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "velero.io/restore-uid": "12e363d0-7cd7-472b-8e81-b1539f205c6d" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Restore", "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "uid": "12e363d0-7cd7-472b-8e81-b1539f205c6d", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:33:54Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:33:54Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/restore-name": {}, "f:velero.io/restore-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"12e363d0-7cd7-472b-8e81-b1539f205c6d\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:operationTimeout": {}, "f:snapshotID": {}, "f:sourceNamespace": {}, "f:targetVolume": { ".": {}, "f:namespace": {}, "f:pv": {}, "f:pvc": {} } }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "targetVolume": { "pvc": "mysql-data", "pv": "", "namespace": "test-oadp-439" }, "backupStorageLocation": "ts-dpa-1", "snapshotID": "f1eb00f72166e7717a9f66f49b332508", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataDownload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-zpzw2 phase: Accepted DataDownload Name: mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-zpzw2 and status: Accepted 2024/08/19 11:34:11 { "kind": "DataDownload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-zpzw2", "generateName": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "ac474086-c3d2-4267-be54-8a0f57e331dc", "resourceVersion": "34428", "generation": 2, "creationTimestamp": "2024-08-19T11:33:54Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z", "velero.io/async-operation-id": "dd-12e363d0-7cd7-472b-8e81-b1539f205c6d.70bdc4ca-b4f6-40e9eea75", "velero.io/restore-name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "velero.io/restore-uid": "12e363d0-7cd7-472b-8e81-b1539f205c6d" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Restore", "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "uid": "12e363d0-7cd7-472b-8e81-b1539f205c6d", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:33:54Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:33:54Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/restore-name": {}, "f:velero.io/restore-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"12e363d0-7cd7-472b-8e81-b1539f205c6d\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:operationTimeout": {}, "f:snapshotID": {}, "f:sourceNamespace": {}, "f:targetVolume": { ".": {}, "f:namespace": {}, "f:pv": {}, "f:pvc": {} } }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "targetVolume": { "pvc": "mysql-data1", "pv": "", "namespace": "test-oadp-439" }, "backupStorageLocation": "ts-dpa-1", "snapshotID": "7da14135f45d9eda8e55b4d75bd2484a", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } restore phase: WaitingForPluginOperations DataDownload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hpf84 phase: InProgress DataDownload Name: mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hpf84 and status: InProgress 2024/08/19 11:34:31 { "kind": "DataDownload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-hpf84", "generateName": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "89456797-6b06-4811-992e-010259144aee", "resourceVersion": "34752", "generation": 4, "creationTimestamp": "2024-08-19T11:33:54Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "dd-12e363d0-7cd7-472b-8e81-b1539f205c6d.17782e33-4830-4ce1cb2d5", "velero.io/restore-name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "velero.io/restore-uid": "12e363d0-7cd7-472b-8e81-b1539f205c6d" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Restore", "name": "mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef", "uid": "12e363d0-7cd7-472b-8e81-b1539f205c6d", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:33:54Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/restore-name": {}, "f:velero.io/restore-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"12e363d0-7cd7-472b-8e81-b1539f205c6d\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:operationTimeout": {}, "f:snapshotID": {}, "f:sourceNamespace": {}, "f:targetVolume": { ".": {}, "f:namespace": {}, "f:pv": {}, "f:pvc": {} } }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:34:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:node": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "targetVolume": { "pvc": "mysql-data", "pv": "", "namespace": "test-oadp-439" }, "backupStorageLocation": "ts-dpa-1", "snapshotID": "f1eb00f72166e7717a9f66f49b332508", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "InProgress", "startTimestamp": "2024-08-19T11:34:30Z", "progress": {}, "node": "ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z" } } DataDownload mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-zpzw2 phase: Completed restore phase: Completed STEP: Validate the application after restore @ 08/19/24 11:34:52.01 STEP: Verify Application deployment for case mysql @ 08/19/24 11:34:52.01 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 11:34:57 2024-08-19 11:34:53,618 p=22743 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:34:53,618 p=22743 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:34:53,864 p=22743 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:34:53,865 p=22743 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:34:54,143 p=22743 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:34:54,143 p=22743 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:34:54,159 p=22743 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:34:54,159 p=22743 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:34:54,507 p=22743 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:34:54,507 p=22743 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:34:54,541 p=22743 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:34:54,541 p=22743 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:34:54,562 p=22743 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:34:54,562 p=22743 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:34:54,577 p=22743 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:34:55,129 p=22743 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:34:55,129 p=22743 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:34:56,181 p=22743 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-08-19 11:34:56,181 p=22743 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:34:56,181 p=22743 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:34:56,510 p=22743 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 11:34:56,510 p=22743 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:34:56,944 p=22743 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-08-19 11:34:56,945 p=22743 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:34:57,661 p=22743 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-08-19 11:34:57,661 p=22743 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:34:57,675 p=22743 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:34:57,675 p=22743 u=1015750000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 11:34:57 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/08/19 11:34:57 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2024/08/19 11:35:16 2024-08-19 11:34:59,262 p=23044 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:34:59,262 p=23044 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:34:59,506 p=23044 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:34:59,506 p=23044 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:34:59,779 p=23044 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:34:59,779 p=23044 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:34:59,796 p=23044 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:34:59,796 p=23044 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:00,122 p=23044 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:35:00,123 p=23044 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:00,155 p=23044 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:35:00,155 p=23044 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:00,174 p=23044 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:35:00,174 p=23044 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:00,187 p=23044 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:35:00,719 p=23044 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:35:00,720 p=23044 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:16,639 p=23044 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** 2024-08-19 11:35:16,640 p=23044 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:35:16,640 p=23044 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:35:16,894 p=23044 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:35:16,894 p=23044 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 2024/08/19 11:35:16 Cleaning setup resources for the backup 2024/08/19 11:35:16 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 11:35:16 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier 2024/08/19 11:35:16 Deleting VolumeSnapshotClass 'example-snapclass' • [288.031 seconds] ------------------------------ [datamover] DataMover: Backup/Restore stateful application with CSI  [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:130 2024/08/19 11:35:17 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 08/19/24 11:35:17.356 2024/08/19 11:35:17 native-datamover 2024/08/19 11:35:17 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "4f0a3b0b-cdb3-4f58-a723-edbff9617672", "resourceVersion": "35144", "generation": 1, "creationTimestamp": "2024-08-19T11:35:17Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T11:35:17Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 08/19/24 11:35:17.432 2024/08/19 11:35:17 Waiting for velero pod to be running 2024/08/19 11:35:17 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/08/19 11:35:17 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "4f0a3b0b-cdb3-4f58-a723-edbff9617672", "resourceVersion": "35144", "generation": 1, "creationTimestamp": "2024-08-19T11:35:17Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T11:35:17Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 11:35:22.469 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/08/19 11:35:22 The 'openshift-storage' namespace does not exist 2024/08/19 11:35:22 Using default CSI driver based on infrastructure: vpc.block.csi.ibm.io 2024/08/19 11:35:22 Snapclass 'example-snapclass' doesn't exist, creating 2024/08/19 11:35:22 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 11:35:22 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier 2024/08/19 11:35:22 Checking for correct number of running NodeAgent pods... STEP: Installing application for case cassandra-e2e @ 08/19/24 11:35:22.701 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pods status (30 retries left). FAILED - RETRYING: [localhost]: Check pods status (29 retries left). FAILED - RETRYING: [localhost]: Check pods status (28 retries left). FAILED - RETRYING: [localhost]: Check pods status (27 retries left). FAILED - RETRYING: [localhost]: Check pods status (26 retries left). FAILED - RETRYING: [localhost]: Check pods status (25 retries left). FAILED - RETRYING: [localhost]: Check pods status (24 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (30 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (29 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (28 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (27 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (26 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (25 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (24 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (23 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (22 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (21 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (20 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (19 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (18 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=18  changed=9  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/08/19 11:38:50 2024-08-19 11:35:24,098 p=23272 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:35:24,098 p=23272 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:35:24,334 p=23272 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:35:24,335 p=23272 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:35:24,570 p=23272 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:35:24,570 p=23272 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:35:24,585 p=23272 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:35:24,586 p=23272 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:24,864 p=23272 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:35:24,864 p=23272 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:24,890 p=23272 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:35:24,890 p=23272 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:24,907 p=23272 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:35:24,908 p=23272 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:24,919 p=23272 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:35:25,486 p=23272 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:35:25,486 p=23272 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:26,555 p=23272 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** 2024-08-19 11:35:26,556 p=23272 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:35:26,556 p=23272 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:35:27,062 p=23272 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** 2024-08-19 11:35:27,063 p=23272 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:35:27,355 p=23272 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** 2024-08-19 11:35:27,355 p=23272 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:35:28,110 p=23272 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** 2024-08-19 11:35:28,110 p=23272 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:35:28,707 p=23272 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** 2024-08-19 11:35:28,707 p=23272 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:36:08,205 p=23272 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-08-19 11:36:08,205 p=23272 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:38:46,708 p=23272 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-08-19 11:38:46,709 p=23272 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:38:50,714 p=23272 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** 2024-08-19 11:38:50,714 p=23272 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:38:50,811 p=23272 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:38:50,812 p=23272 u=1015750000 n=ansible | localhost : ok=18 changed=9 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 08/19/24 11:38:50.863 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=15  changed=7  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/08/19 11:39:02 2024-08-19 11:38:52,300 p=24162 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:38:52,300 p=24162 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:38:52,528 p=24162 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:38:52,528 p=24162 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:38:52,759 p=24162 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:38:52,759 p=24162 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:38:52,774 p=24162 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:38:52,774 p=24162 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:38:53,062 p=24162 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:38:53,062 p=24162 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:38:53,089 p=24162 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:38:53,089 p=24162 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:38:53,107 p=24162 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:38:53,107 p=24162 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:38:53,117 p=24162 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:38:53,632 p=24162 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:38:53,632 p=24162 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:38:54,509 p=24162 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-08-19 11:38:54,509 p=24162 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:38:54,509 p=24162 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:39:00,230 p=24162 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-08-19 11:39:00,230 p=24162 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:39:01,518 p=24162 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** 2024-08-19 11:39:01,518 p=24162 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:39:01,977 p=24162 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** 2024-08-19 11:39:01,977 p=24162 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:39:02,145 p=24162 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** 2024-08-19 11:39:02,145 p=24162 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:39:02,159 p=24162 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:39:02,159 p=24162 u=1015750000 n=ansible | localhost : ok=15 changed=7 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Creating backup cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef @ 08/19/24 11:39:02.199 2024/08/19 11:39:02 Wait until backup cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef is completed backup phase: WaitingForPluginOperations DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54 and status: Accepted 2024/08/19 11:39:22 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "747c7836-ea69-460f-821c-5b8dce19e1d8", "resourceVersion": "36754", "generation": 2, "creationTimestamp": "2024-08-19T11:39:08Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.596a43b7-9dda-4586231bd", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "596a43b7-9dda-4580-806b-eab5aa6441c3" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:08Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:08Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-7hf85", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46 and status: Accepted 2024/08/19 11:39:22 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "9c3ff5b5-e8c9-49b4-b0ae-3bea220fe731", "resourceVersion": "36814", "generation": 2, "creationTimestamp": "2024-08-19T11:39:13Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.63a19307-5f52-461739f0f", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "63a19307-5f52-461e-85ed-02548941d2d5" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:13Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:14Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-blfpw", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 and status: Accepted 2024/08/19 11:39:22 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "ac63c8c2-57a3-4957-a19c-52e6acfa9413", "resourceVersion": "36866", "generation": 2, "creationTimestamp": "2024-08-19T11:39:19Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.d9cef209-e44d-4b8c39b0a", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "d9cef209-e44d-4b83-8554-1a891cfd040a" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-6g45h", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54 and status: Accepted 2024/08/19 11:39:42 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "747c7836-ea69-460f-821c-5b8dce19e1d8", "resourceVersion": "36754", "generation": 2, "creationTimestamp": "2024-08-19T11:39:08Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.596a43b7-9dda-4586231bd", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "596a43b7-9dda-4580-806b-eab5aa6441c3" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:08Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:08Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-7hf85", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46 and status: Accepted 2024/08/19 11:39:42 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "9c3ff5b5-e8c9-49b4-b0ae-3bea220fe731", "resourceVersion": "36814", "generation": 2, "creationTimestamp": "2024-08-19T11:39:13Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.63a19307-5f52-461739f0f", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "63a19307-5f52-461e-85ed-02548941d2d5" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:13Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:14Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-blfpw", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 and status: Accepted 2024/08/19 11:39:42 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "ac63c8c2-57a3-4957-a19c-52e6acfa9413", "resourceVersion": "36866", "generation": 2, "creationTimestamp": "2024-08-19T11:39:19Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.d9cef209-e44d-4b8c39b0a", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "d9cef209-e44d-4b83-8554-1a891cfd040a" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-6g45h", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54 and status: Accepted 2024/08/19 11:40:02 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "747c7836-ea69-460f-821c-5b8dce19e1d8", "resourceVersion": "36754", "generation": 2, "creationTimestamp": "2024-08-19T11:39:08Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.596a43b7-9dda-4586231bd", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "596a43b7-9dda-4580-806b-eab5aa6441c3" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:08Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:08Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-7hf85", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46 and status: Accepted 2024/08/19 11:40:02 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "9c3ff5b5-e8c9-49b4-b0ae-3bea220fe731", "resourceVersion": "36814", "generation": 2, "creationTimestamp": "2024-08-19T11:39:13Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.63a19307-5f52-461739f0f", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "63a19307-5f52-461e-85ed-02548941d2d5" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:13Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:14Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-blfpw", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 and status: Accepted 2024/08/19 11:40:02 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "ac63c8c2-57a3-4957-a19c-52e6acfa9413", "resourceVersion": "36866", "generation": 2, "creationTimestamp": "2024-08-19T11:39:19Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.d9cef209-e44d-4b8c39b0a", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "d9cef209-e44d-4b83-8554-1a891cfd040a" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-6g45h", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54 phase: Completed DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46 phase: Completed DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 and status: Accepted 2024/08/19 11:40:22 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "ac63c8c2-57a3-4957-a19c-52e6acfa9413", "resourceVersion": "36866", "generation": 2, "creationTimestamp": "2024-08-19T11:39:19Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.d9cef209-e44d-4b8c39b0a", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "d9cef209-e44d-4b83-8554-1a891cfd040a" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-6g45h", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fhx54 phase: Completed DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-zwn46 phase: Completed DataUpload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 phase: Accepted DataUpload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8 and status: Accepted 2024/08/19 11:40:42 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-x8mc8", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "ac63c8c2-57a3-4957-a19c-52e6acfa9413", "resourceVersion": "36866", "generation": 2, "creationTimestamp": "2024-08-19T11:39:19Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "du-587b936d-abc6-4c82-a45f-f1e136f3f927.d9cef209-e44d-4b8c39b0a", "velero.io/backup-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/backup-uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "velero.io/pvc-uid": "d9cef209-e44d-4b83-8554-1a891cfd040a" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "587b936d-abc6-4c82-a45f-f1e136f3f927", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:39:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"587b936d-abc6-4c82-a45f-f1e136f3f927\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-6g45h", "storageClass": "ibmc-vpc-block-10iops-tier", "snapshotClass": "example-snapclass" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } backup phase: Completed STEP: Verify backup cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef has completed successfully @ 08/19/24 11:41:02.434 2024/08/19 11:41:02 Backup for case cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef succeeded STEP: Delete the appplication resources cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef @ 08/19/24 11:41:02.441 STEP: Cleanup Application for case cassandra-e2e @ 08/19/24 11:41:02.442 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/08/19 11:41:31 2024-08-19 11:41:03,797 p=24469 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:41:03,797 p=24469 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:41:04,024 p=24469 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:41:04,024 p=24469 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:41:04,256 p=24469 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:41:04,257 p=24469 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:41:04,273 p=24469 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:41:04,273 p=24469 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:41:04,557 p=24469 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:41:04,558 p=24469 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:41:04,584 p=24469 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:41:04,584 p=24469 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:41:04,601 p=24469 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:41:04,601 p=24469 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:41:04,611 p=24469 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:41:05,142 p=24469 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:41:05,142 p=24469 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:41:31,001 p=24469 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** 2024-08-19 11:41:31,001 p=24469 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:41:31,002 p=24469 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:41:31,217 p=24469 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:41:31,217 p=24469 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 STEP: Create restore cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef from backup cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef @ 08/19/24 11:41:31.259 2024/08/19 11:41:31 Wait until restore cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef completes restore phase: WaitingForPluginOperations DataDownload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-m682x phase: Accepted DataDownload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-m682x and status: Accepted 2024/08/19 11:41:51 { "kind": "DataDownload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-m682x", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "2a573784-c535-4ac7-8dc2-5541f58ee7a5", "resourceVersion": "38149", "generation": 2, "creationTimestamp": "2024-08-19T11:41:33Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "dd-0e9062bf-5522-4d9f-90d6-fbe1c23fddc4.596a43b7-9dda-458614f3b", "velero.io/restore-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/restore-uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Restore", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/restore-name": {}, "f:velero.io/restore-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"0e9062bf-5522-4d9f-90d6-fbe1c23fddc4\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:operationTimeout": {}, "f:snapshotID": {}, "f:sourceNamespace": {}, "f:targetVolume": { ".": {}, "f:namespace": {}, "f:pv": {}, "f:pvc": {} } }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "targetVolume": { "pvc": "cassandra-data-cassandra-0", "pv": "", "namespace": "test-oadp-440" }, "backupStorageLocation": "ts-dpa-1", "snapshotID": "363f4e851439438b01a4a9f8971d67df", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataDownload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-nw2zx phase: Accepted DataDownload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-nw2zx and status: Accepted 2024/08/19 11:41:51 { "kind": "DataDownload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-nw2zx", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "692f5292-b319-40d9-a2fe-c9e15decd496", "resourceVersion": "38156", "generation": 2, "creationTimestamp": "2024-08-19T11:41:33Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z", "velero.io/async-operation-id": "dd-0e9062bf-5522-4d9f-90d6-fbe1c23fddc4.63a19307-5f52-461ab41ce", "velero.io/restore-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/restore-uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Restore", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/restore-name": {}, "f:velero.io/restore-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"0e9062bf-5522-4d9f-90d6-fbe1c23fddc4\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:operationTimeout": {}, "f:snapshotID": {}, "f:sourceNamespace": {}, "f:targetVolume": { ".": {}, "f:namespace": {}, "f:pv": {}, "f:pvc": {} } }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "targetVolume": { "pvc": "cassandra-data-cassandra-1", "pv": "", "namespace": "test-oadp-440" }, "backupStorageLocation": "ts-dpa-1", "snapshotID": "e24efaab74322ba9996dace33c5b0d2c", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } DataDownload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-w9qqn phase: Accepted DataDownload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-w9qqn and status: Accepted 2024/08/19 11:41:51 { "kind": "DataDownload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-w9qqn", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "7a5e950d-93dd-4a2e-ac7f-f969c7d4b25a", "resourceVersion": "38163", "generation": 2, "creationTimestamp": "2024-08-19T11:41:33Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z", "velero.io/async-operation-id": "dd-0e9062bf-5522-4d9f-90d6-fbe1c23fddc4.d9cef209-e44d-4b872e22f", "velero.io/restore-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/restore-uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Restore", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {} } } }, { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/restore-name": {}, "f:velero.io/restore-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"0e9062bf-5522-4d9f-90d6-fbe1c23fddc4\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:operationTimeout": {}, "f:snapshotID": {}, "f:sourceNamespace": {}, "f:targetVolume": { ".": {}, "f:namespace": {}, "f:pv": {}, "f:pvc": {} } }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "targetVolume": { "pvc": "cassandra-data-cassandra-2", "pv": "", "namespace": "test-oadp-440" }, "backupStorageLocation": "ts-dpa-1", "snapshotID": "f9bcdc1d035d3092a6a71198c91b021e", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "progress": {} } } restore phase: WaitingForPluginOperations DataDownload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-m682x phase: Prepared DataDownload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-m682x and status: Prepared 2024/08/19 11:42:11 { "kind": "DataDownload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-m682x", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "2a573784-c535-4ac7-8dc2-5541f58ee7a5", "resourceVersion": "38453", "generation": 3, "creationTimestamp": "2024-08-19T11:41:33Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz", "velero.io/async-operation-id": "dd-0e9062bf-5522-4d9f-90d6-fbe1c23fddc4.596a43b7-9dda-458614f3b", "velero.io/restore-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/restore-uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Restore", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/restore-name": {}, "f:velero.io/restore-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"0e9062bf-5522-4d9f-90d6-fbe1c23fddc4\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:operationTimeout": {}, "f:snapshotID": {}, "f:sourceNamespace": {}, "f:targetVolume": { ".": {}, "f:namespace": {}, "f:pv": {}, "f:pvc": {} } }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:57Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:node": {}, "f:phase": {} } } } ] }, "spec": { "targetVolume": { "pvc": "cassandra-data-cassandra-0", "pv": "", "namespace": "test-oadp-440" }, "backupStorageLocation": "ts-dpa-1", "snapshotID": "363f4e851439438b01a4a9f8971d67df", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Prepared", "progress": {}, "node": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z" } } DataDownload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-nw2zx phase: InProgress DataDownload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-nw2zx and status: InProgress 2024/08/19 11:42:11 { "kind": "DataDownload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-nw2zx", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "692f5292-b319-40d9-a2fe-c9e15decd496", "resourceVersion": "38511", "generation": 5, "creationTimestamp": "2024-08-19T11:41:33Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z", "velero.io/async-operation-id": "dd-0e9062bf-5522-4d9f-90d6-fbe1c23fddc4.63a19307-5f52-461ab41ce", "velero.io/restore-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/restore-uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Restore", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/restore-name": {}, "f:velero.io/restore-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"0e9062bf-5522-4d9f-90d6-fbe1c23fddc4\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:operationTimeout": {}, "f:snapshotID": {}, "f:sourceNamespace": {}, "f:targetVolume": { ".": {}, "f:namespace": {}, "f:pv": {}, "f:pvc": {} } }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:42:06Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:node": {}, "f:phase": {}, "f:progress": { "f:bytesDone": {}, "f:totalBytes": {} }, "f:startTimestamp": {} } } } ] }, "spec": { "targetVolume": { "pvc": "cassandra-data-cassandra-1", "pv": "", "namespace": "test-oadp-440" }, "backupStorageLocation": "ts-dpa-1", "snapshotID": "e24efaab74322ba9996dace33c5b0d2c", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "InProgress", "startTimestamp": "2024-08-19T11:42:04Z", "progress": { "totalBytes": 67204443, "bytesDone": 67204443 }, "node": "ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz" } } DataDownload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-w9qqn phase: InProgress DataDownload Name: cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-w9qqn and status: InProgress 2024/08/19 11:42:11 { "kind": "DataDownload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-w9qqn", "generateName": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-", "namespace": "openshift-adp", "uid": "7a5e950d-93dd-4a2e-ac7f-f969c7d4b25a", "resourceVersion": "38465", "generation": 5, "creationTimestamp": "2024-08-19T11:41:33Z", "labels": { "velero.io/accepted-by": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z", "velero.io/async-operation-id": "dd-0e9062bf-5522-4d9f-90d6-fbe1c23fddc4.d9cef209-e44d-4b872e22f", "velero.io/restore-name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "velero.io/restore-uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Restore", "name": "cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef", "uid": "0e9062bf-5522-4d9f-90d6-fbe1c23fddc4", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "velero", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:41:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/restore-name": {}, "f:velero.io/restore-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"0e9062bf-5522-4d9f-90d6-fbe1c23fddc4\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:operationTimeout": {}, "f:snapshotID": {}, "f:sourceNamespace": {}, "f:targetVolume": { ".": {}, "f:namespace": {}, "f:pv": {}, "f:pvc": {} } }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-08-19T11:42:00Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:node": {}, "f:phase": {}, "f:progress": { "f:bytesDone": {}, "f:totalBytes": {} }, "f:startTimestamp": {} } } } ] }, "spec": { "targetVolume": { "pvc": "cassandra-data-cassandra-2", "pv": "", "namespace": "test-oadp-440" }, "backupStorageLocation": "ts-dpa-1", "snapshotID": "f9bcdc1d035d3092a6a71198c91b021e", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "InProgress", "startTimestamp": "2024-08-19T11:41:57Z", "progress": { "totalBytes": 67258661, "bytesDone": 67258661 }, "node": "ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z" } } restore phase: Finalizing DataDownload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-m682x phase: Completed DataDownload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-nw2zx phase: Completed DataDownload cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-w9qqn phase: Completed restore phase: Completed STEP: Validate the application after restore @ 08/19/24 11:42:51.424 STEP: Verify Application deployment for case cassandra-e2e @ 08/19/24 11:42:51.424 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. FAILED - RETRYING: [localhost]: Check pods status (30 retries left). FAILED - RETRYING: [localhost]: Check pods status (29 retries left). FAILED - RETRYING: [localhost]: Check pods status (28 retries left). FAILED - RETRYING: [localhost]: Check pods status (27 retries left). FAILED - RETRYING: [localhost]: Check pods status (26 retries left). FAILED - RETRYING: [localhost]: Check pods status (25 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (30 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (29 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (28 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (27 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (26 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (25 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (24 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (23 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (22 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (21 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (20 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (19 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (18 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (17 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (16 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (15 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (14 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (13 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (12 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (11 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (10 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (9 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (8 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (7 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (6 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (5 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (4 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (3 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (2 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (1 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** fatal: [localhost]: FAILED! => {"attempts": 30, "changed": true, "cmd": "oc --server https://api.ci-op-1nitxr4v-2f653.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:6443 --token sha256~Ogh3ROdN-IaL_oEEj2cEaCbsgkiz-BA4wMqVCkRRBgk -n test-oadp-440 exec -it cassandra-0 -- nodetool status", "delta": "0:00:06.423195", "end": "2024-08-19 11:49:12.514174", "msg": "", "rc": 0, "start": "2024-08-19 11:49:06.090979", "stderr": "Unable to use a TTY - input is not a terminal or the right kind of file", "stderr_lines": ["Unable to use a TTY - input is not a terminal or the right kind of file"], "stdout": "Datacenter: DC1\n===============\nStatus=Up/Down\n|/ State=Normal/Leaving/Joining/Moving\n-- Address Load Tokens Owns (effective) Host ID Rack\nUN 10.128.2.20 146.4 KiB 256 100.0% cd3fba23-7f21-46a1-8ec7-586c415dd562 Rack1\nUN 10.129.2.20 170.34 KiB 256 100.0% dd3ee7ac-41ec-4a11-96d5-e402537cd5eb Rack1\nDN 10.131.0.27 ? 256 100.0% ad39ed3c-27fe-48e9-9112-35559d3bae1c r1", "stdout_lines": ["Datacenter: DC1", "===============", "Status=Up/Down", "|/ State=Normal/Leaving/Joining/Moving", "-- Address Load Tokens Owns (effective) Host ID Rack", "UN 10.128.2.20 146.4 KiB 256 100.0% cd3fba23-7f21-46a1-8ec7-586c415dd562 Rack1", "UN 10.129.2.20 170.34 KiB 256 100.0% dd3ee7ac-41ec-4a11-96d5-e402537cd5eb Rack1", "DN 10.131.0.27 ? 256 100.0% ad39ed3c-27fe-48e9-9112-35559d3bae1c r1"]} PLAY RECAP ********************************************************************* localhost : ok=11  changed=3  unreachable=0 failed=1  skipped=9  rescued=0 ignored=0 2024/08/19 11:49:12 2024-08-19 11:42:52,807 p=24678 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:42:52,807 p=24678 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:42:53,050 p=24678 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:42:53,050 p=24678 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:42:53,289 p=24678 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:42:53,289 p=24678 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:42:53,304 p=24678 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:42:53,304 p=24678 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:42:53,598 p=24678 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:42:53,598 p=24678 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:42:53,625 p=24678 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:42:53,626 p=24678 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:42:53,644 p=24678 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:42:53,645 p=24678 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:42:53,655 p=24678 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:42:54,179 p=24678 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:42:54,180 p=24678 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:43:28,610 p=24678 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-08-19 11:43:28,611 p=24678 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:43:28,611 p=24678 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:49:12,534 p=24678 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-08-19 11:49:12,534 p=24678 u=1015750000 n=ansible | fatal: [localhost]: FAILED! => {"attempts": 30, "changed": true, "cmd": "oc --server https://api.ci-op-1nitxr4v-2f653.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:6443 --token sha256~Ogh3ROdN-IaL_oEEj2cEaCbsgkiz-BA4wMqVCkRRBgk -n test-oadp-440 exec -it cassandra-0 -- nodetool status", "delta": "0:00:06.423195", "end": "2024-08-19 11:49:12.514174", "msg": "", "rc": 0, "start": "2024-08-19 11:49:06.090979", "stderr": "Unable to use a TTY - input is not a terminal or the right kind of file", "stderr_lines": ["Unable to use a TTY - input is not a terminal or the right kind of file"], "stdout": "Datacenter: DC1\n===============\nStatus=Up/Down\n|/ State=Normal/Leaving/Joining/Moving\n-- Address Load Tokens Owns (effective) Host ID Rack\nUN 10.128.2.20 146.4 KiB 256 100.0% cd3fba23-7f21-46a1-8ec7-586c415dd562 Rack1\nUN 10.129.2.20 170.34 KiB 256 100.0% dd3ee7ac-41ec-4a11-96d5-e402537cd5eb Rack1\nDN 10.131.0.27 ? 256 100.0% ad39ed3c-27fe-48e9-9112-35559d3bae1c r1", "stdout_lines": ["Datacenter: DC1", "===============", "Status=Up/Down", "|/ State=Normal/Leaving/Joining/Moving", "-- Address Load Tokens Owns (effective) Host ID Rack", "UN 10.128.2.20 146.4 KiB 256 100.0% cd3fba23-7f21-46a1-8ec7-586c415dd562 Rack1", "UN 10.129.2.20 170.34 KiB 256 100.0% dd3ee7ac-41ec-4a11-96d5-e402537cd5eb Rack1", "DN 10.131.0.27 ? 256 100.0% ad39ed3c-27fe-48e9-9112-35559d3bae1c r1"]} 2024-08-19 11:49:12,535 p=24678 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:49:12,535 p=24678 u=1015750000 n=ansible | localhost : ok=11 changed=3 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0 Run the command: oc get event -n test-oadp-440 2024/08/19 11:49:12 LAST SEEN TYPE REASON OBJECT MESSAGE 6m29s Normal Scheduled pod/cassandra-0 Successfully assigned test-oadp-440/cassandra-0 to ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z 6m25s Normal SuccessfulAttachVolume pod/cassandra-0 AttachVolume.Attach succeeded for volume "pvc-dfc5c30d-9ede-4865-a726-35b42c466633" 6m24s Normal AddedInterface pod/cassandra-0 Add eth0 [10.129.2.20/23] from ovn-kubernetes 6m24s Normal Pulling pod/cassandra-0 Pulling image "quay.io/migqe/cassandra:multiarch" 6m24s Normal Pulled pod/cassandra-0 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 221ms (221ms including waiting) 6m24s Normal Created pod/cassandra-0 Created container cassandra 6m24s Normal Started pod/cassandra-0 Started container cassandra 6m59s Normal Scheduled pod/cassandra-1 Successfully assigned test-oadp-440/cassandra-1 to ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz 6m54s Normal AddedInterface pod/cassandra-1 Add eth0 [10.128.2.20/23] from ovn-kubernetes 5m58s Normal Pulling pod/cassandra-1 Pulling image "quay.io/migqe/cassandra:multiarch" 6m54s Normal Pulled pod/cassandra-1 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 352ms (352ms including waiting) 5m58s Normal Created pod/cassandra-1 Created container cassandra 5m58s Normal Started pod/cassandra-1 Started container cassandra 6m49s Normal Pulled pod/cassandra-1 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 197ms (197ms including waiting) 6m13s Warning BackOff pod/cassandra-1 Back-off restarting failed container cassandra in pod cassandra-1_test-oadp-440(ab5ccd2a-d0de-49ab-a71b-3459c8d3b297) 6m30s Normal Pulled pod/cassandra-1 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 207ms (207ms including waiting) 5m58s Normal Pulled pod/cassandra-1 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 219ms (219ms including waiting) 6m59s Normal Scheduled pod/cassandra-2 Successfully assigned test-oadp-440/cassandra-2 to ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z 6m54s Normal SuccessfulAttachVolume pod/cassandra-2 AttachVolume.Attach succeeded for volume "pvc-bf92cc56-d098-4eb4-8791-612185261686" 6m53s Normal AddedInterface pod/cassandra-2 Add eth0 [10.131.0.27/23] from ovn-kubernetes 110s Normal Pulling pod/cassandra-2 Pulling image "quay.io/migqe/cassandra:multiarch" 6m53s Normal Pulled pod/cassandra-2 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 221ms (221ms including waiting) 5m Normal Created pod/cassandra-2 Created container cassandra 5m Normal Started pod/cassandra-2 Started container cassandra 6m27s Normal Pulled pod/cassandra-2 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 181ms (181ms including waiting) 3m58s Warning BackOff pod/cassandra-2 Back-off restarting failed container cassandra in pod cassandra-2_test-oadp-440(78333688-4139-44c2-9c85-61e0609c6073) 5m48s Normal Pulled pod/cassandra-2 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 246ms (246ms including waiting) 5m Normal Pulled pod/cassandra-2 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 187ms (187ms including waiting) 3m46s Normal Pulled pod/cassandra-2 Successfully pulled image "quay.io/migqe/cassandra:multiarch" in 369ms (369ms including waiting) 7m39s Normal WaitForFirstConsumer persistentvolumeclaim/cassandra-data-cassandra-0 waiting for first consumer to be created before binding 7m7s Normal Provisioning persistentvolumeclaim/cassandra-data-cassandra-0 External provisioner is provisioning volume for claim "test-oadp-440/cassandra-data-cassandra-0" 7m7s Warning ProvisioningFailed persistentvolumeclaim/cassandra-data-cassandra-0 failed to provision volume with StorageClass "ibmc-vpc-block-10iops-tier": claim Selector is not supported 7m Normal ExternalProvisioning persistentvolumeclaim/cassandra-data-cassandra-0 Waiting for a volume to be created either by the external provisioner 'vpc.block.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 6m50s Warning FailedBinding persistentvolumeclaim/cassandra-data-cassandra-0 volume "pvc-dfc5c30d-9ede-4865-a726-35b42c466633" already bound to a different claim. 7m39s Normal WaitForFirstConsumer persistentvolumeclaim/cassandra-data-cassandra-1 waiting for first consumer to be created before binding 7m7s Normal Provisioning persistentvolumeclaim/cassandra-data-cassandra-1 External provisioner is provisioning volume for claim "test-oadp-440/cassandra-data-cassandra-1" 7m7s Warning ProvisioningFailed persistentvolumeclaim/cassandra-data-cassandra-1 failed to provision volume with StorageClass "ibmc-vpc-block-10iops-tier": claim Selector is not supported 7m15s Normal ExternalProvisioning persistentvolumeclaim/cassandra-data-cassandra-1 Waiting for a volume to be created either by the external provisioner 'vpc.block.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 7m4s Warning FailedBinding persistentvolumeclaim/cassandra-data-cassandra-1 volume "pvc-f28f9bfe-8d53-4dcb-a6d8-fa11b8d84416" already bound to a different claim. 7m39s Normal WaitForFirstConsumer persistentvolumeclaim/cassandra-data-cassandra-2 waiting for first consumer to be created before binding 7m15s Normal ExternalProvisioning persistentvolumeclaim/cassandra-data-cassandra-2 Waiting for a volume to be created either by the external provisioner 'vpc.block.csi.ibm.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 7m23s Normal Provisioning persistentvolumeclaim/cassandra-data-cassandra-2 External provisioner is provisioning volume for claim "test-oadp-440/cassandra-data-cassandra-2" 7m23s Warning ProvisioningFailed persistentvolumeclaim/cassandra-data-cassandra-2 failed to provision volume with StorageClass "ibmc-vpc-block-10iops-tier": claim Selector is not supported 7m10s Warning FailedBinding persistentvolumeclaim/cassandra-data-cassandra-2 volume "pvc-bf92cc56-d098-4eb4-8791-612185261686" already bound to a different claim. [FAILED] in [It] - /alabama/cspi/test_common/backup_restore_app_case.go:24 @ 08/19/24 11:49:12.673 2024/08/19 11:49:12 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 STEP: Get the failed spec name @ 08/19/24 11:49:12.673 2024/08/19 11:49:12 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application STEP: Create a folder for all must-gather files if it doesn't exists already @ 08/19/24 11:49:12.673 2024/08/19 11:49:12 The folder logs does not exists, creating new folder with the name: logs STEP: Create a folder for the failed spec if it doesn't exists already @ 08/19/24 11:49:12.673 2024/08/19 11:49:12 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application does not exists, creating new folder with the name: logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application STEP: Run must-gather because the spec failed @ 08/19/24 11:49:12.674 2024/08/19 11:49:12 [adm must-gather --dest-dir logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0] STEP: Find must-gather folder and rename it to a shorter more readable name @ 08/19/24 11:50:11.72 2024/08/19 11:50:11 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/08/19 11:50:40 2024-08-19 11:50:13,156 p=26000 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:50:13,156 p=26000 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:50:13,414 p=26000 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:50:13,414 p=26000 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:50:13,661 p=26000 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:50:13,661 p=26000 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:50:13,679 p=26000 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:50:13,679 p=26000 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:13,982 p=26000 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:50:13,982 p=26000 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:14,013 p=26000 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:50:14,013 p=26000 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:14,032 p=26000 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:50:14,033 p=26000 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:14,044 p=26000 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:50:14,600 p=26000 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:50:14,600 p=26000 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:40,500 p=26000 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** 2024-08-19 11:50:40,501 p=26000 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:50:40,501 p=26000 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:50:40,709 p=26000 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:50:40,709 p=26000 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/08/19 11:50:40 Cleaning setup resources for the backup 2024/08/19 11:50:40 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 11:50:40 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier 2024/08/19 11:50:40 Deleting VolumeSnapshotClass 'example-snapclass' • [FAILED] [923.797 seconds] [datamover] DataMover: Backup/Restore stateful application with CSI  [It] [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:130 [FAILED] Unexpected error: <*errors.Error | 0xc000e7c0c0>: Error during command execution: ansible-playbook error: one or more host failed Command executed: /usr/local/bin/ansible-playbook --extra-vars {"namespace":"test-oadp-440","use_role":"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra","with_validate":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml exit status 2 { context: "(DefaultExecute::Execute)", message: "Error during command execution: ansible-playbook error: one or more host failed\n\nCommand executed: /usr/local/bin/ansible-playbook --extra-vars {\"namespace\":\"test-oadp-440\",\"use_role\":\"/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra\",\"with_validate\":true} --connection local /alabama/cspi/sample-applications/ansible/main.yml\n\nexit status 2", wrappedErrors: nil, } occurred In [It] at: /alabama/cspi/test_common/backup_restore_app_case.go:24 @ 08/19/24 11:49:12.673 ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ [skip-disconnected] Restore hooks tests Successful Init hook [tc-id:OADP-164][interop][smoke] [bug-id:OADP-1076] MySQL app with Restic /alabama/cspi/e2e/hooks/restore_hooks.go:129 2024/08/19 11:50:41 Delete all downloadrequest cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-01bfc883-ac38-4284-b65f-dc11fab739ad cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-17338b1c-85b1-4ab0-9e74-968491d624f0 cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-5339e3e2-71e6-49b5-9b00-2b4c8934f24a cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-c546bf4b-ef42-4661-ae7e-1cd86c0bfeae cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-c6e2c284-953d-4e37-bae7-ec54b62c265b cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-d0f452f8-9258-4720-b93a-7f98eae1eecb cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-d299e463-5eaf-4fe0-9018-a5122d49721a cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-f30b3fb8-42f0-4c9a-a35a-0a6f3a5537a7 cassandra-e2e-1bb2536b-5e1f-11ef-906a-0a580a810aef-fea96955-0024-4988-b1f7-c337892b3154 mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-018d39b9-57c6-4322-b49a-e631a314fa1d mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-6027d121-07bd-4f64-af51-1b8d603b6d8c mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-8713aa49-e1d9-477d-809d-0186cdacfe76 mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-98495313-9322-4e9e-9e66-8db15762205c mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-992f3a72-2193-4ced-aa76-e0d7641fb474 mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-a569210c-0d50-4ac0-8102-96e095c92562 mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-ad292872-beaa-457d-874c-6c879ac04973 mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-ddeb259e-54a6-4445-8251-5d87c1dfd505 mysql-6fd6a3ea-5e1e-11ef-906a-0a580a810aef-e6616baa-66a6-491d-8137-9e9ab096fc02 STEP: Create DPA CR @ 08/19/24 11:50:42.778 2024/08/19 11:50:42 restic 2024/08/19 11:50:42 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "3f314bbd-e793-4244-a9e9-47d40c2b6614", "resourceVersion": "41770", "generation": 1, "creationTimestamp": "2024-08-19T11:50:42Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T11:50:42Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 08/19/24 11:50:42.821 2024/08/19 11:50:42 Waiting for velero pod to be running 2024/08/19 11:50:47 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 11:50:47.854 2024/08/19 11:50:47 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql-hooks-e2e @ 08/19/24 11:50:47.876 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-164] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). FAILED - RETRYING: [localhost]: Check pod status (28 retries left). FAILED - RETRYING: [localhost]: Check pod status (27 retries left). FAILED - RETRYING: [localhost]: Check pod status (26 retries left). FAILED - RETRYING: [localhost]: Check pod status (25 retries left). FAILED - RETRYING: [localhost]: Check pod status (24 retries left). FAILED - RETRYING: [localhost]: Check pod status (23 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] Pausing for 30 seconds TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=20  changed=10  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/08/19 11:52:22 2024-08-19 11:50:49,241 p=26209 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:50:49,241 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:50:49,470 p=26209 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:50:49,470 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:50:49,708 p=26209 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:50:49,708 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:50:49,723 p=26209 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:50:49,723 p=26209 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:50,010 p=26209 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:50:50,010 p=26209 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:50,035 p=26209 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:50:50,035 p=26209 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:50,053 p=26209 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:50:50,053 p=26209 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:50,063 p=26209 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:50:50,605 p=26209 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:50:50,605 p=26209 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:51,349 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-164] *** 2024-08-19 11:50:51,349 p=26209 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:50:51,349 p=26209 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:50:51,800 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-08-19 11:50:51,800 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:50:52,682 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-08-19 11:50:52,682 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:51:43,783 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-08-19 11:51:43,784 p=26209 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:51:44,342 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-08-19 11:51:44,342 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:51:49,905 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 11:51:49,905 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:51:50,480 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-08-19 11:51:50,480 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:51:51,345 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-08-19 11:51:51,345 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:51:51,935 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-08-19 11:51:51,935 p=26209 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:51:51,953 p=26209 u=1015750000 n=ansible | Pausing for 30 seconds 2024-08-19 11:52:21,975 p=26209 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** 2024-08-19 11:52:21,975 p=26209 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:22,046 p=26209 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:52:22,046 p=26209 u=1015750000 n=ansible | localhost : ok=20 changed=10 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 08/19/24 11:52:22.093 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 11:52:27 2024-08-19 11:52:23,497 p=26778 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:52:23,498 p=26778 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:52:23,744 p=26778 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:52:23,744 p=26778 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:52:23,973 p=26778 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:52:23,973 p=26778 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:52:23,988 p=26778 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:52:23,988 p=26778 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:24,270 p=26778 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:52:24,270 p=26778 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:24,296 p=26778 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:52:24,296 p=26778 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:24,314 p=26778 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:52:24,314 p=26778 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:24,324 p=26778 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:52:24,849 p=26778 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:52:24,849 p=26778 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:25,800 p=26778 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-08-19 11:52:25,800 p=26778 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:52:25,800 p=26778 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:26,153 p=26778 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 11:52:26,153 p=26778 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:52:26,622 p=26778 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-08-19 11:52:26,622 p=26778 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:52:27,198 p=26778 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-08-19 11:52:27,198 p=26778 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:52:27,210 p=26778 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:52:27,210 p=26778 u=1015750000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 11:52:27 ExtractTarGz: Create file /tmp/tempDir1550787918/world-db/world.sql 2024/08/19 11:52:27 2024/08/19 11:52:27 {{ } { } [{{ } {mysql-data test-oadp-164 e56f6f2a-928f-4412-a742-db4bbee230bf 42070 0 2024-08-19 11:50:52 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/selected-node:ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-08-19 11:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-08-19 11:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-08-19 11:51:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-08-19 11:51:05 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-e56f6f2a-928f-4412-a742-db4bbee230bf 0xc000a3bb10 0xc000a3bbb0 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} {{ } {mysql-data1 test-oadp-164 ffd11343-bba6-48c9-93f8-5efc5477d585 41994 0 2024-08-19 11:50:52 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/selected-node:ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-08-19 11:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-08-19 11:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-08-19 11:50:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-08-19 11:50:55 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-ffd11343-bba6-48c9-93f8-5efc5477d585 0xc000a3bf00 0xc000a3bf10 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }}]} STEP: Creating backup mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef @ 08/19/24 11:52:27.805 2024/08/19 11:52:27 Wait until backup mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef is completed backup phase: Completed 2024/08/19 11:52:47 Verify the PodVolumeBackup is completed successfully and BackupRepository type is matching with DPA.nodeAgent.uploaderType 2024/08/19 11:52:47 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql-data creationTimestamp: "2024-08-19T11:52:33Z" generateName: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef- generation: 4 labels: velero.io/backup-name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef velero.io/backup-uid: 864ea731-d923-40cd-a22e-d2468a061562 velero.io/pvc-uid: e56f6f2a-928f-4412-a742-db4bbee230bf managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"864ea731-d923-40cd-a22e-d2468a061562"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T11:52:33Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T11:52:36Z" name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef-hgzpt namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef uid: 864ea731-d923-40cd-a22e-d2468a061562 resourceVersion: "42593" uid: d26461b7-f58f-41ef-9566-d3a07b6282a0 spec: backupStorageLocation: ts-dpa-1 node: ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z pod: kind: Pod name: mysql-845bdd7d8d-8s9zm namespace: test-oadp-164 uid: c34f9dfe-1492-4074-a8a5-62f8c7a658ac repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-1nitxr4v-interopoadp/velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef/restic/test-oadp-164 tags: backup: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef backup-uid: 864ea731-d923-40cd-a22e-d2468a061562 ns: test-oadp-164 pod: mysql-845bdd7d8d-8s9zm pod-uid: c34f9dfe-1492-4074-a8a5-62f8c7a658ac pvc-uid: e56f6f2a-928f-4412-a742-db4bbee230bf volume: mysql-data uploaderType: restic volume: mysql-data status: completionTimestamp: "2024-08-19T11:52:36Z" path: /host_pods/c34f9dfe-1492-4074-a8a5-62f8c7a658ac/volumes/kubernetes.io~csi/pvc-e56f6f2a-928f-4412-a742-db4bbee230bf/mount phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 snapshotID: 95b22f1d startTimestamp: "2024-08-19T11:52:33Z" 2024/08/19 11:52:47 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql-data1 creationTimestamp: "2024-08-19T11:52:33Z" generateName: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef- generation: 4 labels: velero.io/backup-name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef velero.io/backup-uid: 864ea731-d923-40cd-a22e-d2468a061562 velero.io/pvc-uid: ffd11343-bba6-48c9-93f8-5efc5477d585 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"864ea731-d923-40cd-a22e-d2468a061562"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T11:52:33Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T11:52:42Z" name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef-xms4m namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef uid: 864ea731-d923-40cd-a22e-d2468a061562 resourceVersion: "42634" uid: 4447368e-e2a1-4911-bb08-cf8a3c30018a spec: backupStorageLocation: ts-dpa-1 node: ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z pod: kind: Pod name: mysql-845bdd7d8d-8s9zm namespace: test-oadp-164 uid: c34f9dfe-1492-4074-a8a5-62f8c7a658ac repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-1nitxr4v-interopoadp/velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef/restic/test-oadp-164 tags: backup: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef backup-uid: 864ea731-d923-40cd-a22e-d2468a061562 ns: test-oadp-164 pod: mysql-845bdd7d8d-8s9zm pod-uid: c34f9dfe-1492-4074-a8a5-62f8c7a658ac pvc-uid: ffd11343-bba6-48c9-93f8-5efc5477d585 volume: mysql-data1 uploaderType: restic volume: mysql-data1 status: completionTimestamp: "2024-08-19T11:52:42Z" path: /host_pods/c34f9dfe-1492-4074-a8a5-62f8c7a658ac/volumes/kubernetes.io~csi/pvc-ffd11343-bba6-48c9-93f8-5efc5477d585/mount phase: Completed progress: bytesDone: 105256269 totalBytes: 105256269 snapshotID: 557e2792 startTimestamp: "2024-08-19T11:52:38Z" STEP: Verify backup mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef has completed successfully @ 08/19/24 11:52:47.863 2024/08/19 11:52:47 Backup for case mysql-hooks-e2e succeeded STEP: Delete the appplication resources mysql-hooks-e2e @ 08/19/24 11:52:47.922 STEP: Cleanup Application for case mysql-hooks-e2e @ 08/19/24 11:52:47.922 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-164] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2024/08/19 11:53:01 2024-08-19 11:52:49,311 p=27086 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:52:49,311 p=27086 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:52:49,532 p=27086 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:52:49,532 p=27086 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:52:49,753 p=27086 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:52:49,753 p=27086 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:52:49,768 p=27086 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:52:49,768 p=27086 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:50,045 p=27086 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:52:50,046 p=27086 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:50,071 p=27086 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:52:50,072 p=27086 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:50,089 p=27086 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:52:50,089 p=27086 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:52:50,099 p=27086 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:52:50,614 p=27086 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:52:50,614 p=27086 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:01,380 p=27086 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-164] *** 2024-08-19 11:53:01,381 p=27086 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:53:01,381 p=27086 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:01,645 p=27086 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:53:01,645 p=27086 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 2024/08/19 11:53:01 Creating restore mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef for case mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef STEP: Create restore mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef from backup mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef @ 08/19/24 11:53:01.688 2024/08/19 11:53:01 Wait until restore mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef is complete restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: Finalizing restore phase: Completed 2024/08/19 11:53:51 Verify the PodVolumeBackup and PodVolumeRestore count is equal 2024/08/19 11:53:51 Verify the PodVolumeRestore is completed sucessfully and uploaderType is matching 2024/08/19 11:53:51 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-08-19T11:53:04Z" generateName: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef- generation: 5 labels: velero.io/pod-uid: 5b6a2b6d-6dbd-4e25-bf42-a83f2c14d297 velero.io/pvc-uid: c375c07d-1e55-4631-bc6e-6d9b1c1d3f00 velero.io/restore-name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef velero.io/restore-uid: 000bd2f7-ee26-4d95-a3ba-2662805eb0d3 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"000bd2f7-ee26-4d95-a3ba-2662805eb0d3"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T11:53:04Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T11:53:41Z" name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef-6dc8h namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef uid: 000bd2f7-ee26-4d95-a3ba-2662805eb0d3 resourceVersion: "43138" uid: 57167345-8136-4b57-8c58-7357d8dcbec9 spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-845bdd7d8d-8s9zm namespace: test-oadp-164 uid: 5b6a2b6d-6dbd-4e25-bf42-a83f2c14d297 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-1nitxr4v-interopoadp/velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef/restic/test-oadp-164 snapshotID: 557e2792 sourceNamespace: test-oadp-164 uploaderType: restic volume: mysql-data1 status: completionTimestamp: "2024-08-19T11:53:41Z" phase: Completed progress: bytesDone: 105256269 totalBytes: 105256269 startTimestamp: "2024-08-19T11:53:37Z" 2024/08/19 11:53:51 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-08-19T11:53:04Z" generateName: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef- generation: 5 labels: velero.io/pod-uid: 5b6a2b6d-6dbd-4e25-bf42-a83f2c14d297 velero.io/pvc-uid: 79a2d5e4-62b0-4c99-b8c2-a33a681090fe velero.io/restore-name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef velero.io/restore-uid: 000bd2f7-ee26-4d95-a3ba-2662805eb0d3 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"000bd2f7-ee26-4d95-a3ba-2662805eb0d3"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T11:53:04Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T11:53:34Z" name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef-cn4js namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aef uid: 000bd2f7-ee26-4d95-a3ba-2662805eb0d3 resourceVersion: "43105" uid: a5f7a3a5-4976-4591-86e5-b96a185ee0bb spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-845bdd7d8d-8s9zm namespace: test-oadp-164 uid: 5b6a2b6d-6dbd-4e25-bf42-a83f2c14d297 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-1nitxr4v-interopoadp/velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef/restic/test-oadp-164 snapshotID: 95b22f1d sourceNamespace: test-oadp-164 uploaderType: restic volume: mysql-data status: completionTimestamp: "2024-08-19T11:53:34Z" phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 startTimestamp: "2024-08-19T11:53:32Z" STEP: Verify restore mysql-hooks-e2e-42537db0-5e21-11ef-906a-0a580a810aefhas completed successfully @ 08/19/24 11:53:51.795 STEP: Verify Application restore @ 08/19/24 11:53:51.805 STEP: Verify Application deployment for case mysql-hooks-e2e @ 08/19/24 11:53:51.805 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 11:53:56 2024-08-19 11:53:53,193 p=27298 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:53:53,193 p=27298 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:53,421 p=27298 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:53:53,421 p=27298 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:53,646 p=27298 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:53:53,646 p=27298 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:53,660 p=27298 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:53:53,660 p=27298 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:53,935 p=27298 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:53:53,936 p=27298 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:53,961 p=27298 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:53:53,961 p=27298 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:53,979 p=27298 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:53:53,979 p=27298 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:53,989 p=27298 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:53:54,519 p=27298 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:53:54,520 p=27298 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:55,429 p=27298 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-08-19 11:53:55,430 p=27298 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:53:55,430 p=27298 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:55,719 p=27298 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 11:53:55,719 p=27298 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:56,161 p=27298 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-08-19 11:53:56,161 p=27298 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:56,750 p=27298 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-08-19 11:53:56,750 p=27298 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:56,762 p=27298 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:53:56,762 p=27298 u=1015750000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 11:53:56 stderr: ERROR 1049 (42000): Unknown database 'world' 2024/08/19 11:53:56 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/08/19 11:53:56 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-164] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2024/08/19 11:54:10 2024-08-19 11:53:58,364 p=27611 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:53:58,364 p=27611 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:58,597 p=27611 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:53:58,597 p=27611 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:58,823 p=27611 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:53:58,823 p=27611 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:53:58,839 p=27611 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:53:58,839 p=27611 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:59,119 p=27611 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:53:59,119 p=27611 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:59,146 p=27611 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:53:59,146 p=27611 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:59,163 p=27611 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:53:59,164 p=27611 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:53:59,173 p=27611 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:53:59,710 p=27611 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:53:59,710 p=27611 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:10,464 p=27611 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-164] *** 2024-08-19 11:54:10,464 p=27611 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:54:10,464 p=27611 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:54:10,697 p=27611 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:54:10,697 p=27611 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 2024/08/19 11:54:10 Cleaning setup resources for the backup • [209.921 seconds] ------------------------------ SSSSSSSSS ------------------------------ Incremental backup restore tests Incremental restore pod count [tc-id:OADP-165][interop] Todolist app with CSI - policy: update /alabama/cspi/e2e/incremental_restore/backup_restore_incremental.go:94 2024/08/19 11:54:10 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 08/19/24 11:54:10.8 2024/08/19 11:54:10 csi 2024/08/19 11:54:10 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "798a26ab-388b-43b1-babc-7d0b61744776", "resourceVersion": "43358", "generation": 1, "creationTimestamp": "2024-08-19T11:54:10Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T11:54:10Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "csi" ] } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 08/19/24 11:54:10.886 2024/08/19 11:54:10 Waiting for velero pod to be running 2024/08/19 11:54:10 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/08/19 11:54:10 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "798a26ab-388b-43b1-babc-7d0b61744776", "resourceVersion": "43358", "generation": 1, "creationTimestamp": "2024-08-19T11:54:10Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T11:54:10Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "csi" ] } }, "features": null }, "status": {} } STEP: Installing application for case todolist-backup @ 08/19/24 11:54:15.935 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check namespace todolist-mariadb-csi-policy-update] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Create namespace todolist-mariadb-csi-policy-update] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Ensure namespace todolist-mariadb-csi-policy-update is present] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Deploy todolist-mysql application] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check mysql pod status (30 retries left). FAILED - RETRYING: [localhost]: Check mysql pod status (29 retries left). FAILED - RETRYING: [localhost]: Check mysql pod status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Check todolist pod status (30 retries left). FAILED - RETRYING: [localhost]: Check todolist pod status (29 retries left). FAILED - RETRYING: [localhost]: Check todolist pod status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until service is ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Add additional items todo list] *** changed: [localhost] Pausing for 30 seconds TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait for 30 seconds] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=20  changed=8  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/08/19 11:55:15 2024-08-19 11:54:17,309 p=27820 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:54:17,309 p=27820 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:54:17,535 p=27820 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:54:17,535 p=27820 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:54:17,762 p=27820 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:54:17,763 p=27820 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:54:17,777 p=27820 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:54:17,777 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:18,071 p=27820 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:54:18,071 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:18,099 p=27820 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:54:18,099 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:18,120 p=27820 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:54:18,120 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:18,130 p=27820 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:54:18,653 p=27820 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:54:18,653 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:19,416 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check namespace todolist-mariadb-csi-policy-update] *** 2024-08-19 11:54:19,417 p=27820 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:54:19,417 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:19,850 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Create namespace todolist-mariadb-csi-policy-update] *** 2024-08-19 11:54:19,851 p=27820 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:54:20,408 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Ensure namespace todolist-mariadb-csi-policy-update is present] *** 2024-08-19 11:54:20,408 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:21,477 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Deploy todolist-mysql application] *** 2024-08-19 11:54:21,477 p=27820 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:54:32,696 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod status] *** 2024-08-19 11:54:32,697 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:43,987 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod status] *** 2024-08-19 11:54:43,988 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:54:44,280 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until service is ready for connections] *** 2024-08-19 11:54:44,280 p=27820 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:54:44,554 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** 2024-08-19 11:54:44,554 p=27820 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:54:44,934 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Add additional items todo list] *** 2024-08-19 11:54:44,934 p=27820 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:54:44,951 p=27820 u=1015750000 n=ansible | Pausing for 30 seconds 2024-08-19 11:55:14,983 p=27820 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait for 30 seconds] *** 2024-08-19 11:55:14,984 p=27820 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:15,134 p=27820 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:55:15,135 p=27820 u=1015750000 n=ansible | localhost : ok=20 changed=8 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Verify Application deployment @ 08/19/24 11:55:15.179 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=17  changed=5  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 11:55:22 2024-08-19 11:55:16,624 p=28297 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:55:16,624 p=28297 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:55:16,866 p=28297 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:55:16,867 p=28297 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:55:17,110 p=28297 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:55:17,110 p=28297 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:55:17,126 p=28297 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:55:17,126 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:17,423 p=28297 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:55:17,424 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:17,452 p=28297 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:55:17,453 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:17,471 p=28297 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:55:17,471 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:17,481 p=28297 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:55:18,027 p=28297 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:55:18,028 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:19,023 p=28297 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** 2024-08-19 11:55:19,024 p=28297 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:55:19,024 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:19,342 p=28297 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** 2024-08-19 11:55:19,343 p=28297 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:55:20,026 p=28297 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** 2024-08-19 11:55:20,026 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:20,379 p=28297 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** 2024-08-19 11:55:20,379 p=28297 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:55:21,303 p=28297 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** 2024-08-19 11:55:21,303 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:21,678 p=28297 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** 2024-08-19 11:55:21,678 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:21,958 p=28297 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** 2024-08-19 11:55:21,958 p=28297 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:55:21,971 p=28297 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:55:21,972 p=28297 u=1015750000 n=ansible | localhost : ok=17 changed=5 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 11:55:22.011 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/08/19 11:55:22 The 'openshift-storage' namespace does not exist 2024/08/19 11:55:22 Using default CSI driver based on infrastructure: vpc.block.csi.ibm.io 2024/08/19 11:55:22 Snapclass 'example-snapclass' doesn't exist, creating 2024/08/19 11:55:22 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 11:55:22 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier 2024/08/19 11:55:22 {{ } { } [{{ } {mysql todolist-mariadb-csi-policy-update f311ddec-e688-4b16-aebd-dab995bf6f86 43604 0 2024-08-19 11:54:21 +0000 UTC map[app:mysql] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/selected-node:ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-08-19 11:54:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-08-19 11:54:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-08-19 11:54:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-08-19 11:54:23 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{1073741824 0} {} 1Gi BinarySI}]} pvc-f311ddec-e688-4b16-aebd-dab995bf6f86 0xc000738cd0 0xc000738ce0 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }}]} STEP: Creating backup todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef @ 08/19/24 11:55:22.192 2024/08/19 11:55:22 Wait until backup todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef is completed backup phase: WaitingForPluginOperations backup phase: Completed 2024/08/19 11:56:02 Verify the PVC and VSC count is matching 2024/08/19 11:56:02 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2024/08/19 11:56:02 Run velero describe on the backup 2024/08/19 11:56:02 [./velero describe backup todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef -n openshift-adp --details --insecure-skip-tls-verify] 2024/08/19 11:56:02 Exec stderr: "" 2024/08/19 11:56:02 Name: todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.7+4510e9c velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: Completed Namespaces: Included: todolist-mariadb-csi-policy-update Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-08-19 11:55:22 +0000 UTC Completed: 2024-08-19 11:55:51 +0000 UTC Expiration: 2024-09-18 11:55:22 +0000 UTC Total items to be backed up: 64 Items backed up: 64 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io todolist-mariadb-csi-policy-update/velero-mysql-8hw2b: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: todolist-mariadb-csi-policy-update/velero-mysql-8hw2b/2024-08-19T11:55:28Z Items to Update: volumesnapshots.snapshot.storage.k8s.io todolist-mariadb-csi-policy-update/velero-mysql-8hw2b volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-0ca33133-b247-4df2-80cb-9b0546206060 Phase: Completed Created: 2024-08-19 11:55:28 +0000 UTC Started: 2024-08-19 11:55:28 +0000 UTC Updated: 2024-08-19 11:55:50 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist apps/v1/ReplicaSet: - todolist-mariadb-csi-policy-update/mysql-849798ff64 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b authorization.openshift.io/v1/RoleBinding: - todolist-mariadb-csi-policy-update/admin - todolist-mariadb-csi-policy-update/system:deployers - todolist-mariadb-csi-policy-update/system:image-builders - todolist-mariadb-csi-policy-update/system:image-pullers discovery.k8s.io/v1/EndpointSlice: - todolist-mariadb-csi-policy-update/mysql-lj7b2 - todolist-mariadb-csi-policy-update/todolist-b75c2 operators.coreos.com/v1alpha1/ClusterServiceVersion: - todolist-mariadb-csi-policy-update/volsync-product.v0.10.0 rbac.authorization.k8s.io/v1/RoleBinding: - todolist-mariadb-csi-policy-update/admin - todolist-mariadb-csi-policy-update/system:deployers - todolist-mariadb-csi-policy-update/system:image-builders - todolist-mariadb-csi-policy-update/system:image-pullers route.openshift.io/v1/Route: - todolist-mariadb-csi-policy-update/todolist-route security.openshift.io/v1/SecurityContextConstraints: - todolist-mariadb-csi-policy-update-scc snapshot.storage.k8s.io/v1/VolumeSnapshot: - todolist-mariadb-csi-policy-update/velero-mysql-8hw2b snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-0ca33133-b247-4df2-80cb-9b0546206060 v1/ConfigMap: - todolist-mariadb-csi-policy-update/kube-root-ca.crt - todolist-mariadb-csi-policy-update/openshift-service-ca.crt v1/Endpoints: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist v1/Event: - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f2242437afc - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f236fed793a - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f239ead3608 - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f23a0e1c05b - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f23a5e0084f - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f23a701d213 - todolist-mariadb-csi-policy-update/mysql-849798ff64.17ed1f218bd87412 - todolist-mariadb-csi-policy-update/mysql.17ed1f2184f285b7 - todolist-mariadb-csi-policy-update/mysql.17ed1f218a57daba - todolist-mariadb-csi-policy-update/mysql.17ed1f218f8c04e6 - todolist-mariadb-csi-policy-update/mysql.17ed1f218f9ee452 - todolist-mariadb-csi-policy-update/mysql.17ed1f222541fd6e - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f2193603645 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f21ae044de8 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f21b04ea42f - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f22deb018bc - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f22e61772fc - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f22e75bd77a - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f254ecdc71b - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f2621d4a16d - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f26298b0f1b - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f262aaea97a - todolist-mariadb-csi-policy-update/todolist-fdc764b8b.17ed1f21915ab6b8 - todolist-mariadb-csi-policy-update/todolist.17ed1f218f800456 v1/Namespace: - todolist-mariadb-csi-policy-update v1/PersistentVolume: - pvc-f311ddec-e688-4b16-aebd-dab995bf6f86 v1/PersistentVolumeClaim: - todolist-mariadb-csi-policy-update/mysql v1/Pod: - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn v1/Secret: - todolist-mariadb-csi-policy-update/builder-dockercfg-bc9tt - todolist-mariadb-csi-policy-update/default-dockercfg-nqj4h - todolist-mariadb-csi-policy-update/deployer-dockercfg-cncnf - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa-dockercfg-jsb2h v1/Service: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist v1/ServiceAccount: - todolist-mariadb-csi-policy-update/builder - todolist-mariadb-csi-policy-update/default - todolist-mariadb-csi-policy-update/deployer - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa Backup Volumes: Velero-Native Snapshots: CSI Snapshots: todolist-mariadb-csi-policy-update/mysql: Snapshot: Operation ID: todolist-mariadb-csi-policy-update/velero-mysql-8hw2b/2024-08-19T11:55:28Z Snapshot Content Name: snapcontent-0ca33133-b247-4df2-80cb-9b0546206060 Storage Snapshot ID: r014-aad2455d-c2ce-41d9-a567-0f40daebcefa Snapshot Size (bytes): 10737418240 CSI Driver: vpc.block.csi.ibm.io Pod Volume Backups: HooksAttempted: 0 HooksFailed: 0 STEP: Verify backup todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef has completed successfully @ 08/19/24 11:56:02.942 2024/08/19 11:56:03 Backup for case todolist-backup succeeded STEP: Scale application @ 08/19/24 11:56:03.015 2024/08/19 11:56:03 Scaling deployment 'todolist' to 2 replicas 2024/08/19 11:56:03 Deployment updated successfully 2024/08/19 11:56:03 number of running pods: 1 2024/08/19 11:56:08 number of running pods: 1 2024/08/19 11:56:13 number of running pods: 1 2024/08/19 11:56:18 number of running pods: 1 2024/08/19 11:56:23 Application reached target number of replicas: 2 STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 11:56:23.102 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/08/19 11:56:23 The 'openshift-storage' namespace does not exist 2024/08/19 11:56:23 Using default CSI driver based on infrastructure: vpc.block.csi.ibm.io 2024/08/19 11:56:23 Snapclass 'example-snapclass' already exists, skip creating 2024/08/19 11:56:23 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 11:56:23 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier 2024/08/19 11:56:23 {{ } { } [{{ } {mysql todolist-mariadb-csi-policy-update f311ddec-e688-4b16-aebd-dab995bf6f86 44111 0 2024-08-19 11:54:21 +0000 UTC map[app:mysql] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/selected-node:ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-08-19 11:54:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-08-19 11:54:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-08-19 11:54:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-08-19 11:54:23 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{1073741824 0} {} 1Gi BinarySI}]} pvc-f311ddec-e688-4b16-aebd-dab995bf6f86 0xc0013c9470 0xc0013c9480 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }}]} STEP: Creating backup todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef @ 08/19/24 11:56:23.274 2024/08/19 11:56:23 Wait until backup todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef is completed backup phase: WaitingForPluginOperations backup phase: Completed 2024/08/19 11:57:03 Verify the PVC and VSC count is matching 2024/08/19 11:57:03 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2024/08/19 11:57:03 Run velero describe on the backup 2024/08/19 11:57:03 [./velero describe backup todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef -n openshift-adp --details --insecure-skip-tls-verify] 2024/08/19 11:57:04 Exec stderr: "" 2024/08/19 11:57:04 Name: todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.7+4510e9c velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: Completed Namespaces: Included: todolist-mariadb-csi-policy-update Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-08-19 11:56:23 +0000 UTC Completed: 2024-08-19 11:56:51 +0000 UTC Expiration: 2024-09-18 11:56:23 +0000 UTC Total items to be backed up: 80 Items backed up: 80 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io todolist-mariadb-csi-policy-update/velero-mysql-d544m: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: todolist-mariadb-csi-policy-update/velero-mysql-d544m/2024-08-19T11:56:29Z Items to Update: volumesnapshots.snapshot.storage.k8s.io todolist-mariadb-csi-policy-update/velero-mysql-d544m volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-a4806551-7e49-4b59-9d2f-1924895ad52f Phase: Completed Created: 2024-08-19 11:56:29 +0000 UTC Started: 2024-08-19 11:56:29 +0000 UTC Updated: 2024-08-19 11:56:50 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist apps/v1/ReplicaSet: - todolist-mariadb-csi-policy-update/mysql-849798ff64 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b authorization.openshift.io/v1/RoleBinding: - todolist-mariadb-csi-policy-update/admin - todolist-mariadb-csi-policy-update/system:deployers - todolist-mariadb-csi-policy-update/system:image-builders - todolist-mariadb-csi-policy-update/system:image-pullers discovery.k8s.io/v1/EndpointSlice: - todolist-mariadb-csi-policy-update/mysql-lj7b2 - todolist-mariadb-csi-policy-update/todolist-b75c2 operators.coreos.com/v1alpha1/ClusterServiceVersion: - todolist-mariadb-csi-policy-update/volsync-product.v0.10.0 rbac.authorization.k8s.io/v1/RoleBinding: - todolist-mariadb-csi-policy-update/admin - todolist-mariadb-csi-policy-update/system:deployers - todolist-mariadb-csi-policy-update/system:image-builders - todolist-mariadb-csi-policy-update/system:image-pullers route.openshift.io/v1/Route: - todolist-mariadb-csi-policy-update/todolist-route security.openshift.io/v1/SecurityContextConstraints: - todolist-mariadb-csi-policy-update-scc snapshot.storage.k8s.io/v1/VolumeSnapshot: - todolist-mariadb-csi-policy-update/velero-mysql-d544m snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-a4806551-7e49-4b59-9d2f-1924895ad52f v1/ConfigMap: - todolist-mariadb-csi-policy-update/kube-root-ca.crt - todolist-mariadb-csi-policy-update/openshift-service-ca.crt v1/Endpoints: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist v1/Event: - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f2242437afc - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f236fed793a - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f239ead3608 - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f23a0e1c05b - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f23a5e0084f - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv.17ed1f23a701d213 - todolist-mariadb-csi-policy-update/mysql-849798ff64.17ed1f218bd87412 - todolist-mariadb-csi-policy-update/mysql.17ed1f2184f285b7 - todolist-mariadb-csi-policy-update/mysql.17ed1f218a57daba - todolist-mariadb-csi-policy-update/mysql.17ed1f218f8c04e6 - todolist-mariadb-csi-policy-update/mysql.17ed1f218f9ee452 - todolist-mariadb-csi-policy-update/mysql.17ed1f222541fd6e - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f2193603645 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f21ae044de8 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f21b04ea42f - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f22deb018bc - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f22e61772fc - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f22e75bd77a - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f254ecdc71b - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f2621d4a16d - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f26298b0f1b - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn.17ed1f262aaea97a - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f394049e6a1 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f395c415022 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f395e8b38bb - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f3aa2af7d89 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f3aaa77ad01 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f3aab767031 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f3d2917b5e4 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f3d64e951f7 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f3d6d6f8f4e - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99.17ed1f3d6eba14b3 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b.17ed1f21915ab6b8 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b.17ed1f393f4e4556 - todolist-mariadb-csi-policy-update/todolist.17ed1f218f800456 - todolist-mariadb-csi-policy-update/todolist.17ed1f393d4cbffc - todolist-mariadb-csi-policy-update/velero-mysql-8hw2b.17ed1f3014f99d7c - todolist-mariadb-csi-policy-update/velero-mysql-8hw2b.17ed1f30d0a6a93f - todolist-mariadb-csi-policy-update/velero-mysql-8hw2b.17ed1f355bd945c5 v1/Namespace: - todolist-mariadb-csi-policy-update v1/PersistentVolume: - pvc-f311ddec-e688-4b16-aebd-dab995bf6f86 v1/PersistentVolumeClaim: - todolist-mariadb-csi-policy-update/mysql v1/Pod: - todolist-mariadb-csi-policy-update/mysql-849798ff64-mw2bv - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-8swwn - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-c6x99 v1/Secret: - todolist-mariadb-csi-policy-update/builder-dockercfg-bc9tt - todolist-mariadb-csi-policy-update/default-dockercfg-nqj4h - todolist-mariadb-csi-policy-update/deployer-dockercfg-cncnf - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa-dockercfg-jsb2h v1/Service: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist v1/ServiceAccount: - todolist-mariadb-csi-policy-update/builder - todolist-mariadb-csi-policy-update/default - todolist-mariadb-csi-policy-update/deployer - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa Backup Volumes: Velero-Native Snapshots: CSI Snapshots: todolist-mariadb-csi-policy-update/mysql: Snapshot: Operation ID: todolist-mariadb-csi-policy-update/velero-mysql-d544m/2024-08-19T11:56:29Z Snapshot Content Name: snapcontent-a4806551-7e49-4b59-9d2f-1924895ad52f Storage Snapshot ID: r014-ba9c2c49-f454-4786-94e4-61739da4f78a Snapshot Size (bytes): 10737418240 CSI Driver: vpc.block.csi.ibm.io Pod Volume Backups: HooksAttempted: 0 HooksFailed: 0 STEP: Verify backup todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef has completed successfully @ 08/19/24 11:57:04.247 2024/08/19 11:57:04 Backup for case todolist-backup succeeded STEP: Cleanup application and restore 1st backup @ 08/19/24 11:57:04.376 STEP: Delete the appplication resources todolist-backup @ 08/19/24 11:57:04.376 STEP: Cleanup Application for case todolist-backup @ 08/19/24 11:57:04.376 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove namespace todolist-mariadb-csi-policy-update] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove todolist-mariadb-csi-policy-update SCC] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=12  changed=5  unreachable=0 failed=0 skipped=17  rescued=0 ignored=0 2024/08/19 11:57:24 2024-08-19 11:57:05,752 p=28668 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:57:05,752 p=28668 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:57:05,980 p=28668 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:57:05,981 p=28668 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:57:06,209 p=28668 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:57:06,210 p=28668 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:57:06,224 p=28668 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:57:06,224 p=28668 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:57:06,500 p=28668 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:57:06,500 p=28668 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:57:06,526 p=28668 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:57:06,526 p=28668 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:57:06,543 p=28668 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:57:06,544 p=28668 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:57:06,554 p=28668 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:57:07,065 p=28668 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:57:07,065 p=28668 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:57:22,962 p=28668 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove namespace todolist-mariadb-csi-policy-update] *** 2024-08-19 11:57:22,963 p=28668 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:57:22,963 p=28668 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:57:23,888 p=28668 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove todolist-mariadb-csi-policy-update SCC] *** 2024-08-19 11:57:23,888 p=28668 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:57:24,204 p=28668 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:57:24,204 p=28668 u=1015750000 n=ansible | localhost : ok=12 changed=5 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0 2024/08/19 11:57:24 Creating restore todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef for case todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef STEP: Create restore todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef from backup todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef @ 08/19/24 11:57:24.252 2024/08/19 11:57:24 Wait until restore todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef is complete restore phase: Finalizing restore phase: Finalizing restore phase: Completed STEP: Verify restore todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aefhas completed successfully @ 08/19/24 11:57:54.315 STEP: Verify Application restore @ 08/19/24 11:57:54.324 STEP: Verify Application deployment for case todolist-backup @ 08/19/24 11:57:54.324 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. FAILED - RETRYING: [localhost]: Check mysql pod is running (30 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check todolist pod is running (30 retries left). FAILED - RETRYING: [localhost]: Check todolist pod is running (29 retries left). FAILED - RETRYING: [localhost]: Check todolist pod is running (28 retries left). FAILED - RETRYING: [localhost]: Check todolist pod is running (27 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=17  changed=5  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 11:58:28 2024-08-19 11:57:55,697 p=28896 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:57:55,697 p=28896 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:57:55,933 p=28896 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:57:55,934 p=28896 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:57:56,169 p=28896 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:57:56,169 p=28896 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:57:56,183 p=28896 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:57:56,184 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:57:56,459 p=28896 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:57:56,459 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:57:56,485 p=28896 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:57:56,485 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:57:56,502 p=28896 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:57:56,502 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:57:56,514 p=28896 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:57:57,031 p=28896 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:57:57,031 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:03,477 p=28896 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** 2024-08-19 11:58:03,478 p=28896 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:58:03,478 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:03,776 p=28896 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** 2024-08-19 11:58:03,776 p=28896 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:26,507 p=28896 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** 2024-08-19 11:58:26,507 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:26,810 p=28896 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** 2024-08-19 11:58:26,810 p=28896 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:27,750 p=28896 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** 2024-08-19 11:58:27,750 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:28,142 p=28896 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** 2024-08-19 11:58:28,142 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:28,416 p=28896 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** 2024-08-19 11:58:28,416 p=28896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:28,428 p=28896 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:58:28,429 p=28896 u=1015750000 n=ansible | localhost : ok=17 changed=5 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 11:58:28 Application reached target number of replicas: 1 STEP: Restore 2nd backup with existingRessourcePolicy: update @ 08/19/24 11:58:28.477 2024/08/19 11:58:28 Creating restore todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef for case todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef STEP: Create restore todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef from backup todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef @ 08/19/24 11:58:28.477 2024/08/19 11:58:28 Wait until restore todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef is complete restore phase: Completed STEP: Verify restore todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aefhas completed successfully @ 08/19/24 11:58:38.516 STEP: Verify Application restore @ 08/19/24 11:58:38.524 STEP: Verify Application deployment for case todolist-backup @ 08/19/24 11:58:38.524 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=17  changed=5  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 11:58:45 2024-08-19 11:58:39,913 p=29293 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:58:39,914 p=29293 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:40,153 p=29293 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:58:40,153 p=29293 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:40,393 p=29293 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:58:40,393 p=29293 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:40,408 p=29293 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:58:40,408 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:40,715 p=29293 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:58:40,715 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:40,744 p=29293 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:58:40,744 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:40,762 p=29293 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:58:40,762 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:40,772 p=29293 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:58:41,332 p=29293 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:58:41,332 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:42,237 p=29293 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** 2024-08-19 11:58:42,237 p=29293 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:58:42,237 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:42,521 p=29293 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** 2024-08-19 11:58:42,521 p=29293 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:43,137 p=29293 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** 2024-08-19 11:58:43,138 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:43,417 p=29293 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** 2024-08-19 11:58:43,418 p=29293 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:44,337 p=29293 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** 2024-08-19 11:58:44,338 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:44,718 p=29293 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** 2024-08-19 11:58:44,718 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:44,991 p=29293 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** 2024-08-19 11:58:44,992 p=29293 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:45,004 p=29293 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:58:45,004 p=29293 u=1015750000 n=ansible | localhost : ok=17 changed=5 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 11:58:45 Application reached target number of replicas: 2 2024/08/19 11:58:45 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/08/19 11:58:45 Cleaning setup resources for the backup 2024/08/19 11:58:45 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 11:58:45 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier 2024/08/19 11:58:45 Deleting VolumeSnapshotClass 'example-snapclass' 2024/08/19 11:58:45 Cleaning setup resources for the backup 2024/08/19 11:58:45 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 11:58:45 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier 2024/08/19 11:58:45 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove namespace todolist-mariadb-csi-policy-update] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove todolist-mariadb-csi-policy-update SCC] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=12  changed=5  unreachable=0 failed=0 skipped=17  rescued=0 ignored=0 2024/08/19 11:59:05 2024-08-19 11:58:46,495 p=29625 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:58:46,495 p=29625 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:46,719 p=29625 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:58:46,719 p=29625 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:46,942 p=29625 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:58:46,943 p=29625 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:58:46,957 p=29625 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:58:46,957 p=29625 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:47,244 p=29625 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:58:47,245 p=29625 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:47,271 p=29625 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:58:47,272 p=29625 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:47,289 p=29625 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:58:47,289 p=29625 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:58:47,299 p=29625 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:58:47,823 p=29625 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:58:47,823 p=29625 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:59:03,688 p=29625 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove namespace todolist-mariadb-csi-policy-update] *** 2024-08-19 11:59:03,688 p=29625 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:59:03,688 p=29625 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:04,750 p=29625 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove todolist-mariadb-csi-policy-update SCC] *** 2024-08-19 11:59:04,750 p=29625 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:05,047 p=29625 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 11:59:05,048 p=29625 u=1015750000 n=ansible | localhost : ok=12 changed=5 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0 • [294.359 seconds] ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ Backup restore tests Application backup [tc-id:OADP-371] [bug-id:OADP-1077] [interop] [smoke] MySQL application with Restic [mr-check] /alabama/cspi/e2e/app_backup/backup_restore.go:48 2024/08/19 11:59:05 Delete all downloadrequest todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef-0f89befd-92f8-429a-80e1-e13cbaa1ff0c todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef-25818f29-0356-45de-a2d6-46af7f09d2a8 todolist-backup-0e25bcfe-5e22-11ef-906a-0a580a810aef-9139dcd8-ae09-40a9-be7a-0c7fcb92d431 todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef-3eb3d0bd-a3c5-4776-be5c-3536222e0772 todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef-774c7ee5-f93f-4b2d-bac1-473e401e69eb todolist-backup-e9bc01b1-5e21-11ef-906a-0a580a810aef-d0228996-7925-4ad7-95a1-8f782b632972 STEP: Create DPA CR @ 08/19/24 11:59:05.555 2024/08/19 11:59:05 restic 2024/08/19 11:59:05 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "52e866f6-8473-4145-9f09-acbdc9212267", "resourceVersion": "45906", "generation": 1, "creationTimestamp": "2024-08-19T11:59:05Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T11:59:05Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 08/19/24 11:59:05.613 2024/08/19 11:59:05 Waiting for velero pod to be running 2024/08/19 11:59:05 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/08/19 11:59:05 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "52e866f6-8473-4145-9f09-acbdc9212267", "resourceVersion": "45906", "generation": 1, "creationTimestamp": "2024-08-19T11:59:05Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T11:59:05Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 11:59:10.644 2024/08/19 11:59:10 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 08/19/24 11:59:10.667 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-1077] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). FAILED - RETRYING: [localhost]: Check pod status (28 retries left). FAILED - RETRYING: [localhost]: Check pod status (27 retries left). FAILED - RETRYING: [localhost]: Check pod status (26 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] Pausing for 30 seconds TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=20  changed=10  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/08/19 12:00:22 2024-08-19 11:59:12,103 p=29853 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 11:59:12,103 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:12,344 p=29853 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 11:59:12,345 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:12,574 p=29853 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 11:59:12,575 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:12,589 p=29853 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:59:12,590 p=29853 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:59:12,868 p=29853 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 11:59:12,869 p=29853 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:59:12,897 p=29853 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 11:59:12,897 p=29853 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:59:12,916 p=29853 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 11:59:12,917 p=29853 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:59:12,928 p=29853 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 11:59:13,457 p=29853 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 11:59:13,457 p=29853 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:59:14,205 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-1077] *** 2024-08-19 11:59:14,205 p=29853 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 11:59:14,205 p=29853 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:59:14,606 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-08-19 11:59:14,606 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:15,627 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-08-19 11:59:15,627 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:44,197 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-08-19 11:59:44,197 p=29853 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 11:59:44,780 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-08-19 11:59:44,780 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:50,410 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 11:59:50,410 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:50,963 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-08-19 11:59:50,963 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:51,802 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-08-19 11:59:51,803 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:52,376 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-08-19 11:59:52,377 p=29853 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 11:59:52,394 p=29853 u=1015750000 n=ansible | Pausing for 30 seconds 2024-08-19 12:00:22,412 p=29853 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** 2024-08-19 12:00:22,412 p=29853 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:22,491 p=29853 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:00:22,491 p=29853 u=1015750000 n=ansible | localhost : ok=20 changed=10 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 08/19/24 12:00:22.538 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 12:00:28 2024-08-19 12:00:24,229 p=30379 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:00:24,229 p=30379 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:00:24,500 p=30379 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:00:24,500 p=30379 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:00:24,745 p=30379 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:00:24,745 p=30379 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:00:24,761 p=30379 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:00:24,761 p=30379 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:25,079 p=30379 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:00:25,079 p=30379 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:25,125 p=30379 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:00:25,126 p=30379 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:25,155 p=30379 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:00:25,156 p=30379 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:25,172 p=30379 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:00:25,862 p=30379 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:00:25,862 p=30379 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:26,846 p=30379 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-08-19 12:00:26,846 p=30379 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:00:26,846 p=30379 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:27,152 p=30379 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 12:00:27,152 p=30379 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:00:27,604 p=30379 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-08-19 12:00:27,604 p=30379 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:00:28,202 p=30379 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-08-19 12:00:28,202 p=30379 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:00:28,214 p=30379 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:00:28,214 p=30379 u=1015750000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 12:00:28 {{ } { } [{{ } {mysql-data test-oadp-1077 90387d99-86bf-4a8d-b870-a6684c85891a 46151 0 2024-08-19 11:59:15 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/selected-node:ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-08-19 11:59:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-08-19 11:59:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-08-19 11:59:19 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-08-19 11:59:19 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-90387d99-86bf-4a8d-b870-a6684c85891a 0xc0004c3960 0xc0004c3a50 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} {{ } {mysql-data1 test-oadp-1077 67c62e5c-b608-403f-808c-00abb36bba59 46206 0 2024-08-19 11:59:15 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/selected-node:ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-08-19 11:59:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-08-19 11:59:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-08-19 11:59:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-08-19 11:59:29 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-67c62e5c-b608-403f-808c-00abb36bba59 0xc000944090 0xc0009440a0 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }}]} STEP: Creating backup mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef @ 08/19/24 12:00:28.265 2024/08/19 12:00:28 Wait until backup mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef is completed backup phase: Completed 2024/08/19 12:00:48 Verify the PodVolumeBackup is completed successfully and BackupRepository type is matching with DPA.nodeAgent.uploaderType 2024/08/19 12:00:48 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql-data creationTimestamp: "2024-08-19T12:00:33Z" generateName: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef- generation: 4 labels: velero.io/backup-name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef velero.io/backup-uid: 09c11574-cdac-417f-b4bf-24474f5eb779 velero.io/pvc-uid: 90387d99-86bf-4a8d-b870-a6684c85891a managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"09c11574-cdac-417f-b4bf-24474f5eb779"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T12:00:33Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T12:00:36Z" name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef-8987l namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef uid: 09c11574-cdac-417f-b4bf-24474f5eb779 resourceVersion: "46659" uid: b73a57bf-1013-4867-86a7-172adeaf1bff spec: backupStorageLocation: ts-dpa-1 node: ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z pod: kind: Pod name: mysql-845bdd7d8d-65hwv namespace: test-oadp-1077 uid: f5f08769-a69e-4147-ad6e-7cbb69f22938 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-1nitxr4v-interopoadp/velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef/restic/test-oadp-1077 tags: backup: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef backup-uid: 09c11574-cdac-417f-b4bf-24474f5eb779 ns: test-oadp-1077 pod: mysql-845bdd7d8d-65hwv pod-uid: f5f08769-a69e-4147-ad6e-7cbb69f22938 pvc-uid: 90387d99-86bf-4a8d-b870-a6684c85891a volume: mysql-data uploaderType: restic volume: mysql-data status: completionTimestamp: "2024-08-19T12:00:36Z" path: /host_pods/f5f08769-a69e-4147-ad6e-7cbb69f22938/volumes/kubernetes.io~csi/pvc-90387d99-86bf-4a8d-b870-a6684c85891a/mount phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 snapshotID: fa9d8742 startTimestamp: "2024-08-19T12:00:33Z" 2024/08/19 12:00:48 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql-data1 creationTimestamp: "2024-08-19T12:00:33Z" generateName: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef- generation: 4 labels: velero.io/backup-name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef velero.io/backup-uid: 09c11574-cdac-417f-b4bf-24474f5eb779 velero.io/pvc-uid: 67c62e5c-b608-403f-808c-00abb36bba59 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"09c11574-cdac-417f-b4bf-24474f5eb779"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T12:00:33Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T12:00:42Z" name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef-9l2bh namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef uid: 09c11574-cdac-417f-b4bf-24474f5eb779 resourceVersion: "46685" uid: 91f27991-e94b-4f9f-8bf2-d4670685dbf0 spec: backupStorageLocation: ts-dpa-1 node: ci-op-1nitxr4v-2f653-hxcd7-worker-3-5dl9z pod: kind: Pod name: mysql-845bdd7d8d-65hwv namespace: test-oadp-1077 uid: f5f08769-a69e-4147-ad6e-7cbb69f22938 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-1nitxr4v-interopoadp/velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef/restic/test-oadp-1077 tags: backup: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef backup-uid: 09c11574-cdac-417f-b4bf-24474f5eb779 ns: test-oadp-1077 pod: mysql-845bdd7d8d-65hwv pod-uid: f5f08769-a69e-4147-ad6e-7cbb69f22938 pvc-uid: 67c62e5c-b608-403f-808c-00abb36bba59 volume: mysql-data1 uploaderType: restic volume: mysql-data1 status: completionTimestamp: "2024-08-19T12:00:42Z" path: /host_pods/f5f08769-a69e-4147-ad6e-7cbb69f22938/volumes/kubernetes.io~csi/pvc-67c62e5c-b608-403f-808c-00abb36bba59/mount phase: Completed progress: bytesDone: 104857640 totalBytes: 104857640 snapshotID: 8b8b0816 startTimestamp: "2024-08-19T12:00:38Z" STEP: Verify backup mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef has completed successfully @ 08/19/24 12:00:48.319 2024/08/19 12:00:48 Backup for case mysql succeeded STEP: Delete the appplication resources mysql @ 08/19/24 12:00:48.407 STEP: Cleanup Application for case mysql @ 08/19/24 12:00:48.407 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-1077] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2024/08/19 12:01:07 2024-08-19 12:00:49,889 p=30686 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:00:49,889 p=30686 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:00:50,152 p=30686 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:00:50,152 p=30686 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:00:50,431 p=30686 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:00:50,432 p=30686 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:00:50,452 p=30686 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:00:50,452 p=30686 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:50,760 p=30686 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:00:50,760 p=30686 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:50,788 p=30686 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:00:50,788 p=30686 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:50,807 p=30686 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:00:50,807 p=30686 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:00:50,818 p=30686 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:00:51,400 p=30686 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:00:51,400 p=30686 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:01:07,222 p=30686 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-1077] *** 2024-08-19 12:01:07,222 p=30686 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:01:07,222 p=30686 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:01:07,482 p=30686 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:01:07,482 p=30686 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 2024/08/19 12:01:07 Creating restore mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef for case mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef STEP: Create restore mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef from backup mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef @ 08/19/24 12:01:07.531 2024/08/19 12:01:07 Wait until restore mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef is complete restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: Completed 2024/08/19 12:02:07 Verify the PodVolumeBackup and PodVolumeRestore count is equal 2024/08/19 12:02:07 Verify the PodVolumeRestore is completed sucessfully and uploaderType is matching 2024/08/19 12:02:07 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-08-19T12:01:10Z" generateName: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef- generation: 5 labels: velero.io/pod-uid: 0b69b5d7-fdd6-4070-8b44-af6aec570260 velero.io/pvc-uid: 7410566a-386c-4f8a-85b7-20b93a57acc9 velero.io/restore-name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef velero.io/restore-uid: e38a872e-9d09-466b-b1b7-571947ab8bf4 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"e38a872e-9d09-466b-b1b7-571947ab8bf4"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T12:01:10Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T12:01:52Z" name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef-rjls7 namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef uid: e38a872e-9d09-466b-b1b7-571947ab8bf4 resourceVersion: "47279" uid: 07bb1987-81c4-4c32-a8b1-7f7013205637 spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-845bdd7d8d-65hwv namespace: test-oadp-1077 uid: 0b69b5d7-fdd6-4070-8b44-af6aec570260 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-1nitxr4v-interopoadp/velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef/restic/test-oadp-1077 snapshotID: 8b8b0816 sourceNamespace: test-oadp-1077 uploaderType: restic volume: mysql-data1 status: completionTimestamp: "2024-08-19T12:01:52Z" phase: Completed progress: bytesDone: 104857640 totalBytes: 104857640 startTimestamp: "2024-08-19T12:01:39Z" 2024/08/19 12:02:07 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-08-19T12:01:10Z" generateName: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef- generation: 5 labels: velero.io/pod-uid: 0b69b5d7-fdd6-4070-8b44-af6aec570260 velero.io/pvc-uid: 5ef7eda8-aa7c-45ab-85e5-d496d4fa0a4a velero.io/restore-name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef velero.io/restore-uid: e38a872e-9d09-466b-b1b7-571947ab8bf4 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"e38a872e-9d09-466b-b1b7-571947ab8bf4"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T12:01:10Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T12:01:36Z" name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef-x6d6h namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aef uid: e38a872e-9d09-466b-b1b7-571947ab8bf4 resourceVersion: "47203" uid: 32f6db45-33fe-46b5-a7dd-5f78a09e3b4c spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-845bdd7d8d-65hwv namespace: test-oadp-1077 uid: 0b69b5d7-fdd6-4070-8b44-af6aec570260 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-1nitxr4v-interopoadp/velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef/restic/test-oadp-1077 snapshotID: fa9d8742 sourceNamespace: test-oadp-1077 uploaderType: restic volume: mysql-data status: completionTimestamp: "2024-08-19T12:01:36Z" phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 startTimestamp: "2024-08-19T12:01:34Z" STEP: Verify restore mysql-6ee9a1d9-5e22-11ef-906a-0a580a810aefhas completed successfully @ 08/19/24 12:02:07.718 STEP: Verify Application restore @ 08/19/24 12:02:07.726 STEP: Verify Application deployment for case mysql @ 08/19/24 12:02:07.726 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 12:02:12 2024-08-19 12:02:09,125 p=30889 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:02:09,126 p=30889 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:09,359 p=30889 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:02:09,359 p=30889 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:09,589 p=30889 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:02:09,589 p=30889 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:09,604 p=30889 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:02:09,604 p=30889 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:09,892 p=30889 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:02:09,892 p=30889 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:09,920 p=30889 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:02:09,920 p=30889 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:09,939 p=30889 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:02:09,939 p=30889 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:09,949 p=30889 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:02:10,494 p=30889 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:02:10,494 p=30889 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:11,410 p=30889 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-08-19 12:02:11,411 p=30889 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:02:11,411 p=30889 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:11,758 p=30889 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 12:02:11,758 p=30889 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:12,196 p=30889 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-08-19 12:02:12,196 p=30889 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:12,798 p=30889 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-08-19 12:02:12,798 p=30889 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:12,810 p=30889 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:02:12,810 p=30889 u=1015750000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 12:02:12 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/08/19 12:02:12 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-1077] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2024/08/19 12:02:26 2024-08-19 12:02:14,256 p=31199 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:02:14,257 p=31199 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:14,495 p=31199 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:02:14,496 p=31199 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:14,728 p=31199 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:02:14,728 p=31199 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:14,743 p=31199 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:02:14,744 p=31199 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:15,043 p=31199 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:02:15,043 p=31199 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:15,072 p=31199 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:02:15,072 p=31199 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:15,094 p=31199 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:02:15,094 p=31199 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:15,104 p=31199 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:02:15,642 p=31199 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:02:15,642 p=31199 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:26,452 p=31199 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-1077] *** 2024-08-19 12:02:26,452 p=31199 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:02:26,452 p=31199 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:26,708 p=31199 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:02:26,708 p=31199 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 2024/08/19 12:02:26 Cleaning setup resources for the backup • [201.651 seconds] ------------------------------ Backup restore tests Application backup [tc-id:OADP-437][interop][smoke] MySQL application with filesystem, Kopia [mr-check] /alabama/cspi/e2e/app_backup/backup_restore.go:62 2024/08/19 12:02:26 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 08/19/24 12:02:26.819 2024/08/19 12:02:26 kopia 2024/08/19 12:02:26 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "20ba404a-cda9-4f59-a569-3745b461817e", "resourceVersion": "47651", "generation": 1, "creationTimestamp": "2024-08-19T12:02:26Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T12:02:26Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 08/19/24 12:02:26.923 2024/08/19 12:02:26 Waiting for velero pod to be running 2024/08/19 12:02:26 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/08/19 12:02:26 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "20ba404a-cda9-4f59-a569-3745b461817e", "resourceVersion": "47651", "generation": 1, "creationTimestamp": "2024-08-19T12:02:26Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T12:02:26Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 12:02:31.954 2024/08/19 12:02:31 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 08/19/24 12:02:31.976 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-437-kopia] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). FAILED - RETRYING: [localhost]: Check pod status (28 retries left). FAILED - RETRYING: [localhost]: Check pod status (27 retries left). FAILED - RETRYING: [localhost]: Check pod status (26 retries left). FAILED - RETRYING: [localhost]: Check pod status (25 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] Pausing for 30 seconds TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=20  changed=10  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/08/19 12:03:43 2024-08-19 12:02:33,315 p=31410 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:02:33,315 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:33,539 p=31410 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:02:33,540 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:33,772 p=31410 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:02:33,772 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:33,787 p=31410 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:02:33,788 p=31410 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:34,072 p=31410 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:02:34,072 p=31410 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:34,099 p=31410 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:02:34,099 p=31410 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:34,117 p=31410 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:02:34,118 p=31410 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:34,128 p=31410 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:02:34,654 p=31410 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:02:34,654 p=31410 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:35,389 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-437-kopia] *** 2024-08-19 12:02:35,390 p=31410 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:02:35,390 p=31410 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:02:35,823 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-08-19 12:02:35,824 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:02:36,711 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-08-19 12:02:36,711 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:10,569 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-08-19 12:03:10,570 p=31410 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:03:11,102 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-08-19 12:03:11,102 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:11,428 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 12:03:11,428 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:12,043 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-08-19 12:03:12,043 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:12,862 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-08-19 12:03:12,863 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:13,494 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-08-19 12:03:13,495 p=31410 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:13,513 p=31410 u=1015750000 n=ansible | Pausing for 30 seconds 2024-08-19 12:03:43,546 p=31410 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Pause After Create md5 hashes for the files] *** 2024-08-19 12:03:43,547 p=31410 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:03:43,617 p=31410 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:03:43,618 p=31410 u=1015750000 n=ansible | localhost : ok=20 changed=10 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 08/19/24 12:03:43.659 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 12:03:48 2024-08-19 12:03:45,047 p=31919 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:03:45,047 p=31919 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:45,290 p=31919 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:03:45,291 p=31919 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:45,552 p=31919 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:03:45,552 p=31919 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:45,568 p=31919 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:03:45,568 p=31919 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:03:45,861 p=31919 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:03:45,862 p=31919 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:03:45,891 p=31919 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:03:45,891 p=31919 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:03:45,915 p=31919 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:03:45,915 p=31919 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:03:45,932 p=31919 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:03:46,488 p=31919 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:03:46,488 p=31919 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:03:47,472 p=31919 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-08-19 12:03:47,473 p=31919 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:03:47,473 p=31919 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:03:47,804 p=31919 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 12:03:47,805 p=31919 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:48,259 p=31919 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-08-19 12:03:48,259 p=31919 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:48,846 p=31919 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-08-19 12:03:48,847 p=31919 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:03:48,859 p=31919 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:03:48,859 p=31919 u=1015750000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 12:03:48 {{ } { } [{{ } {mysql-data test-oadp-437-kopia e52b3e52-156a-4026-abd3-0ba672be4273 48010 0 2024-08-19 12:02:36 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/selected-node:ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-08-19 12:02:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-08-19 12:02:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-08-19 12:02:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-08-19 12:02:49 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-e52b3e52-156a-4026-abd3-0ba672be4273 0xc00043b4a0 0xc00043b4f0 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} {{ } {mysql-data1 test-oadp-437-kopia b8ace385-b1c8-45a2-a69c-a7dfae716e72 48015 0 2024-08-19 12:02:36 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/selected-node:ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-08-19 12:02:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-08-19 12:02:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-08-19 12:02:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-08-19 12:02:50 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-b8ace385-b1c8-45a2-a69c-a7dfae716e72 0xc00043b760 0xc00043b770 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }}]} STEP: Creating backup mysql-e6ec223c-5e22-11ef-906a-0a580a810aef @ 08/19/24 12:03:48.911 2024/08/19 12:03:48 Wait until backup mysql-e6ec223c-5e22-11ef-906a-0a580a810aef is completed backup phase: Completed 2024/08/19 12:04:08 Verify the PodVolumeBackup is completed successfully and BackupRepository type is matching with DPA.nodeAgent.uploaderType 2024/08/19 12:04:08 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql-data creationTimestamp: "2024-08-19T12:03:53Z" generateName: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef- generation: 5 labels: velero.io/backup-name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef velero.io/backup-uid: d36915d7-3d77-4148-b15c-d4bc0e5734fb velero.io/pvc-uid: e52b3e52-156a-4026-abd3-0ba672be4273 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"d36915d7-3d77-4148-b15c-d4bc0e5734fb"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T12:03:53Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T12:03:55Z" name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef-2xt7t namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef uid: d36915d7-3d77-4148-b15c-d4bc0e5734fb resourceVersion: "48442" uid: 4e372499-36c0-4b47-a569-623577b285d4 spec: backupStorageLocation: ts-dpa-1 node: ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z pod: kind: Pod name: mysql-845bdd7d8d-45gzj namespace: test-oadp-437-kopia uid: cdfa6e37-95a6-4f7d-bc40-9ac407b34206 repoIdentifier: "" tags: backup: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef backup-uid: d36915d7-3d77-4148-b15c-d4bc0e5734fb ns: test-oadp-437-kopia pod: mysql-845bdd7d8d-45gzj pod-uid: cdfa6e37-95a6-4f7d-bc40-9ac407b34206 pvc-uid: e52b3e52-156a-4026-abd3-0ba672be4273 volume: mysql-data uploaderType: kopia volume: mysql-data status: completionTimestamp: "2024-08-19T12:03:55Z" path: /host_pods/cdfa6e37-95a6-4f7d-bc40-9ac407b34206/volumes/kubernetes.io~csi/pvc-e52b3e52-156a-4026-abd3-0ba672be4273/mount phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 snapshotID: 2eaa8110b4fe48ef3c3d1e0b33a89590 startTimestamp: "2024-08-19T12:03:53Z" 2024/08/19 12:04:08 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql-data1 creationTimestamp: "2024-08-19T12:03:53Z" generateName: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef- generation: 5 labels: velero.io/backup-name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef velero.io/backup-uid: d36915d7-3d77-4148-b15c-d4bc0e5734fb velero.io/pvc-uid: b8ace385-b1c8-45a2-a69c-a7dfae716e72 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"d36915d7-3d77-4148-b15c-d4bc0e5734fb"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T12:03:53Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T12:04:03Z" name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef-wvp7x namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef uid: d36915d7-3d77-4148-b15c-d4bc0e5734fb resourceVersion: "48486" uid: 81c0879d-f9ab-4332-a03e-e7cc960b8bad spec: backupStorageLocation: ts-dpa-1 node: ci-op-1nitxr4v-2f653-hxcd7-worker-2-66x5z pod: kind: Pod name: mysql-845bdd7d8d-45gzj namespace: test-oadp-437-kopia uid: cdfa6e37-95a6-4f7d-bc40-9ac407b34206 repoIdentifier: "" tags: backup: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef backup-uid: d36915d7-3d77-4148-b15c-d4bc0e5734fb ns: test-oadp-437-kopia pod: mysql-845bdd7d8d-45gzj pod-uid: cdfa6e37-95a6-4f7d-bc40-9ac407b34206 pvc-uid: b8ace385-b1c8-45a2-a69c-a7dfae716e72 volume: mysql-data1 uploaderType: kopia volume: mysql-data1 status: completionTimestamp: "2024-08-19T12:04:03Z" path: /host_pods/cdfa6e37-95a6-4f7d-bc40-9ac407b34206/volumes/kubernetes.io~csi/pvc-b8ace385-b1c8-45a2-a69c-a7dfae716e72/mount phase: Completed progress: bytesDone: 104857640 totalBytes: 104857640 snapshotID: ca41aa4bf4ca63a1e94f45429a50cd15 startTimestamp: "2024-08-19T12:03:59Z" STEP: Verify backup mysql-e6ec223c-5e22-11ef-906a-0a580a810aef has completed successfully @ 08/19/24 12:04:08.963 2024/08/19 12:04:09 Backup for case mysql succeeded STEP: Delete the appplication resources mysql @ 08/19/24 12:04:09.043 STEP: Cleanup Application for case mysql @ 08/19/24 12:04:09.044 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-437-kopia] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2024/08/19 12:04:27 2024-08-19 12:04:10,521 p=32226 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:04:10,521 p=32226 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:04:10,761 p=32226 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:04:10,761 p=32226 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:04:10,994 p=32226 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:04:10,995 p=32226 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:04:11,009 p=32226 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:04:11,010 p=32226 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:04:11,302 p=32226 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:04:11,303 p=32226 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:04:11,329 p=32226 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:04:11,329 p=32226 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:04:11,346 p=32226 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:04:11,347 p=32226 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:04:11,356 p=32226 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:04:11,893 p=32226 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:04:11,893 p=32226 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:04:27,714 p=32226 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-437-kopia] *** 2024-08-19 12:04:27,715 p=32226 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:04:27,715 p=32226 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:04:27,950 p=32226 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:04:27,950 p=32226 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 2024/08/19 12:04:27 Creating restore mysql-e6ec223c-5e22-11ef-906a-0a580a810aef for case mysql-e6ec223c-5e22-11ef-906a-0a580a810aef STEP: Create restore mysql-e6ec223c-5e22-11ef-906a-0a580a810aef from backup mysql-e6ec223c-5e22-11ef-906a-0a580a810aef @ 08/19/24 12:04:27.994 2024/08/19 12:04:28 Wait until restore mysql-e6ec223c-5e22-11ef-906a-0a580a810aef is complete restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: Completed 2024/08/19 12:05:08 Verify the PodVolumeBackup and PodVolumeRestore count is equal 2024/08/19 12:05:08 Verify the PodVolumeRestore is completed sucessfully and uploaderType is matching 2024/08/19 12:05:08 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-08-19T12:04:31Z" generateName: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef- generation: 4 labels: velero.io/pod-uid: 1d92c1a2-401e-41cc-835d-8ce5801f9b78 velero.io/pvc-uid: 1effde79-ce56-4e04-a3d9-a9d8f0de8cdf velero.io/restore-name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef velero.io/restore-uid: 59721ece-6b42-4786-bd0d-d6041a49430d managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"59721ece-6b42-4786-bd0d-d6041a49430d"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T12:04:31Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T12:04:56Z" name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef-7dr6c namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef uid: 59721ece-6b42-4786-bd0d-d6041a49430d resourceVersion: "48979" uid: 4260257a-636b-48ef-ab38-4a657042b503 spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-845bdd7d8d-45gzj namespace: test-oadp-437-kopia uid: 1d92c1a2-401e-41cc-835d-8ce5801f9b78 repoIdentifier: "" snapshotID: 2eaa8110b4fe48ef3c3d1e0b33a89590 sourceNamespace: test-oadp-437-kopia uploaderType: kopia volume: mysql-data status: completionTimestamp: "2024-08-19T12:04:56Z" phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 startTimestamp: "2024-08-19T12:04:53Z" 2024/08/19 12:05:08 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-08-19T12:04:31Z" generateName: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef- generation: 4 labels: velero.io/pod-uid: 1d92c1a2-401e-41cc-835d-8ce5801f9b78 velero.io/pvc-uid: 23ba6d46-cf32-4808-9b92-06427d85c966 velero.io/restore-name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef velero.io/restore-uid: 59721ece-6b42-4786-bd0d-d6041a49430d managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"59721ece-6b42-4786-bd0d-d6041a49430d"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-08-19T12:04:31Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-08-19T12:05:03Z" name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef-mzbvn namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-e6ec223c-5e22-11ef-906a-0a580a810aef uid: 59721ece-6b42-4786-bd0d-d6041a49430d resourceVersion: "49035" uid: 21ecc77a-3c44-47e1-bd07-3dca601ed835 spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-845bdd7d8d-45gzj namespace: test-oadp-437-kopia uid: 1d92c1a2-401e-41cc-835d-8ce5801f9b78 repoIdentifier: "" snapshotID: ca41aa4bf4ca63a1e94f45429a50cd15 sourceNamespace: test-oadp-437-kopia uploaderType: kopia volume: mysql-data1 status: completionTimestamp: "2024-08-19T12:05:03Z" phase: Completed progress: bytesDone: 104857640 totalBytes: 104857640 startTimestamp: "2024-08-19T12:04:59Z" STEP: Verify restore mysql-e6ec223c-5e22-11ef-906a-0a580a810aefhas completed successfully @ 08/19/24 12:05:08.117 STEP: Verify Application restore @ 08/19/24 12:05:08.126 STEP: Verify Application deployment for case mysql @ 08/19/24 12:05:08.126 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 12:05:13 2024-08-19 12:05:09,693 p=32437 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:05:09,693 p=32437 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:09,968 p=32437 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:05:09,968 p=32437 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:10,215 p=32437 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:05:10,216 p=32437 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:10,231 p=32437 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:05:10,231 p=32437 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:10,526 p=32437 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:05:10,527 p=32437 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:10,555 p=32437 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:05:10,555 p=32437 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:10,577 p=32437 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:05:10,577 p=32437 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:10,588 p=32437 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:05:11,162 p=32437 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:05:11,162 p=32437 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:12,164 p=32437 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-08-19 12:05:12,164 p=32437 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:05:12,164 p=32437 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:12,532 p=32437 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-08-19 12:05:12,532 p=32437 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:13,023 p=32437 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-08-19 12:05:13,023 p=32437 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:13,618 p=32437 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-08-19 12:05:13,618 p=32437 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:13,631 p=32437 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:05:13,631 p=32437 u=1015750000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 12:05:13 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/08/19 12:05:13 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-437-kopia] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=15  rescued=0 ignored=0 2024/08/19 12:05:32 2024-08-19 12:05:15,129 p=32744 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:05:15,130 p=32744 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:15,374 p=32744 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:05:15,374 p=32744 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:15,607 p=32744 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:05:15,607 p=32744 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:15,623 p=32744 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:05:15,623 p=32744 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:15,913 p=32744 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:05:15,914 p=32744 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:15,941 p=32744 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:05:15,941 p=32744 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:15,960 p=32744 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:05:15,961 p=32744 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:15,971 p=32744 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:05:16,508 p=32744 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:05:16,508 p=32744 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:32,354 p=32744 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-437-kopia] *** 2024-08-19 12:05:32,354 p=32744 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:05:32,355 p=32744 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:32,607 p=32744 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:05:32,607 p=32744 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 2024/08/19 12:05:32 Cleaning setup resources for the backup • [185.907 seconds] ------------------------------ S ------------------------------ Backup restore tests Application backup [tc-id:OADP-122] [interop] [skip-disconnected] Django application with BSL&CSI [exclude_aro-4] /alabama/cspi/e2e/app_backup/backup_restore.go:93 2024/08/19 12:05:32 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 08/19/24 12:05:32.711 2024/08/19 12:05:32 csi 2024/08/19 12:05:32 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "74881a43-a276-464e-8002-c51bc55da45c", "resourceVersion": "49271", "generation": 1, "creationTimestamp": "2024-08-19T12:05:32Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T12:05:32Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "csi" ] } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 08/19/24 12:05:32.841 2024/08/19 12:05:32 Waiting for velero pod to be running 2024/08/19 12:05:32 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/08/19 12:05:32 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "74881a43-a276-464e-8002-c51bc55da45c", "resourceVersion": "49271", "generation": 1, "creationTimestamp": "2024-08-19T12:05:32Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T12:05:32Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "csi" ] } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 12:05:37.887 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/08/19 12:05:38 The 'openshift-storage' namespace does not exist 2024/08/19 12:05:38 Using default CSI driver based on infrastructure: vpc.block.csi.ibm.io 2024/08/19 12:05:38 Snapclass 'example-snapclass' doesn't exist, creating 2024/08/19 12:05:38 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 12:05:38 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier STEP: Installing application for case django-persistent @ 08/19/24 12:05:38.106 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check namespace test-oadp-122] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create namespace test-oadp-122] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create the mtc test django psql persistent template] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create openshift django psql persistent application from openshift templates] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/08/19 12:05:44 2024-08-19 12:05:39,564 p=32976 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:05:39,565 p=32976 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:39,801 p=32976 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:05:39,801 p=32976 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:40,041 p=32976 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:05:40,041 p=32976 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:40,057 p=32976 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:05:40,057 p=32976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:40,350 p=32976 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:05:40,350 p=32976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:40,378 p=32976 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:05:40,378 p=32976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:40,396 p=32976 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:05:40,396 p=32976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:40,406 p=32976 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:05:40,939 p=32976 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:05:40,940 p=32976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:41,736 p=32976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check namespace test-oadp-122] *** 2024-08-19 12:05:41,737 p=32976 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:05:41,737 p=32976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:42,196 p=32976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create namespace test-oadp-122] *** 2024-08-19 12:05:42,196 p=32976 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:43,032 p=32976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create the mtc test django psql persistent template] *** 2024-08-19 12:05:43,032 p=32976 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:43,813 p=32976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create openshift django psql persistent application from openshift templates] *** 2024-08-19 12:05:43,814 p=32976 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:43,962 p=32976 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:05:43,962 p=32976 u=1015750000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Verify Application deployment @ 08/19/24 12:05:44.011 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. FAILED - RETRYING: [localhost]: Check postgresql pod status (30 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (29 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (28 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (27 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (26 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (25 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (24 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (23 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Check application pod status (30 retries left). FAILED - RETRYING: [localhost]: Check application pod status (29 retries left). FAILED - RETRYING: [localhost]: Check application pod status (28 retries left). FAILED - RETRYING: [localhost]: Check application pod status (27 retries left). FAILED - RETRYING: [localhost]: Check application pod status (26 retries left). FAILED - RETRYING: [localhost]: Check application pod status (25 retries left). FAILED - RETRYING: [localhost]: Check application pod status (24 retries left). FAILED - RETRYING: [localhost]: Check application pod status (23 retries left). FAILED - RETRYING: [localhost]: Check application pod status (22 retries left). FAILED - RETRYING: [localhost]: Check application pod status (21 retries left). FAILED - RETRYING: [localhost]: Check application pod status (20 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** ok: [localhost] => {  "msg": "PASS: # of visits should be 1; actual 1" } PLAY RECAP ********************************************************************* localhost : ok=17  changed=3  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/08/19 12:07:36 2024-08-19 12:05:45,519 p=33261 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:05:45,520 p=33261 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:45,760 p=33261 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:05:45,760 p=33261 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:46,019 p=33261 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:05:46,020 p=33261 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:05:46,035 p=33261 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:05:46,035 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:46,353 p=33261 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:05:46,353 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:46,380 p=33261 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:05:46,380 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:46,398 p=33261 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:05:46,398 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:05:46,408 p=33261 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:05:46,940 p=33261 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:05:46,940 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:06:32,500 p=33261 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** 2024-08-19 12:06:32,500 p=33261 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:06:32,500 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:07:34,839 p=33261 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** 2024-08-19 12:07:34,839 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:07:35,926 p=33261 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** 2024-08-19 12:07:35,926 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:07:36,415 p=33261 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** 2024-08-19 12:07:36,415 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:07:36,442 p=33261 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** 2024-08-19 12:07:36,443 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:07:36,752 p=33261 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** 2024-08-19 12:07:36,752 p=33261 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:07:36,794 p=33261 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** 2024-08-19 12:07:36,794 p=33261 u=1015750000 n=ansible | ok: [localhost] => { "msg": "PASS: # of visits should be 1; actual 1" } 2024-08-19 12:07:36,806 p=33261 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:07:36,807 p=33261 u=1015750000 n=ansible | localhost : ok=17 changed=3 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 2024/08/19 12:07:36 {{ } { } [{{ } {postgresql test-oadp-122 f6060269-56cf-475d-9986-1836facb1012 49629 0 2024-08-19 12:05:43 +0000 UTC map[app:django-psql-persistent template:django-psql-persistent] map[openshift.io/generated-by:OpenShiftNewApp pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/selected-node:ci-op-1nitxr4v-2f653-hxcd7-worker-1-p8dtz volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{oc Update v1 2024-08-19 12:05:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/generated-by":{}},"f:labels":{".":{},"f:app":{},"f:template":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-08-19 12:05:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-08-19 12:05:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-08-19 12:05:59 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{1073741824 0} {} 1Gi BinarySI}]} pvc-f6060269-56cf-475d-9986-1836facb1012 0xc000738660 0xc000738670 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }}]} STEP: Creating backup django-persistent-55bac912-5e23-11ef-906a-0a580a810aef @ 08/19/24 12:07:36.883 2024/08/19 12:07:36 Wait until backup django-persistent-55bac912-5e23-11ef-906a-0a580a810aef is completed backup phase: WaitingForPluginOperations backup phase: WaitingForPluginOperations backup phase: Completed 2024/08/19 12:08:37 Verify the PVC and VSC count is matching 2024/08/19 12:08:37 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2024/08/19 12:08:37 Run velero describe on the backup 2024/08/19 12:08:37 [./velero describe backup django-persistent-55bac912-5e23-11ef-906a-0a580a810aef -n openshift-adp --details --insecure-skip-tls-verify] 2024/08/19 12:08:37 Exec stderr: "" 2024/08/19 12:08:37 Name: django-persistent-55bac912-5e23-11ef-906a-0a580a810aef Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.7+4510e9c velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: Completed Namespaces: Included: test-oadp-122 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-08-19 12:07:37 +0000 UTC Completed: 2024-08-19 12:08:19 +0000 UTC Expiration: 2024-09-18 12:07:36 +0000 UTC Total items to be backed up: 96 Items backed up: 96 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io test-oadp-122/velero-postgresql-q627x: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: test-oadp-122/velero-postgresql-q627x/2024-08-19T12:07:43Z Items to Update: volumesnapshots.snapshot.storage.k8s.io test-oadp-122/velero-postgresql-q627x volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-e2c4a73a-74c9-4f1b-9373-5c87d9198062 Phase: Completed Created: 2024-08-19 12:07:43 +0000 UTC Started: 2024-08-19 12:07:43 +0000 UTC Updated: 2024-08-19 12:08:18 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps.openshift.io/v1/DeploymentConfig: - test-oadp-122/django-psql-persistent - test-oadp-122/postgresql authorization.openshift.io/v1/RoleBinding: - test-oadp-122/admin - test-oadp-122/system:deployers - test-oadp-122/system:image-builders - test-oadp-122/system:image-pullers build.openshift.io/v1/Build: - test-oadp-122/django-psql-persistent-1 build.openshift.io/v1/BuildConfig: - test-oadp-122/django-psql-persistent discovery.k8s.io/v1/EndpointSlice: - test-oadp-122/django-psql-persistent-hzcxf - test-oadp-122/postgresql-w9pph image.openshift.io/v1/ImageStream: - test-oadp-122/django-psql-persistent image.openshift.io/v1/ImageStreamTag: - test-oadp-122/django-psql-persistent:latest image.openshift.io/v1/ImageTag: - test-oadp-122/django-psql-persistent:latest operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-122/volsync-product.v0.10.0 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-122/admin - test-oadp-122/system:deployers - test-oadp-122/system:image-builders - test-oadp-122/system:image-pullers route.openshift.io/v1/Route: - test-oadp-122/django-psql-persistent snapshot.storage.k8s.io/v1/VolumeSnapshot: - test-oadp-122/velero-postgresql-q627x snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-e2c4a73a-74c9-4f1b-9373-5c87d9198062 template.openshift.io/v1/Template: - test-oadp-122/mtc-test-django-psql-persistent v1/ConfigMap: - test-oadp-122/django-psql-persistent-1-ca - test-oadp-122/django-psql-persistent-1-global-ca - test-oadp-122/django-psql-persistent-1-sys-config - test-oadp-122/kube-root-ca.crt - test-oadp-122/openshift-service-ca.crt v1/Endpoints: - test-oadp-122/django-psql-persistent - test-oadp-122/postgresql v1/Event: - test-oadp-122/django-psql-persistent-1-build.17ed1fc0869169a8 - test-oadp-122/django-psql-persistent-1-build.17ed1fc09c8aadba - test-oadp-122/django-psql-persistent-1-build.17ed1fc09f47024a - test-oadp-122/django-psql-persistent-1-build.17ed1fc1d1f8bde6 - test-oadp-122/django-psql-persistent-1-build.17ed1fc1de65b622 - test-oadp-122/django-psql-persistent-1-build.17ed1fc1df7a98e7 - test-oadp-122/django-psql-persistent-1-build.17ed1fc222612b14 - test-oadp-122/django-psql-persistent-1-build.17ed1fc22f0fbfae - test-oadp-122/django-psql-persistent-1-build.17ed1fc2302010ab - test-oadp-122/django-psql-persistent-1-build.17ed1fc25e3f586a - test-oadp-122/django-psql-persistent-1-build.17ed1fc26a9d212a - test-oadp-122/django-psql-persistent-1-build.17ed1fc26ba228da - test-oadp-122/django-psql-persistent-1-deploy.17ed1fd551cd1919 - test-oadp-122/django-psql-persistent-1-deploy.17ed1fd56c86e3d4 - test-oadp-122/django-psql-persistent-1-deploy.17ed1fd56eb8e144 - test-oadp-122/django-psql-persistent-1-deploy.17ed1fd574e77d84 - test-oadp-122/django-psql-persistent-1-deploy.17ed1fd575f71d6d - test-oadp-122/django-psql-persistent-1-stdzq.17ed1fd5809201f2 - test-oadp-122/django-psql-persistent-1-stdzq.17ed1fd59b21d485 - test-oadp-122/django-psql-persistent-1-stdzq.17ed1fd59d34534b - test-oadp-122/django-psql-persistent-1-stdzq.17ed1fd948df2240 - test-oadp-122/django-psql-persistent-1-stdzq.17ed1fd9503ff6d4 - test-oadp-122/django-psql-persistent-1-stdzq.17ed1fd951683117 - test-oadp-122/django-psql-persistent-1.17ed1fc1ea152f3c - test-oadp-122/django-psql-persistent-1.17ed1fd57faed157 - test-oadp-122/django-psql-persistent-1.17ed1fd5962191c3 - test-oadp-122/django-psql-persistent.17ed1fd54e64d725 - test-oadp-122/postgresql-1-deploy.17ed1fc080a2aece - test-oadp-122/postgresql-1-deploy.17ed1fc0abe246b2 - test-oadp-122/postgresql-1-deploy.17ed1fc0ae3d2a63 - test-oadp-122/postgresql-1-deploy.17ed1fc0d420cd76 - test-oadp-122/postgresql-1-deploy.17ed1fc0dd2ab8f7 - test-oadp-122/postgresql-1-deploy.17ed1fc0de6a0de4 - test-oadp-122/postgresql-1-xbcjx.17ed1fc42e31ca86 - test-oadp-122/postgresql-1-xbcjx.17ed1fc6d7dc2d8f - test-oadp-122/postgresql-1-xbcjx.17ed1fc702945748 - test-oadp-122/postgresql-1-xbcjx.17ed1fc704bd6749 - test-oadp-122/postgresql-1-xbcjx.17ed1fc8bcbb1b29 - test-oadp-122/postgresql-1-xbcjx.17ed1fc8c48e02ac - test-oadp-122/postgresql-1-xbcjx.17ed1fc8c57c426f - test-oadp-122/postgresql-1.17ed1fc0e99f5cb5 - test-oadp-122/postgresql.17ed1fc06c122603 - test-oadp-122/postgresql.17ed1fc0797283bc - test-oadp-122/postgresql.17ed1fc0eb486651 - test-oadp-122/postgresql.17ed1fc0eb5d40e1 - test-oadp-122/postgresql.17ed1fc408798315 v1/Namespace: - test-oadp-122 v1/PersistentVolume: - pvc-f6060269-56cf-475d-9986-1836facb1012 v1/PersistentVolumeClaim: - test-oadp-122/postgresql v1/Pod: - test-oadp-122/django-psql-persistent-1-build - test-oadp-122/django-psql-persistent-1-deploy - test-oadp-122/django-psql-persistent-1-stdzq - test-oadp-122/postgresql-1-deploy - test-oadp-122/postgresql-1-xbcjx v1/ReplicationController: - test-oadp-122/django-psql-persistent-1 - test-oadp-122/postgresql-1 v1/Secret: - test-oadp-122/builder-dockercfg-mt98s - test-oadp-122/default-dockercfg-wk52v - test-oadp-122/deployer-dockercfg-5s5jp - test-oadp-122/django-psql-persistent v1/Service: - test-oadp-122/django-psql-persistent - test-oadp-122/postgresql v1/ServiceAccount: - test-oadp-122/builder - test-oadp-122/default - test-oadp-122/deployer Backup Volumes: Velero-Native Snapshots: CSI Snapshots: test-oadp-122/postgresql: Snapshot: Operation ID: test-oadp-122/velero-postgresql-q627x/2024-08-19T12:07:43Z Snapshot Content Name: snapcontent-e2c4a73a-74c9-4f1b-9373-5c87d9198062 Storage Snapshot ID: r014-4bec70f1-17cb-4d74-9e73-5d16b4d1348a Snapshot Size (bytes): 10737418240 CSI Driver: vpc.block.csi.ibm.io Pod Volume Backups: HooksAttempted: 0 HooksFailed: 0 STEP: Verify backup django-persistent-55bac912-5e23-11ef-906a-0a580a810aef has completed successfully @ 08/19/24 12:08:37.822 2024/08/19 12:08:37 Backup for case django-persistent succeeded STEP: Delete the appplication resources django-persistent @ 08/19/24 12:08:37.902 STEP: Cleanup Application for case django-persistent @ 08/19/24 12:08:37.902 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-122] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 12:08:56 2024-08-19 12:08:39,349 p=33768 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:08:39,349 p=33768 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:08:39,631 p=33768 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:08:39,631 p=33768 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:08:39,896 p=33768 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:08:39,897 p=33768 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:08:39,912 p=33768 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:08:39,912 p=33768 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:08:40,220 p=33768 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:08:40,220 p=33768 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:08:40,248 p=33768 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:08:40,248 p=33768 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:08:40,270 p=33768 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:08:40,271 p=33768 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:08:40,283 p=33768 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:08:40,813 p=33768 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:08:40,813 p=33768 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:08:56,677 p=33768 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-122] *** 2024-08-19 12:08:56,678 p=33768 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:08:56,678 p=33768 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:08:56,878 p=33768 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:08:56,879 p=33768 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 12:08:56 Creating restore django-persistent-55bac912-5e23-11ef-906a-0a580a810aef for case django-persistent-55bac912-5e23-11ef-906a-0a580a810aef STEP: Create restore django-persistent-55bac912-5e23-11ef-906a-0a580a810aef from backup django-persistent-55bac912-5e23-11ef-906a-0a580a810aef @ 08/19/24 12:08:56.933 2024/08/19 12:08:56 Wait until restore django-persistent-55bac912-5e23-11ef-906a-0a580a810aef is complete restore phase: Finalizing restore phase: Completed STEP: Verify restore django-persistent-55bac912-5e23-11ef-906a-0a580a810aefhas completed successfully @ 08/19/24 12:09:16.985 STEP: Verify Application restore @ 08/19/24 12:09:16.994 STEP: Verify Application deployment for case django-persistent @ 08/19/24 12:09:16.994 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Check application pod status (30 retries left). FAILED - RETRYING: [localhost]: Check application pod status (29 retries left). FAILED - RETRYING: [localhost]: Check application pod status (28 retries left). FAILED - RETRYING: [localhost]: Check application pod status (27 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** ok: [localhost] => {  "msg": "PASS: # of visits should be 2; actual 2" } PLAY RECAP ********************************************************************* localhost : ok=17  changed=3  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/08/19 12:09:45 2024-08-19 12:09:18,414 p=33976 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:09:18,415 p=33976 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:09:18,663 p=33976 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:09:18,664 p=33976 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:09:18,915 p=33976 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:09:18,915 p=33976 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:09:18,930 p=33976 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:09:18,931 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:19,237 p=33976 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:09:19,237 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:19,269 p=33976 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:09:19,269 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:19,289 p=33976 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:09:19,290 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:19,301 p=33976 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:09:19,872 p=33976 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:09:19,872 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:20,774 p=33976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** 2024-08-19 12:09:20,774 p=33976 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:09:20,774 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:43,776 p=33976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** 2024-08-19 12:09:43,776 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:44,781 p=33976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** 2024-08-19 12:09:44,782 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:45,229 p=33976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** 2024-08-19 12:09:45,230 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:45,255 p=33976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** 2024-08-19 12:09:45,256 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:45,597 p=33976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** 2024-08-19 12:09:45,597 p=33976 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:45,639 p=33976 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** 2024-08-19 12:09:45,639 p=33976 u=1015750000 n=ansible | ok: [localhost] => { "msg": "PASS: # of visits should be 2; actual 2" } 2024-08-19 12:09:45,651 p=33976 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:09:45,651 p=33976 u=1015750000 n=ansible | localhost : ok=17 changed=3 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 2024/08/19 12:09:45 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/08/19 12:09:45 Reset number of visits to 0 2024/08/19 12:09:45 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-122] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/08/19 12:10:04 2024-08-19 12:09:47,138 p=34289 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:09:47,138 p=34289 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:09:47,367 p=34289 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:09:47,368 p=34289 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:09:47,600 p=34289 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:09:47,600 p=34289 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:09:47,615 p=34289 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:09:47,615 p=34289 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:47,907 p=34289 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:09:47,908 p=34289 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:47,936 p=34289 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:09:47,936 p=34289 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:47,957 p=34289 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:09:47,957 p=34289 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:09:47,972 p=34289 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:09:48,496 p=34289 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:09:48,496 p=34289 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:04,340 p=34289 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-122] *** 2024-08-19 12:10:04,341 p=34289 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:10:04,341 p=34289 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:04,572 p=34289 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:10:04,572 p=34289 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/08/19 12:10:04 Cleaning setup resources for the backup 2024/08/19 12:10:04 Setting new default StorageClass 'ibmc-vpc-block-10iops-tier' 2024/08/19 12:10:04 Checking default storage class count Skipping creation of StorageClass The current StorageClass: ibmc-vpc-block-10iops-tier matches the new StorageClass: ibmc-vpc-block-10iops-tier 2024/08/19 12:10:04 Deleting VolumeSnapshotClass 'example-snapclass' • [272.016 seconds] ------------------------------ SSSSS ------------------------------ Backup restore tests Application backup [tc-id:OADP-97][interop] Empty-project application with Restic /alabama/cspi/e2e/app_backup/backup_restore.go:191 2024/08/19 12:10:04 Delete all downloadrequest django-persistent-55bac912-5e23-11ef-906a-0a580a810aef-a141019c-820f-4895-8c06-4e46fc4c2b6e django-persistent-55bac912-5e23-11ef-906a-0a580a810aef-fd32084d-4a3e-4965-84b6-660c24691fa4 django-persistent-55bac912-5e23-11ef-906a-0a580a810aef-ff377339-938a-4533-b1a2-6da88527ddca STEP: Create DPA CR @ 08/19/24 12:10:04.809 2024/08/19 12:10:04 restic 2024/08/19 12:10:04 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "ae2c08b1-e7ce-44c9-b6b8-25df3952b974", "resourceVersion": "51737", "generation": 1, "creationTimestamp": "2024-08-19T12:10:04Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T12:10:04Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 08/19/24 12:10:04.871 2024/08/19 12:10:04 Waiting for velero pod to be running 2024/08/19 12:10:04 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/08/19 12:10:04 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "ae2c08b1-e7ce-44c9-b6b8-25df3952b974", "resourceVersion": "51737", "generation": 1, "creationTimestamp": "2024-08-19T12:10:04Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T12:10:04Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 12:10:09.905 2024/08/19 12:10:09 Checking for correct number of running NodeAgent pods... STEP: Installing application for case empty-project-e2e @ 08/19/24 12:10:09.927 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Deploy project with labels and selectors] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/08/19 12:10:13 2024-08-19 12:10:11,499 p=34497 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:10:11,500 p=34497 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:11,752 p=34497 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:10:11,753 p=34497 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:12,006 p=34497 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:10:12,007 p=34497 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:12,022 p=34497 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:10:12,022 p=34497 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:12,345 p=34497 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:10:12,345 p=34497 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:12,377 p=34497 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:10:12,377 p=34497 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:12,397 p=34497 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:10:12,397 p=34497 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:12,410 p=34497 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:10:12,971 p=34497 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:10:12,971 p=34497 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:13,839 p=34497 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Deploy project with labels and selectors] *** 2024-08-19 12:10:13,840 p=34497 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:10:13,840 p=34497 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:13,886 p=34497 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:10:13,886 p=34497 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 STEP: Verify Application deployment @ 08/19/24 12:10:13.933 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Check project status] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=3  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/08/19 12:10:17 2024-08-19 12:10:15,295 p=34697 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:10:15,296 p=34697 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:15,532 p=34697 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:10:15,532 p=34697 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:15,788 p=34697 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:10:15,789 p=34697 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:15,809 p=34697 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:10:15,809 p=34697 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:16,152 p=34697 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:10:16,152 p=34697 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:16,179 p=34697 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:10:16,180 p=34697 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:16,198 p=34697 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:10:16,198 p=34697 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:16,208 p=34697 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:10:16,737 p=34697 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:10:16,737 p=34697 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:17,522 p=34697 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Check project status] *** 2024-08-19 12:10:17,522 p=34697 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:10:17,522 p=34697 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:17,554 p=34697 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:10:17,554 p=34697 u=1015750000 n=ansible | localhost : ok=11 changed=3 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/08/19 12:10:17 {{ } { } []} STEP: Creating backup empty-project-e2e-f7dd1e39-5e23-11ef-906a-0a580a810aef @ 08/19/24 12:10:17.653 2024/08/19 12:10:17 Wait until backup empty-project-e2e-f7dd1e39-5e23-11ef-906a-0a580a810aef is completed backup phase: Completed STEP: Verify backup empty-project-e2e-f7dd1e39-5e23-11ef-906a-0a580a810aef has completed successfully @ 08/19/24 12:10:37.681 2024/08/19 12:10:37 Backup for case empty-project-e2e succeeded STEP: Delete the appplication resources empty-project-e2e @ 08/19/24 12:10:37.748 STEP: Cleanup Application for case empty-project-e2e @ 08/19/24 12:10:37.748 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Remove namespace test-oadp-97] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/08/19 12:10:51 2024-08-19 12:10:39,142 p=34896 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:10:39,142 p=34896 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:39,401 p=34896 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:10:39,402 p=34896 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:39,661 p=34896 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:10:39,661 p=34896 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:39,677 p=34896 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:10:39,678 p=34896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:39,976 p=34896 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:10:39,977 p=34896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:40,005 p=34896 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:10:40,006 p=34896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:40,026 p=34896 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:10:40,026 p=34896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:40,036 p=34896 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:10:40,557 p=34896 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:10:40,557 p=34896 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:10:51,390 p=34896 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Remove namespace test-oadp-97] *** 2024-08-19 12:10:51,390 p=34896 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:10:51,390 p=34896 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:10:51,449 p=34896 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:10:51,449 p=34896 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/08/19 12:10:51 Creating restore empty-project-e2e-f7dd1e39-5e23-11ef-906a-0a580a810aef for case empty-project-e2e-f7dd1e39-5e23-11ef-906a-0a580a810aef STEP: Create restore empty-project-e2e-f7dd1e39-5e23-11ef-906a-0a580a810aef from backup empty-project-e2e-f7dd1e39-5e23-11ef-906a-0a580a810aef @ 08/19/24 12:10:51.487 2024/08/19 12:10:51 Wait until restore empty-project-e2e-f7dd1e39-5e23-11ef-906a-0a580a810aef is complete restore phase: Completed 2024/08/19 12:11:01 No PodVolumeBackup CR found for the Restore STEP: Verify restore empty-project-e2e-f7dd1e39-5e23-11ef-906a-0a580a810aefhas completed successfully @ 08/19/24 12:11:01.537 STEP: Verify Application restore @ 08/19/24 12:11:01.545 STEP: Verify Application deployment for case empty-project-e2e @ 08/19/24 12:11:01.545 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Check project status] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=3  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/08/19 12:11:05 2024-08-19 12:11:02,907 p=35096 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:11:02,907 p=35096 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:03,145 p=35096 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:11:03,146 p=35096 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:03,396 p=35096 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:11:03,396 p=35096 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:03,415 p=35096 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:11:03,415 p=35096 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:03,703 p=35096 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:11:03,704 p=35096 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:03,730 p=35096 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:11:03,731 p=35096 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:03,748 p=35096 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:11:03,748 p=35096 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:03,758 p=35096 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:11:04,283 p=35096 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:11:04,284 p=35096 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:05,069 p=35096 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Check project status] *** 2024-08-19 12:11:05,070 p=35096 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:11:05,070 p=35096 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:05,102 p=35096 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:11:05,102 p=35096 u=1015750000 n=ansible | localhost : ok=11 changed=3 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/08/19 12:11:05 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/08/19 12:11:05 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Remove namespace test-oadp-97] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/08/19 12:11:18 2024-08-19 12:11:06,543 p=35296 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:11:06,544 p=35296 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:06,780 p=35296 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:11:06,780 p=35296 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:07,019 p=35296 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:11:07,019 p=35296 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:07,035 p=35296 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:11:07,035 p=35296 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:07,351 p=35296 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:11:07,351 p=35296 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:07,378 p=35296 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:11:07,378 p=35296 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:07,396 p=35296 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:11:07,396 p=35296 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:07,406 p=35296 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:11:07,926 p=35296 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:11:07,926 p=35296 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:18,717 p=35296 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Remove namespace test-oadp-97] *** 2024-08-19 12:11:18,717 p=35296 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:11:18,718 p=35296 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:18,785 p=35296 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:11:18,786 p=35296 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/08/19 12:11:18 Cleaning setup resources for the backup • [74.170 seconds] ------------------------------ SSSSSSSSSSSSSSSSSS ------------------------------ Backup hooks tests Pre exec hook [tc-id:OADP-92][interop][smoke] Cassandra app with Restic /alabama/cspi/e2e/hooks/backup_hooks.go:109 2024/08/19 12:11:18 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 08/19/24 12:11:18.88 2024/08/19 12:11:18 restic 2024/08/19 12:11:18 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "935b97de-d346-4f8a-b709-c476f4086442", "resourceVersion": "52407", "generation": 1, "creationTimestamp": "2024-08-19T12:11:18Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T12:11:18Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 08/19/24 12:11:19.003 2024/08/19 12:11:19 Waiting for velero pod to be running 2024/08/19 12:11:19 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/08/19 12:11:19 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "935b97de-d346-4f8a-b709-c476f4086442", "resourceVersion": "52407", "generation": 1, "creationTimestamp": "2024-08-19T12:11:18Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-08-19T12:11:18Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-1nitxr4v-interopoadp", "prefix": "velero-e2e-6fcf3f84-5e1e-11ef-906a-0a580a810aef" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 08/19/24 12:11:24.042 2024/08/19 12:11:24 Checking for correct number of running NodeAgent pods... STEP: Installing application for case cassandra-hooks-e2e @ 08/19/24 12:11:24.062 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pods status (30 retries left). FAILED - RETRYING: [localhost]: Check pods status (29 retries left). FAILED - RETRYING: [localhost]: Check pods status (28 retries left). FAILED - RETRYING: [localhost]: Check pods status (27 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (30 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (29 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (28 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (27 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (26 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (25 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (24 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (23 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (22 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (21 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=18  changed=9  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/08/19 12:14:25 2024-08-19 12:11:25,460 p=35496 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:11:25,460 p=35496 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:25,698 p=35496 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:11:25,698 p=35496 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:25,931 p=35496 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:11:25,932 p=35496 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:25,947 p=35496 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:11:25,947 p=35496 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:26,257 p=35496 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:11:26,257 p=35496 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:26,286 p=35496 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:11:26,286 p=35496 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:26,306 p=35496 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:11:26,306 p=35496 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:26,322 p=35496 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:11:26,855 p=35496 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:11:26,855 p=35496 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:27,628 p=35496 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** 2024-08-19 12:11:27,629 p=35496 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:11:27,629 p=35496 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:11:28,033 p=35496 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** 2024-08-19 12:11:28,033 p=35496 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:28,322 p=35496 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** 2024-08-19 12:11:28,322 p=35496 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:29,204 p=35496 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** 2024-08-19 12:11:29,204 p=35496 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:29,926 p=35496 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** 2024-08-19 12:11:29,927 p=35496 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:11:52,813 p=35496 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-08-19 12:11:52,813 p=35496 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:14:21,169 p=35496 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-08-19 12:14:21,169 p=35496 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:14:25,484 p=35496 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** 2024-08-19 12:14:25,485 p=35496 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:14:25,575 p=35496 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:14:25,575 p=35496 u=1015750000 n=ansible | localhost : ok=18 changed=9 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 08/19/24 12:14:25.616 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=15  changed=6  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/08/19 12:14:37 2024-08-19 12:14:27,042 p=36231 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:14:27,042 p=36231 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:14:27,282 p=36231 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:14:27,282 p=36231 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:14:27,538 p=36231 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:14:27,539 p=36231 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:14:27,555 p=36231 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:14:27,555 p=36231 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:14:27,855 p=36231 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:14:27,855 p=36231 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:14:27,882 p=36231 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:14:27,882 p=36231 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:14:27,900 p=36231 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:14:27,900 p=36231 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:14:27,910 p=36231 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:14:28,454 p=36231 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:14:28,454 p=36231 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:14:29,379 p=36231 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-08-19 12:14:29,379 p=36231 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:14:29,380 p=36231 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:14:35,690 p=36231 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-08-19 12:14:35,691 p=36231 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:14:37,175 p=36231 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** 2024-08-19 12:14:37,176 p=36231 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:14:37,656 p=36231 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** 2024-08-19 12:14:37,657 p=36231 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:14:37,827 p=36231 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** 2024-08-19 12:14:37,827 p=36231 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:14:37,840 p=36231 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:14:37,840 p=36231 u=1015750000 n=ansible | localhost : ok=15 changed=6 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Creating backup cassandra-hooks-e2e @ 08/19/24 12:14:37.885 2024/08/19 12:14:37 Wait until backup cassandra-hooks-e2e is completed backup phase: InProgress backup phase: InProgress backup phase: InProgress backup phase: Completed STEP: Verify backup cassandra-hooks-e2e has completed successfully @ 08/19/24 12:15:57.978 2024/08/19 12:15:58 Backup for case cassandra-hooks-e2e succeeded STEP: Verify pre-backup hooks were executed; verify that the cassandra app is quiesced and it's not possible to connect to the DB server @ 08/19/24 12:15:58.069 2024/08/19 12:15:58 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.4.0 2024/08/19 12:15:58 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-92] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/08/19 12:16:17 2024-08-19 12:15:59,952 p=36537 u=1015750000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-08-19 12:15:59,952 p=36537 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:16:00,195 p=36537 u=1015750000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-08-19 12:16:00,195 p=36537 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:16:00,441 p=36537 u=1015750000 n=ansible | TASK [Get current admin token] ************************************************* 2024-08-19 12:16:00,441 p=36537 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:16:00,456 p=36537 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:16:00,456 p=36537 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:16:00,760 p=36537 u=1015750000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-08-19 12:16:00,760 p=36537 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:16:00,787 p=36537 u=1015750000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-08-19 12:16:00,787 p=36537 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:16:00,805 p=36537 u=1015750000 n=ansible | TASK [set_fact] **************************************************************** 2024-08-19 12:16:00,805 p=36537 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:16:00,815 p=36537 u=1015750000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-08-19 12:16:01,327 p=36537 u=1015750000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-08-19 12:16:01,327 p=36537 u=1015750000 n=ansible | ok: [localhost] 2024-08-19 12:16:17,140 p=36537 u=1015750000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-92] *** 2024-08-19 12:16:17,140 p=36537 u=1015750000 n=ansible | [WARNING]: kubernetes<24.2.0 is not supported or tested. Some features may not work. 2024-08-19 12:16:17,141 p=36537 u=1015750000 n=ansible | changed: [localhost] 2024-08-19 12:16:17,354 p=36537 u=1015750000 n=ansible | PLAY RECAP ********************************************************************* 2024-08-19 12:16:17,354 p=36537 u=1015750000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/08/19 12:16:17 Cleaning setup resources for the backup • [298.557 seconds] ------------------------------ S ------------------------------ [SynchronizedAfterSuite]  /alabama/cspi/e2e/e2e_suite_test.go:181 2024/08/19 12:16:17 Deleting Velero CR [SynchronizedAfterSuite] PASSED [0.012 seconds] ------------------------------ [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report autogenerated by Ginkgo [ReportAfterSuite] PASSED [0.014 seconds] ------------------------------ Summarizing 1 Failure: [FAIL] [datamover] DataMover: Backup/Restore stateful application with CSI  [It] [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application /alabama/cspi/test_common/backup_restore_app_case.go:24 Ran 9 of 193 Specs in 2748.470 seconds FAIL! -- 8 Passed | 1 Failed | 0 Pending | 184 Skipped --- FAIL: TestOADPE2E (2748.50s) FAIL Ginkgo ran 1 suite in 46m33.667518797s Test Suite Failed