Extract /home/jenkins/oadp-e2e-qe.tar.gz to /alabama/cspi Extract /home/jenkins/oadp-apps-deployer.tar.gz to /alabama/oadpApps Extract /home/jenkins/mtc-python-client.tar.gz to /alabama/pyclient Create and populate /tmp/test-settings... Login as Kubeadmin to the test cluster at https://api.ci-op-zciz01md-10d22.cspilp.interop.ccitredhat.com:6443... WARNING: Using insecure TLS client config. Setting this option is not supported! Login successful. You have access to 71 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". Create virtual environment and install required packages... Collecting ansible_runner Downloading ansible_runner-2.3.6-py3-none-any.whl (81 kB) Collecting python-daemon Downloading python_daemon-3.0.1-py3-none-any.whl (31 kB) Collecting packaging Downloading packaging-24.0-py3-none-any.whl (53 kB) Collecting pexpect>=4.5 Downloading pexpect-4.9.0-py2.py3-none-any.whl (63 kB) Collecting six Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting pyyaml Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB) Collecting ptyprocess>=0.5 Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB) Collecting setuptools>=62.4.0 Downloading setuptools-69.5.1-py3-none-any.whl (894 kB) Collecting docutils Downloading docutils-0.21.1-py3-none-any.whl (587 kB) Collecting lockfile>=0.10 Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB) Installing collected packages: setuptools, ptyprocess, lockfile, docutils, six, pyyaml, python-daemon, pexpect, packaging, ansible-runner Attempting uninstall: setuptools Found existing installation: setuptools 57.4.0 Uninstalling setuptools-57.4.0: Successfully uninstalled setuptools-57.4.0 Successfully installed ansible-runner-2.3.6 docutils-0.21.1 lockfile-0.12.2 packaging-24.0 pexpect-4.9.0 ptyprocess-0.7.0 python-daemon-3.0.1 pyyaml-6.0.1 setuptools-69.5.1 six-1.16.0 WARNING: You are using pip version 21.2.3; however, version 24.0 is available. You should consider upgrading via the '/alabama/venv/bin/python3 -m pip install --upgrade pip' command. Processing /alabama/oadpApps DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing wheel metadata: started Preparing wheel metadata: finished with status 'done' Building wheels for collected packages: ocpdeployer Building wheel for ocpdeployer (PEP 517): started Building wheel for ocpdeployer (PEP 517): finished with status 'done' Created wheel for ocpdeployer: filename=ocpdeployer-0.0.1-py2.py3-none-any.whl size=213270 sha256=ac996738b151a051ed60afe792899be3addfbf0c9197a43d53aabea89426542e Stored in directory: /tmp/pip-ephem-wheel-cache-cvac1exj/wheels/ea/a6/87/9d98fc51cc395d30bd2147a7e53e3f5a2e80044d9dd9e64977 Successfully built ocpdeployer Installing collected packages: ocpdeployer WARNING: Value for scheme.platlib does not match. Please report this to distutils: /tmp/pip-target-x5qc0w_v/lib64/python sysconfig: /tmp/pip-target-x5qc0w_v/lib/python WARNING: Additional context: user = False home = '/tmp/pip-target-x5qc0w_v' root = None prefix = None Successfully installed ocpdeployer-0.0.1 WARNING: You are using pip version 21.2.3; however, version 24.0 is available. You should consider upgrading via the '/alabama/venv/bin/python3 -m pip install --upgrade pip' command. Processing /alabama/pyclient DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. Collecting suds-py3 Downloading suds_py3-1.4.5.0-py3-none-any.whl (298 kB) Collecting requests Downloading requests-2.31.0-py3-none-any.whl (62 kB) Collecting jinja2 Downloading Jinja2-3.1.3-py3-none-any.whl (133 kB) Collecting kubernetes==11.0.0 Downloading kubernetes-11.0.0-py3-none-any.whl (1.5 MB) Collecting openshift==0.11.2 Downloading openshift-0.11.2.tar.gz (19 kB) Requirement already satisfied: setuptools>=21.0.0 in /alabama/venv/lib/python3.10/site-packages (from kubernetes==11.0.0->mtc==0.0.1) (69.5.1) Collecting certifi>=14.05.14 Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) Requirement already satisfied: pyyaml>=3.12 in /alabama/venv/lib/python3.10/site-packages (from kubernetes==11.0.0->mtc==0.0.1) (6.0.1) Requirement already satisfied: six>=1.9.0 in /alabama/venv/lib/python3.10/site-packages (from kubernetes==11.0.0->mtc==0.0.1) (1.16.0) Collecting urllib3>=1.24.2 Downloading urllib3-2.2.1-py3-none-any.whl (121 kB) Collecting google-auth>=1.0.1 Downloading google_auth-2.29.0-py2.py3-none-any.whl (189 kB) Collecting python-dateutil>=2.5.3 Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 Downloading websocket_client-1.7.0-py3-none-any.whl (58 kB) Collecting requests-oauthlib Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl (24 kB) Collecting python-string-utils Downloading python_string_utils-1.0.0-py3-none-any.whl (26 kB) Collecting ruamel.yaml>=0.15 Downloading ruamel.yaml-0.18.6-py3-none-any.whl (117 kB) Collecting cachetools<6.0,>=2.0.0 Downloading cachetools-5.3.3-py3-none-any.whl (9.3 kB) Collecting rsa<5,>=3.1.4 Downloading rsa-4.9-py3-none-any.whl (34 kB) Collecting pyasn1-modules>=0.2.1 Downloading pyasn1_modules-0.4.0-py3-none-any.whl (181 kB) Collecting pyasn1<0.7.0,>=0.4.6 Downloading pyasn1-0.6.0-py2.py3-none-any.whl (85 kB) Collecting ruamel.yaml.clib>=0.2.7 Downloading ruamel.yaml.clib-0.2.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (526 kB) Collecting MarkupSafe>=2.0 Downloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB) Collecting idna<4,>=2.5 Downloading idna-3.7-py3-none-any.whl (66 kB) Collecting charset-normalizer<4,>=2 Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB) Collecting oauthlib>=3.0.0 Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB) Using legacy 'setup.py install' for mtc, since package 'wheel' is not installed. Using legacy 'setup.py install' for openshift, since package 'wheel' is not installed. Installing collected packages: urllib3, pyasn1, idna, charset-normalizer, certifi, rsa, requests, pyasn1-modules, oauthlib, cachetools, websocket-client, ruamel.yaml.clib, requests-oauthlib, python-dateutil, MarkupSafe, google-auth, ruamel.yaml, python-string-utils, kubernetes, jinja2, suds-py3, openshift, mtc Running setup.py install for openshift: started Running setup.py install for openshift: finished with status 'done' Running setup.py install for mtc: started Running setup.py install for mtc: finished with status 'done' Successfully installed MarkupSafe-2.1.5 cachetools-5.3.3 certifi-2024.2.2 charset-normalizer-3.3.2 google-auth-2.29.0 idna-3.7 jinja2-3.1.3 kubernetes-11.0.0 mtc-0.0.1 oauthlib-3.2.2 openshift-0.11.2 pyasn1-0.6.0 pyasn1-modules-0.4.0 python-dateutil-2.9.0.post0 python-string-utils-1.0.0 requests-2.31.0 requests-oauthlib-2.0.0 rsa-4.9 ruamel.yaml-0.18.6 ruamel.yaml.clib-0.2.8 suds-py3-1.4.5.0 urllib3-2.2.1 websocket-client-1.7.0 WARNING: You are using pip version 21.2.3; however, version 24.0 is available. You should consider upgrading via the '/alabama/venv/bin/python3 -m pip install --upgrade pip' command. Executing tests... + readonly 'RED=\e[31m' + RED='\e[31m' + readonly 'BLUE=\033[34m' + BLUE='\033[34m' + readonly 'CLEAR=\e[39m' + CLEAR='\e[39m' ++ oc get infrastructures cluster -o 'jsonpath={.status.platform}' ++ awk '{print tolower($0)}' + CLOUD_PROVIDER=aws + [[ aws == *-arm* ]] + E2E_TIMEOUT_MULTIPLIER=2 + export NAMESPACE=openshift-adp + NAMESPACE=openshift-adp + export PROVIDER=aws + PROVIDER=aws ++ echo aws ++ awk '{print tolower($0)}' + BACKUP_LOCATION=aws + export BACKUP_LOCATION=aws + BACKUP_LOCATION=aws + export BUCKET=ci-op-zciz01md-interopoadp + BUCKET=ci-op-zciz01md-interopoadp + OADP_CREDS_FILE=/tmp/test-settings/credentials + OADP_VSL_CREDS_FILE=/tmp/test-settings/aws_vsl_creds +++ readlink -f /alabama/cspi/test_settings/scripts/test_runner.sh ++ dirname /alabama/cspi/test_settings/scripts/test_runner.sh + readonly SCRIPT_DIR=/alabama/cspi/test_settings/scripts + SCRIPT_DIR=/alabama/cspi/test_settings/scripts ++ cd /alabama/cspi/test_settings/scripts ++ git rev-parse --show-toplevel + readonly TOP_DIR=/alabama/cspi + TOP_DIR=/alabama/cspi + echo /alabama/cspi /alabama/cspi + TESTS_FOLDER=e2e ++ oc get nodes -o 'jsonpath={.items[*].metadata.labels.kubernetes\.io/arch}' ++ tr ' ' '\n' ++ sort -u ++ xargs + export NODES_ARCHITECTURE=amd64 + NODES_ARCHITECTURE=amd64 ++ oc get ns openshift-storage ++ echo false + OPENSHIFT_STORAGE=false + '[' false == true ']' ++ oc get storageclass -o 'jsonpath={range .items[*]}{@.metadata.name}{" "}{@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}{end}' ++ awk '$2=="true"{print $1}' ++ wc -l + NUM_DEFAULT_STORAGE_CLASS=1 + '[' 1 -ne 1 ']' ++ oc get storageclass -o 'jsonpath={.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=='\''true'\'')].metadata.name}' + DEFAULT_SC=gp3-csi + export STORAGE_CLASS=gp3-csi + STORAGE_CLASS=gp3-csi + '[' -n gp3-csi ']' + '[' gp3-csi '!=' gp3-csi ']' + export STORAGE_CLASS_OPENSHIFT_STORAGE=gp3-csi + STORAGE_CLASS_OPENSHIFT_STORAGE=gp3-csi + echo 'Using the StorageClass for openshift-storage: gp3-csi' Using the StorageClass for openshift-storage: gp3-csi + [[ amd64 != \a\m\d\6\4 ]] + TEST_FILTER='!// || (// && !exclude_aws && (!/target/ || target_aws) ) ' + [[ aws =~ ^osp ]] + [[ aws =~ ^vsphere ]] + [[ aws =~ ^gcp-wif ]] + [[ aws =~ ^ibmcloud ]] ++ oc config current-context ++ awk -F / '{print $2}' + SETTINGS_TMP=/alabama/cspi/output_files/api-ci-op-zciz01md-10d22-cspilp-interop-ccitredhat-com:6443 + mkdir -p /alabama/cspi/output_files/api-ci-op-zciz01md-10d22-cspilp-interop-ccitredhat-com:6443 ++ oc get authentication cluster -o 'jsonpath={.spec.serviceAccountIssuer}' + IS_OIDC= + '[' '!' -z ']' + [[ aws == \a\w\s ]] + export PROVIDER=aws + PROVIDER=aws + export CREDS_SECRET_REF=cloud-credentials + CREDS_SECRET_REF=cloud-credentials ++ oc get infrastructures cluster -o 'jsonpath={.status.platformStatus.aws.region}' --allow-missing-template-keys=false + export REGION=us-east-2 + REGION=us-east-2 + settings_script=aws_settings.sh + '[' aws == aws-sts ']' + BUCKET=ci-op-zciz01md-interopoadp + TMP_DIR=/alabama/cspi/output_files/api-ci-op-zciz01md-10d22-cspilp-interop-ccitredhat-com:6443 + source /alabama/cspi/test_settings/scripts/aws_settings.sh ++ cat ++ [[ aws == *aws* ]] ++ cat ++ echo -e '\n }\n}' +++ cat /alabama/cspi/output_files/api-ci-op-zciz01md-10d22-cspilp-interop-ccitredhat-com:6443/settings.json ++ x='{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-2" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-zciz01md-interopoadp" } } } ] , "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-2" } } } ] } }' ++ echo '{ "metadata": { "namespace": "openshift-adp" }, "spec": { "configuration":{ "velero":{ "defaultPlugins": [ "openshift", "aws" ] } }, "backupLocations": [ { "velero": { "provider": "aws", "default": true, "config": { "region": "us-east-2" }, "credential":{ "name": "cloud-credentials", "key": "cloud" }, "objectStorage":{ "bucket": "ci-op-zciz01md-interopoadp" } } } ] , "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-2" } } } ] } }' ++ grep -o '^[^#]*' + FILE_SETTINGS_NAME=settings.json + printf '\033[34mGenerated settings file under /alabama/cspi/output_files/api-ci-op-zciz01md-10d22-cspilp-interop-ccitredhat-com:6443/settings.json\e[39m\n' Generated settings file under /alabama/cspi/output_files/api-ci-op-zciz01md-10d22-cspilp-interop-ccitredhat-com:6443/settings.json + cat /alabama/cspi/output_files/api-ci-op-zciz01md-10d22-cspilp-interop-ccitredhat-com:6443/settings.json ++ oc get volumesnapshotclass -o name + for i in $(oc get volumesnapshotclass -o name) + oc annotate volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc snapshot.storage.kubernetes.io/is-default-class- volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc annotate ++ ./e2e/must-gather/get-latest-build.sh + oc get configmaps -n default must-gather-image + UPSTREAM_VERSION=99.0.0 ++ oc get OperatorCondition -n openshift-adp -o 'jsonpath={.items[*].metadata.name}' ++ awk -F v '{print $2}' + OADP_VERSION=1.3.1 + '[' -z 1.3.1 ']' + '[' 1.3.1 == 99.0.0 ']' ++ oc get sub redhat-oadp-operator -n openshift-adp -o 'jsonpath={.spec.source}' + OADP_REPO=redhat-operators + '[' -z redhat-operators ']' + '[' redhat-operators == redhat-operators ']' + REGISTRY_PATH=registry.redhat.io/oadp/oadp-mustgather-rhel9: + TAG=1.3.1 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 + echo registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 + export MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 + MUST_GATHER_BUILD=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 + '[' -z registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 ']' + export NUM_OF_OADP_INSTANCES=1 + NUM_OF_OADP_INSTANCES=1 + ginkgo run --flake-attempts=3 -p --nodes=1 -mod=mod e2e/ --show-node-events -- -credentials_file=/tmp/test-settings/credentials -vsl_credentials_file=/tmp/test-settings/aws_vsl_creds -oadp_namespace=openshift-adp -settings=/alabama/cspi/output_files/api-ci-op-zciz01md-10d22-cspilp-interop-ccitredhat-com:6443/settings.json -must_gather_image=registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 -timeout_multiplier=2 --ginkgo.junit-report=/alabama/cspi/e2e/junit_report.xml '--ginkgo.label-filter=!// || (// && !exclude_aws && (!/target/ || target_aws) ) ' --ginkgo.focus=interop Ginkgo detected a version mismatch between the Ginkgo CLI and the version of Ginkgo imported by your packages: Ginkgo CLI Version: 2.17.1 Mismatched package versions found: 2.11.0 used by e2e Ginkgo will continue to attempt to run but you may see errors (including flag parsing errors) and should either update your go.mod or your version of the Ginkgo CLI to match. To install the matching version of the CLI run go install github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file. Alternatively you can use go run github.com/onsi/ginkgo/v2/ginkgo from a path that contains a go.mod file to invoke the matching version of the Ginkgo CLI. If you are attempting to test multiple packages that each have a different version of the Ginkgo library with a single Ginkgo CLI that is currently unsupported.  go: downloading github.com/onsi/gomega v1.27.8 go: downloading github.com/openshift/oadp-operator v1.0.2-0.20231003152846-925e5cc16370 go: downloading github.com/onsi/ginkgo/v2 v2.11.0 go: downloading k8s.io/apimachinery v0.25.6 go: downloading github.com/vmware-tanzu/velero v1.12.0 go: downloading k8s.io/api v0.25.6 go: downloading k8s.io/client-go v0.25.6 go: downloading sigs.k8s.io/controller-runtime v0.12.2 go: downloading k8s.io/utils v0.0.0-20230115233650-391b47cb4029 go: downloading github.com/kubernetes-csi/external-snapshotter/client/v4 v4.2.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/operator-framework/api v0.14.1-0.20220413143725-33310d6154f3 go: downloading github.com/andygrunwald/go-jira v1.16.0 go: downloading github.com/apenella/go-ansible v1.1.5 go: downloading github.com/backube/volsync v0.7.0 go: downloading github.com/konveyor/volume-snapshot-mover v0.0.0-20230510202920-14b0e79dbaaa go: downloading github.com/openshift/api v0.0.0-20230414143018-3367bc7e6ac7 go: downloading github.com/google/uuid v1.3.0 go: downloading github.com/openshift/client-go v0.0.0-20211209144617-7385dd6338e3 go: downloading sigs.k8s.io/yaml v1.3.0 go: downloading k8s.io/kubectl v0.25.6 go: downloading github.com/google/go-cmp v0.5.9 go: downloading github.com/go-logr/logr v1.2.4 go: downloading github.com/evanphx/json-patch v5.6.0+incompatible go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/gofuzz v1.2.0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.90.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.3 go: downloading github.com/fatih/structs v1.1.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading github.com/google/go-querystring v1.1.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/trivago/tgo v1.0.7 go: downloading github.com/apenella/go-common-utils/data v0.0.0-20210528133155-34ba915e28c8 go: downloading github.com/apenella/go-common-utils/error v0.0.0-20210528133155-34ba915e28c8 go: downloading github.com/sirupsen/logrus v1.9.0 go: downloading github.com/imdario/mergo v0.3.13 go: downloading github.com/spf13/pflag v1.0.5 go: downloading golang.org/x/term v0.10.0 go: downloading golang.org/x/net v0.12.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading golang.org/x/sys v0.10.0 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/kube-openapi v0.0.0-20230118215034-64b6bb138190 go: downloading golang.org/x/time v0.3.0 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/golang/protobuf v1.5.3 go: downloading github.com/google/gnostic v0.6.9 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading golang.org/x/oauth2 v0.7.0 go: downloading golang.org/x/text v0.11.0 go: downloading google.golang.org/protobuf v1.30.0 go: downloading github.com/moby/spdystream v0.2.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/spf13/cobra v1.6.1 go: downloading k8s.io/cli-runtime v0.25.6 go: downloading k8s.io/component-base v0.26.1 go: downloading github.com/emicklei/go-restful/v3 v3.10.1 go: downloading github.com/go-openapi/swag v0.22.3 go: downloading github.com/go-openapi/jsonreference v0.20.2 go: downloading github.com/jonboulle/clockwork v0.2.2 go: downloading k8s.io/component-helpers v0.26.1 go: downloading github.com/daviddengcn/go-colortext v1.0.0 go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de go: downloading github.com/docker/distribution v2.8.1+incompatible go: downloading github.com/moby/term v0.0.0-20220808134915-39b0c02b01ae go: downloading github.com/fvbommel/sortorder v1.0.1 go: downloading sigs.k8s.io/kustomize/kustomize/v4 v4.5.7 go: downloading sigs.k8s.io/kustomize/kyaml v0.13.9 go: downloading github.com/lithammer/dedent v1.1.0 go: downloading k8s.io/metrics v0.26.1 go: downloading github.com/chai2010/gettext-go v1.0.2 go: downloading github.com/MakeNowJust/heredoc v1.0.0 go: downloading github.com/mitchellh/go-wordwrap v1.0.0 go: downloading github.com/russross/blackfriday v1.5.2 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/go-openapi/jsonpointer v0.19.6 go: downloading k8s.io/apiextensions-apiserver v0.26.1 go: downloading github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d go: downloading k8s.io/kube-aggregator v0.19.12 go: downloading github.com/fatih/camelcase v1.0.0 go: downloading github.com/bombsimon/logrusr/v3 v3.0.0 go: downloading github.com/prometheus/client_golang v1.15.0 go: downloading sigs.k8s.io/kustomize/api v0.12.1 go: downloading github.com/opencontainers/go-digest v1.0.0 go: downloading google.golang.org/grpc v1.54.0 go: downloading github.com/hashicorp/go-plugin v1.4.3 go: downloading github.com/gobwas/glob v0.2.3 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/robfig/cron v1.1.0 go: downloading github.com/hashicorp/go-hclog v1.2.0 go: downloading github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f go: downloading github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 go: downloading github.com/peterbourgon/diskv v2.0.1+incompatible go: downloading github.com/prometheus/client_model v0.3.0 go: downloading github.com/prometheus/common v0.42.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.2.0 go: downloading github.com/prometheus/procfs v0.9.0 go: downloading github.com/Azure/azure-sdk-for-go v67.2.0+incompatible go: downloading github.com/Azure/go-autorest/autorest/azure/auth v0.5.8 go: downloading github.com/Azure/go-autorest/autorest v0.11.27 go: downloading github.com/aws/aws-sdk-go v1.44.253 go: downloading github.com/joho/godotenv v1.3.0 go: downloading gomodules.xyz/jsonpatch/v2 v2.2.0 go: downloading github.com/Azure/go-autorest v14.2.0+incompatible go: downloading github.com/fatih/color v1.15.0 go: downloading github.com/mattn/go-isatty v0.0.17 go: downloading github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb go: downloading github.com/mitchellh/go-testing-interface v1.0.0 go: downloading github.com/oklog/run v1.0.0 go: downloading github.com/go-errors/errors v1.0.1 go: downloading github.com/google/btree v1.0.1 go: downloading github.com/kopia/kopia v0.13.0 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4 go: downloading github.com/fsnotify/fsnotify v1.6.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/Azure/go-autorest/logger v0.2.1 go: downloading github.com/Azure/go-autorest/tracing v0.6.0 go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.20 go: downloading github.com/Azure/go-autorest/autorest/azure/cli v0.4.2 go: downloading github.com/dimchansky/utfbom v1.1.1 go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 go: downloading github.com/xlab/treeprint v1.1.0 go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0 go: downloading github.com/mitchellh/go-homedir v1.1.0 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading golang.org/x/crypto v0.11.0 go: downloading google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 go: downloading go.uber.org/zap v1.24.0 go: downloading go.opentelemetry.io/otel v1.14.0 go: downloading go.opentelemetry.io/otel/trace v1.14.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/klauspost/compress v1.16.5 go: downloading github.com/klauspost/pgzip v1.2.5 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/gofrs/flock v0.8.1 go: downloading golang.org/x/sync v0.3.0 go: downloading github.com/edsrzf/mmap-go v1.1.0 go: downloading go.uber.org/atomic v1.10.0 go: downloading github.com/zeebo/blake3 v0.2.3 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.3.0 go: downloading cloud.google.com/go/storage v1.30.1 go: downloading google.golang.org/api v0.120.0 go: downloading github.com/minio/minio-go/v7 v7.0.52 go: downloading github.com/chmduquesne/rollinghash v4.0.0+incompatible go: downloading cloud.google.com/go v0.110.0 go: downloading github.com/natefinch/atomic v1.0.1 go: downloading golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb go: downloading github.com/klauspost/reedsolomon v1.11.7 go: downloading go.starlark.net v0.0.0-20201006213952-227f4aabceb5 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading cloud.google.com/go/compute v1.19.0 go: downloading github.com/minio/md5-simd v1.1.2 go: downloading github.com/minio/sha256-simd v1.0.0 go: downloading gopkg.in/ini.v1 v1.67.0 go: downloading cloud.google.com/go/iam v0.13.0 go: downloading github.com/googleapis/gax-go/v2 v2.8.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 go: downloading github.com/rs/xid v1.4.0 go: downloading go.opencensus.io v0.24.0 go: downloading golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.3 go: downloading github.com/google/s2a-go v0.1.2 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading google.golang.org/appengine v1.6.7 go: downloading github.com/Azure/go-autorest/autorest/validation v0.2.0 go: downloading github.com/Azure/go-autorest/autorest/to v0.3.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v0.21.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v0.8.3 2024/04/16 22:16:59 Setting up clients I0416 22:17:00.349581 21357 request.go:690] Waited for 1.027736975s due to client-side throttling, not priority and fairness, request: GET:https://api.ci-op-zciz01md-10d22.cspilp.interop.ccitredhat.com:6443/apis/operators.coreos.com/v1alpha2?timeout=32s 2024/04/16 22:17:01 Getting default StorageClass... 2024/04/16 22:17:01 Checking default storage class count Run the command: oc get sc 2024/04/16 22:17:01 Got default StorageClass gp3-csi 2024/04/16 22:17:02 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 26m gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 26m 2024/04/16 22:17:02 Using velero prefix: velero-e2e-0cbc74da-fc3f-11ee-99f6-0a580a8231b5 2024/04/16 22:17:02 Checking default storage class count Running Suite: OADP E2E Suite - /alabama/cspi/e2e ================================================= Random Seed: 1713305743 Will run 10 of 184 specs ------------------------------ [SynchronizedBeforeSuite]  /alabama/cspi/e2e/e2e_suite_test.go:75 2024/04/16 22:17:02 Error reading VSL credentials file: open /tmp/test-settings/aws_vsl_creds: no such file or directory [SynchronizedBeforeSuite] PASSED [0.075 seconds] ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ [datamover] DataMover: Backup/Restore stateful application with CSI  [tc-id:OADP-439][interop] MySQL application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:114 2024/04/16 22:17:02 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 04/16/24 22:17:02.345 2024/04/16 22:17:02 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "20424b49-f346-4686-b769-94a659734f52", "resourceVersion": "34266", "generation": 1, "creationTimestamp": "2024-04-16T22:17:02Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:17:02Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:17:02.464 2024/04/16 22:17:02 Waiting for velero pod to be running 2024/04/16 22:17:07 pod: velero-5778f4ffcb-k864g is not yet running with status: {Pending [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotInitialized containers with incomplete status: [openshift-velero-plugin velero-plugin-for-aws kubevirt-velero-plugin velero-plugin-for-csi]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC }] 10.0.20.65 [] 2024-04-16 22:17:02 +0000 UTC [{openshift-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 0xc00076437a} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 0xc00076437b} {kubevirt-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f 0xc00076437c} {velero-plugin-for-csi {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:c8979074fe11a5d1d0f5ecb0328fabb752f01103a9c03cf34e8df06bcad27f29 0xc00076437d}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc0007643c6}] Burstable []} 2024/04/16 22:17:12 pod: velero-5778f4ffcb-k864g is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:09 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotInitialized containers with incomplete status: [velero-plugin-for-aws kubevirt-velero-plugin velero-plugin-for-csi]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC }] 10.0.20.65 10.131.0.19 [{10.131.0.19}] 2024-04-16 22:17:02 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:17:09 +0000 UTC,FinishedAt:2024-04-16 22:17:09 +0000 UTC,ContainerID:cri-o://d3e09ece8d3f908e38cb0f0b603764ddb6c8f20c4e6809a77075cc9813d2bc2d,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://d3e09ece8d3f908e38cb0f0b603764ddb6c8f20c4e6809a77075cc9813d2bc2d 0xc000516169} {velero-plugin-for-aws {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 0xc00051616a} {kubevirt-velero-plugin {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f 0xc00051616b} {velero-plugin-for-csi {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:c8979074fe11a5d1d0f5ecb0328fabb752f01103a9c03cf34e8df06bcad27f29 0xc00051616c}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc0005161a6}] Burstable []} 2024/04/16 22:17:17 pod: velero-5778f4ffcb-k864g is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:09 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotInitialized containers with incomplete status: [kubevirt-velero-plugin velero-plugin-for-csi]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC }] 10.0.20.65 10.131.0.19 [{10.131.0.19}] 2024-04-16 22:17:02 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:17:09 +0000 UTC,FinishedAt:2024-04-16 22:17:09 +0000 UTC,ContainerID:cri-o://d3e09ece8d3f908e38cb0f0b603764ddb6c8f20c4e6809a77075cc9813d2bc2d,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://d3e09ece8d3f908e38cb0f0b603764ddb6c8f20c4e6809a77075cc9813d2bc2d 0xc000963559} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:17:12 +0000 UTC,FinishedAt:2024-04-16 22:17:12 +0000 UTC,ContainerID:cri-o://022ad6f77465b8783b91c18bdf3f2ce2471ccbe47a45ee82811a49d6f0cfa88b,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:716cec49a663d87eff7eaf9ddc948d2861645d601a2e15605cb660c9f98ab489 cri-o://022ad6f77465b8783b91c18bdf3f2ce2471ccbe47a45ee82811a49d6f0cfa88b 0xc000963569} {kubevirt-velero-plugin {nil &ContainerStateRunning{StartedAt:2024-04-16 22:17:17 +0000 UTC,} nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f cri-o://1c7457785b53fc1775a901453f2c5c27c684d79b535de00bec6f6c3490222283 0xc00096356a} {velero-plugin-for-csi {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:c8979074fe11a5d1d0f5ecb0328fabb752f01103a9c03cf34e8df06bcad27f29 0xc00096356b}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc000963586}] Burstable []} 2024/04/16 22:17:22 pod: velero-5778f4ffcb-k864g is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:09 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:22 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:17:02 +0000 UTC }] 10.0.20.65 10.131.0.19 [{10.131.0.19}] 2024-04-16 22:17:02 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:17:09 +0000 UTC,FinishedAt:2024-04-16 22:17:09 +0000 UTC,ContainerID:cri-o://d3e09ece8d3f908e38cb0f0b603764ddb6c8f20c4e6809a77075cc9813d2bc2d,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://d3e09ece8d3f908e38cb0f0b603764ddb6c8f20c4e6809a77075cc9813d2bc2d 0xc000963ca9} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:17:12 +0000 UTC,FinishedAt:2024-04-16 22:17:12 +0000 UTC,ContainerID:cri-o://022ad6f77465b8783b91c18bdf3f2ce2471ccbe47a45ee82811a49d6f0cfa88b,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:716cec49a663d87eff7eaf9ddc948d2861645d601a2e15605cb660c9f98ab489 cri-o://022ad6f77465b8783b91c18bdf3f2ce2471ccbe47a45ee82811a49d6f0cfa88b 0xc000963cb9} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:17:17 +0000 UTC,FinishedAt:2024-04-16 22:17:17 +0000 UTC,ContainerID:cri-o://1c7457785b53fc1775a901453f2c5c27c684d79b535de00bec6f6c3490222283,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f cri-o://1c7457785b53fc1775a901453f2c5c27c684d79b535de00bec6f6c3490222283 0xc000963cc9} {velero-plugin-for-csi {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:17:22 +0000 UTC,FinishedAt:2024-04-16 22:17:22 +0000 UTC,ContainerID:cri-o://570dbf16b54294d4906e4fdb6187f31c18e993fefc3bf9f1a08c580703e50ebd,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:c8979074fe11a5d1d0f5ecb0328fabb752f01103a9c03cf34e8df06bcad27f29 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:9833ca3b1b53eccc9de4e117bef53bb94d5be02d89bed222ef80dcd221bc1b18 cri-o://570dbf16b54294d4906e4fdb6187f31c18e993fefc3bf9f1a08c580703e50ebd 0xc000963cd9}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc000963cef}] Burstable []} 2024/04/16 22:17:27 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:17:27.668 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/04/16 22:17:27 The 'openshift-storage' namespace does not exist 2024/04/16 22:17:27 Using default CSI driver based on infrastructure: ebs.csi.aws.com 2024/04/16 22:17:27 Snapclass 'example-snapclass' doesn't exist, creating 2024/04/16 22:17:27 Setting new default StorageClass 'gp2-csi' 2024/04/16 22:17:27 Checking default storage class count Run the command: oc get sc 2024/04/16 22:17:28 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 26m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 26m 2024/04/16 22:17:28 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 04/16/24 22:17:28.152 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] /usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ci-op-zciz01md-10d22.cspilp.interop.ccitredhat.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-439] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). FAILED - RETRYING: [localhost]: Check pod status (28 retries left). FAILED - RETRYING: [localhost]: Check pod status (27 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=19  changed=10  unreachable=0 failed=0 skipped=5  rescued=0 ignored=0 2024/04/16 22:18:06 2024-04-16 22:17:30,330 p=21432 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:17:30,330 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:17:30,593 p=21432 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:17:30,594 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:17:30,880 p=21432 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:17:30,880 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:17:30,899 p=21432 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:17:30,899 p=21432 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:17:31,332 p=21432 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:17:31,332 p=21432 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:17:31,368 p=21432 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:17:31,368 p=21432 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:17:31,395 p=21432 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:17:31,395 p=21432 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:17:31,408 p=21432 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:17:31,957 p=21432 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:17:31,957 p=21432 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:17:32,886 p=21432 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-439] *** 2024-04-16 22:17:32,887 p=21432 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:17:33,347 p=21432 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-04-16 22:17:33,347 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:17:34,510 p=21432 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-04-16 22:17:34,510 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:17:57,861 p=21432 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-04-16 22:17:57,861 p=21432 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:17:58,504 p=21432 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-04-16 22:17:58,504 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:04,198 p=21432 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:18:04,198 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:04,824 p=21432 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-04-16 22:18:04,825 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:05,727 p=21432 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-04-16 22:18:05,727 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:06,339 p=21432 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-04-16 22:18:06,339 p=21432 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:06,445 p=21432 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:18:06,446 p=21432 u=1009100000 n=ansible | localhost : ok=19 changed=10 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:18:06.496 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2024/04/16 22:18:12 2024-04-16 22:18:08,010 p=21934 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:18:08,010 p=21934 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:08,255 p=21934 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:18:08,255 p=21934 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:08,503 p=21934 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:18:08,503 p=21934 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:08,519 p=21934 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:18:08,519 p=21934 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:18:08,817 p=21934 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:18:08,817 p=21934 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:18:08,846 p=21934 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:18:08,846 p=21934 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:18:08,872 p=21934 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:18:08,872 p=21934 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:18:08,884 p=21934 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:18:09,463 p=21934 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:18:09,463 p=21934 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:18:10,635 p=21934 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-04-16 22:18:10,635 p=21934 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:18:10,989 p=21934 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:18:10,990 p=21934 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:11,453 p=21934 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-04-16 22:18:11,454 p=21934 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:12,071 p=21934 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-04-16 22:18:12,072 p=21934 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:18:12,087 p=21934 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:18:12,087 p=21934 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 STEP: Creating backup mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5 @ 04/16/24 22:18:12.142 2024/04/16 22:18:12 Wait until backup mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5 is completed backup phase: WaitingForPluginOperations DataUpload mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-tmpnm phase: Accepted DataUpload Name: mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-tmpnm and status: Accepted 2024/04/16 22:18:32 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-tmpnm", "generateName": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "a3df2355-9295-428f-8cfd-c6b936eccfa8", "resourceVersion": "35009", "generation": 2, "creationTimestamp": "2024-04-16T22:18:21Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-93715512-ad4c-4987-a341-1f818c0b2c01.ad5e9ca2-2156-466521542", "velero.io/backup-name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "93715512-ad4c-4987-a341-1f818c0b2c01", "velero.io/pvc-uid": "ad5e9ca2-2156-4666-9fda-5b5af989d024" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5", "uid": "93715512-ad4c-4987-a341-1f818c0b2c01", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:18:21Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:18:21Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"93715512-ad4c-4987-a341-1f818c0b2c01\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-j4g79", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:18:21Z", "progress": {} } } DataUpload mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-zmbxt phase: Accepted DataUpload Name: mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-zmbxt and status: Accepted 2024/04/16 22:18:32 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-zmbxt", "generateName": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "43124041-2b70-4604-a9e6-6d008fae5c9a", "resourceVersion": "35109", "generation": 2, "creationTimestamp": "2024-04-16T22:18:26Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-93715512-ad4c-4987-a341-1f818c0b2c01.119d6016-fdc7-412afc566", "velero.io/backup-name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "93715512-ad4c-4987-a341-1f818c0b2c01", "velero.io/pvc-uid": "119d6016-fdc7-4127-9e0a-3277d0ef907e" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5", "uid": "93715512-ad4c-4987-a341-1f818c0b2c01", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:18:26Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:18:26Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"93715512-ad4c-4987-a341-1f818c0b2c01\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-1-n7stg", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:18:26Z", "progress": {} } } backup phase: WaitingForPluginOperationsPartiallyFailed DataUpload mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-tmpnm phase: Failed DataUpload Name: mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-tmpnm and status: Failed 2024/04/16 22:18:52 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-tmpnm", "generateName": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "a3df2355-9295-428f-8cfd-c6b936eccfa8", "resourceVersion": "35309", "generation": 3, "creationTimestamp": "2024-04-16T22:18:21Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-93715512-ad4c-4987-a341-1f818c0b2c01.ad5e9ca2-2156-466521542", "velero.io/backup-name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "93715512-ad4c-4987-a341-1f818c0b2c01", "velero.io/pvc-uid": "ad5e9ca2-2156-4666-9fda-5b5af989d024" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5", "uid": "93715512-ad4c-4987-a341-1f818c0b2c01", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:18:21Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"93715512-ad4c-4987-a341-1f818c0b2c01\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:18:39Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-j4g79", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-fffc157f-5bf6-4939-a893-8678e2d5937c: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:18:21Z", "completionTimestamp": "2024-04-16T22:18:39Z", "progress": {} } } DataUpload mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-zmbxt phase: Accepted DataUpload Name: mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-zmbxt and status: Accepted 2024/04/16 22:18:52 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-zmbxt", "generateName": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "43124041-2b70-4604-a9e6-6d008fae5c9a", "resourceVersion": "35109", "generation": 2, "creationTimestamp": "2024-04-16T22:18:26Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-93715512-ad4c-4987-a341-1f818c0b2c01.119d6016-fdc7-412afc566", "velero.io/backup-name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "93715512-ad4c-4987-a341-1f818c0b2c01", "velero.io/pvc-uid": "119d6016-fdc7-4127-9e0a-3277d0ef907e" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5", "uid": "93715512-ad4c-4987-a341-1f818c0b2c01", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:18:26Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:18:26Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"93715512-ad4c-4987-a341-1f818c0b2c01\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-1-n7stg", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:18:26Z", "progress": {} } } backup phase: PartiallyFailed STEP: Verify backup mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:19:12.289 2024/04/16 22:19:12 Run velero describe on the backup 2024/04/16 22:19:12 Exec stderr: "" 2024/04/16 22:19:12 Name: mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: PartiallyFailed (run `velero backup logs mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5` for more information) Errors: Velero: Cluster: Namespaces: Namespaces: Included: test-oadp-439 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-04-16 22:18:12 +0000 UTC Completed: 2024-04-16 22:18:56 +0000 UTC Expiration: 2024-05-16 22:18:12 +0000 UTC Total items to be backed up: 51 Items backed up: 51 Backup Item Operations: Operation for persistentvolumeclaims test-oadp-439/mysql: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-93715512-ad4c-4987-a341-1f818c0b2c01.ad5e9ca2-2156-466521542 Items to Update: datauploads.velero.io openshift-adp/mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-tmpnm Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-fffc157f-5bf6-4939-a893-8678e2d5937c: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:18:21 +0000 UTC Started: 2024-04-16 22:18:21 +0000 UTC Updated: 2024-04-16 22:18:39 +0000 UTC Operation for persistentvolumeclaims test-oadp-439/mysql-1: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-93715512-ad4c-4987-a341-1f818c0b2c01.119d6016-fdc7-412afc566 Items to Update: datauploads.velero.io openshift-adp/mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-zmbxt Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-8ee142f8-924c-45aa-928d-08e37c3c64d2: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:18:26 +0000 UTC Started: 2024-04-16 22:18:26 +0000 UTC Updated: 2024-04-16 22:18:54 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - test-oadp-439/mysql apps/v1/ReplicaSet: - test-oadp-439/mysql-7f77d47fd7 authorization.openshift.io/v1/RoleBinding: - test-oadp-439/admin - test-oadp-439/system:deployers - test-oadp-439/system:image-builders - test-oadp-439/system:image-pullers discovery.k8s.io/v1/EndpointSlice: - test-oadp-439/mysql-4b8xf operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-439/volsync-product.v0.9.1 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-439/admin - test-oadp-439/system:deployers - test-oadp-439/system:image-builders - test-oadp-439/system:image-pullers v1/ConfigMap: - test-oadp-439/kube-root-ca.crt - test-oadp-439/openshift-service-ca.crt v1/Endpoints: - test-oadp-439/mysql v1/Event: - test-oadp-439/mysql-1.17c6e298d92d0bb2 - test-oadp-439/mysql-1.17c6e298dee996b7 - test-oadp-439/mysql-1.17c6e298deeeba67 - test-oadp-439/mysql-1.17c6e299e38ec353 - test-oadp-439/mysql-7f77d47fd7-sfxgn.17c6e29a0914421a - test-oadp-439/mysql-7f77d47fd7-sfxgn.17c6e29a92c0394a - test-oadp-439/mysql-7f77d47fd7-sfxgn.17c6e29a967f98af - test-oadp-439/mysql-7f77d47fd7-sfxgn.17c6e29c0fb13ca5 - test-oadp-439/mysql-7f77d47fd7-sfxgn.17c6e29c113bd2fd - test-oadp-439/mysql-7f77d47fd7-sfxgn.17c6e29d975c5dd7 - test-oadp-439/mysql-7f77d47fd7-sfxgn.17c6e29d9ccbd445 - test-oadp-439/mysql-7f77d47fd7-sfxgn.17c6e29d9d9da4d0 - test-oadp-439/mysql-7f77d47fd7.17c6e298dd7be1b6 - test-oadp-439/mysql.17c6e298d748d461 - test-oadp-439/mysql.17c6e298dce7a5e1 - test-oadp-439/mysql.17c6e298dec078fc - test-oadp-439/mysql.17c6e298dec25e4f - test-oadp-439/mysql.17c6e299e3c119b4 v1/Namespace: - test-oadp-439 v1/PersistentVolume: - pvc-119d6016-fdc7-4127-9e0a-3277d0ef907e - pvc-ad5e9ca2-2156-4666-9fda-5b5af989d024 v1/PersistentVolumeClaim: - test-oadp-439/mysql - test-oadp-439/mysql-1 v1/Pod: - test-oadp-439/mysql-7f77d47fd7-sfxgn v1/Secret: - test-oadp-439/builder-dockercfg-2wg4m - test-oadp-439/builder-token-pnckq - test-oadp-439/default-dockercfg-swc5r - test-oadp-439/default-token-f5xn7 - test-oadp-439/deployer-dockercfg-4ln8s - test-oadp-439/deployer-token-vpccx - test-oadp-439/mysql v1/Service: - test-oadp-439/mysql v1/ServiceAccount: - test-oadp-439/builder - test-oadp-439/default - test-oadp-439/deployer Velero-Native Snapshots: 2024/04/16 22:19:12 { "metadata": { "name": "mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5", "namespace": "openshift-adp", "uid": "93715512-ad4c-4987-a341-1f818c0b2c01", "resourceVersion": "35418", "generation": 8, "creationTimestamp": "2024-04-16T22:18:12Z", "labels": { "velero.io/storage-location": "ts-dpa-1" }, "annotations": { "velero.io/resource-timeout": "10m0s", "velero.io/source-cluster-k8s-gitversion": "v1.29.3+5fa1806", "velero.io/source-cluster-k8s-major-version": "1", "velero.io/source-cluster-k8s-minor-version": "29" }, "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:18:12Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:hooks": {}, "f:includedNamespaces": {}, "f:metadata": {}, "f:snapshotMoveData": {} }, "f:status": {} } }, { "manager": "velero-server", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:18:56Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:annotations": { ".": {}, "f:velero.io/resource-timeout": {}, "f:velero.io/source-cluster-k8s-gitversion": {}, "f:velero.io/source-cluster-k8s-major-version": {}, "f:velero.io/source-cluster-k8s-minor-version": {} }, "f:labels": { ".": {}, "f:velero.io/storage-location": {} } }, "f:spec": { "f:csiSnapshotTimeout": {}, "f:defaultVolumesToFsBackup": {}, "f:itemOperationTimeout": {}, "f:storageLocation": {}, "f:ttl": {} }, "f:status": { "f:backupItemOperationsAttempted": {}, "f:backupItemOperationsFailed": {}, "f:completionTimestamp": {}, "f:errors": {}, "f:expiration": {}, "f:formatVersion": {}, "f:phase": {}, "f:progress": { ".": {}, "f:itemsBackedUp": {}, "f:totalItems": {} }, "f:startTimestamp": {}, "f:version": {} } } } ] }, "spec": { "metadata": {}, "includedNamespaces": [ "test-oadp-439" ], "ttl": "720h0m0s", "hooks": {}, "storageLocation": "ts-dpa-1", "defaultVolumesToFsBackup": false, "csiSnapshotTimeout": "10m0s", "itemOperationTimeout": "4h0m0s", "snapshotMoveData": true }, "status": { "version": 1, "formatVersion": "1.1.0", "expiration": "2024-05-16T22:18:12Z", "phase": "PartiallyFailed", "startTimestamp": "2024-04-16T22:18:12Z", "completionTimestamp": "2024-04-16T22:18:56Z", "errors": 2, "progress": { "totalItems": 51, "itemsBackedUp": 51 }, "backupItemOperationsAttempted": 2, "backupItemOperationsFailed": 2 } } [FAILED] in [It] - /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 @ 04/16/24 22:19:12.913 2024/04/16 22:19:12 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 STEP: Get the failed spec name @ 04/16/24 22:19:12.913 2024/04/16 22:19:12 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-439][interop] MySQL application STEP: Create a folder for all must-gather files if it doesn't exists already @ 04/16/24 22:19:12.913 2024/04/16 22:19:12 The folder logs does not exists, creating new folder with the name: logs STEP: Create a folder for the failed spec if it doesn't exists already @ 04/16/24 22:19:12.913 2024/04/16 22:19:12 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_MySQL_application does not exists, creating new folder with the name: logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_MySQL_application STEP: Run must-gather because the spec failed @ 04/16/24 22:19:12.913 2024/04/16 22:19:12 [adm must-gather --dest-dir logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_MySQL_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1] STEP: Find must-gather folder and rename it to a shorter more readable name @ 04/16/24 22:19:43.263 2024/04/16 22:19:43 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:19:57 2024-04-16 22:19:44,875 p=22275 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:19:44,876 p=22275 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:19:45,122 p=22275 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:19:45,122 p=22275 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:19:45,377 p=22275 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:19:45,378 p=22275 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:19:45,393 p=22275 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:19:45,393 p=22275 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:19:45,690 p=22275 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:19:45,690 p=22275 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:19:45,728 p=22275 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:19:45,728 p=22275 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:19:45,754 p=22275 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:19:45,754 p=22275 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:19:45,770 p=22275 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:19:46,329 p=22275 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:19:46,330 p=22275 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:19:57,261 p=22275 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** 2024-04-16 22:19:57,261 p=22275 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:19:57,585 p=22275 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:19:57,585 p=22275 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:19:57 Cleaning setup resources for the backup 2024/04/16 22:19:57 Setting new default StorageClass 'gp3-csi' 2024/04/16 22:19:57 Checking default storage class count Run the command: oc get sc 2024/04/16 22:19:57 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 29m gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 29m 2024/04/16 22:19:57 Deleting VolumeSnapshotClass 'example-snapclass' Attempt #1 Failed. Retrying ↺ @ 04/16/24 22:19:57.88 2024/04/16 22:19:57 Delete all downloadrequest mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-2cbccd72-9185-4649-82f3-6b786dbd56fa mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-3d33c4e7-89c3-4a5b-8314-a625c0a36a7e mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-94e70f31-5202-4159-9aec-8e7369781171 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-af28f071-161a-4295-ab46-94161bfd7834 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-d95abb9f-01c6-47af-831f-5196e3ca5c14 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-dda3f0ff-1e7b-4a1a-9d85-9024645ef4d9 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-fc5de649-d117-4531-acea-fc309ac9dce6 STEP: Create DPA CR @ 04/16/24 22:19:58.095 2024/04/16 22:19:58 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "32c377ea-4030-4f05-906f-bf35872d48fe", "resourceVersion": "35947", "generation": 1, "creationTimestamp": "2024-04-16T22:19:58Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:19:58Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:19:58.184 2024/04/16 22:19:58 Waiting for velero pod to be running 2024/04/16 22:19:58 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' 2024/04/16 22:19:58 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "32c377ea-4030-4f05-906f-bf35872d48fe", "resourceVersion": "35947", "generation": 1, "creationTimestamp": "2024-04-16T22:19:58Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:19:58Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:20:03.254 2024/04/16 22:20:03 Snapclass 'example-snapclass' doesn't exist, creating 2024/04/16 22:20:03 Setting new default StorageClass 'gp2-csi' 2024/04/16 22:20:03 Checking default storage class count Run the command: oc get sc 2024/04/16 22:20:03 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 29m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 29m 2024/04/16 22:20:03 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 04/16/24 22:20:03.568 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-439] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). FAILED - RETRYING: [localhost]: Check pod status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=19  changed=10  unreachable=0 failed=0 skipped=5  rescued=0 ignored=0 2024/04/16 22:20:29 2024-04-16 22:20:05,130 p=22519 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:20:05,131 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:05,385 p=22519 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:20:05,386 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:05,631 p=22519 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:20:05,632 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:05,648 p=22519 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:20:05,649 p=22519 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:05,965 p=22519 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:20:05,966 p=22519 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:05,996 p=22519 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:20:05,997 p=22519 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:06,023 p=22519 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:20:06,023 p=22519 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:06,035 p=22519 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:20:06,587 p=22519 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:20:06,587 p=22519 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:07,455 p=22519 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-439] *** 2024-04-16 22:20:07,455 p=22519 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:07,900 p=22519 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-04-16 22:20:07,900 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:08,973 p=22519 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-04-16 22:20:08,973 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:26,584 p=22519 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-04-16 22:20:26,585 p=22519 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:27,189 p=22519 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-04-16 22:20:27,189 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:27,529 p=22519 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:20:27,529 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:28,241 p=22519 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-04-16 22:20:28,241 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:29,120 p=22519 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-04-16 22:20:29,121 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:29,745 p=22519 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-04-16 22:20:29,745 p=22519 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:29,861 p=22519 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:20:29,862 p=22519 u=1009100000 n=ansible | localhost : ok=19 changed=10 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:20:29.913 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2024/04/16 22:20:35 2024-04-16 22:20:31,453 p=22979 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:20:31,453 p=22979 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:31,715 p=22979 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:20:31,715 p=22979 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:31,970 p=22979 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:20:31,971 p=22979 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:31,986 p=22979 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:20:31,986 p=22979 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:32,300 p=22979 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:20:32,300 p=22979 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:32,331 p=22979 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:20:32,331 p=22979 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:32,357 p=22979 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:20:32,357 p=22979 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:32,369 p=22979 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:20:32,939 p=22979 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:20:32,939 p=22979 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:34,087 p=22979 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-04-16 22:20:34,087 p=22979 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:20:34,439 p=22979 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:20:34,439 p=22979 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:34,904 p=22979 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-04-16 22:20:34,904 p=22979 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:35,512 p=22979 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-04-16 22:20:35,513 p=22979 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:20:35,532 p=22979 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:20:35,532 p=22979 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 STEP: Creating backup mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5 @ 04/16/24 22:20:35.583 2024/04/16 22:20:35 Wait until backup mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5 is completed backup phase: WaitingForPluginOperations DataUpload mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r phase: Accepted DataUpload Name: mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r and status: Accepted 2024/04/16 22:20:55 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r", "generateName": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "5ca0cddc-a596-41eb-a2ff-09e6b727a1f8", "resourceVersion": "36597", "generation": 2, "creationTimestamp": "2024-04-16T22:20:45Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-7b05df9f-6ee7-4044-924a-39e934328f2e.98637d77-9f13-4eecb4b68", "velero.io/backup-name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "velero.io/pvc-uid": "98637d77-9f13-4eee-af45-a04adbe2a80b" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:45Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:45Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"7b05df9f-6ee7-4044-924a-39e934328f2e\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-fhgd9", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:20:45Z", "progress": {} } } DataUpload mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4 phase: Accepted DataUpload Name: mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4 and status: Accepted 2024/04/16 22:20:55 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4", "generateName": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "eb55834b-c4ed-4c4e-bfb3-986d65e5f8c7", "resourceVersion": "36702", "generation": 2, "creationTimestamp": "2024-04-16T22:20:50Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-7b05df9f-6ee7-4044-924a-39e934328f2e.fd8b0382-3522-48907dbe9", "velero.io/backup-name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "velero.io/pvc-uid": "fd8b0382-3522-4892-8165-6cfd3340ac75" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:50Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:50Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"7b05df9f-6ee7-4044-924a-39e934328f2e\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-1-t6r95", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:20:50Z", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r phase: Accepted DataUpload Name: mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r and status: Accepted 2024/04/16 22:21:15 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r", "generateName": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "5ca0cddc-a596-41eb-a2ff-09e6b727a1f8", "resourceVersion": "36597", "generation": 2, "creationTimestamp": "2024-04-16T22:20:45Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-7b05df9f-6ee7-4044-924a-39e934328f2e.98637d77-9f13-4eecb4b68", "velero.io/backup-name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "velero.io/pvc-uid": "98637d77-9f13-4eee-af45-a04adbe2a80b" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:45Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:45Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"7b05df9f-6ee7-4044-924a-39e934328f2e\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-fhgd9", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:20:45Z", "progress": {} } } DataUpload mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4 phase: Accepted DataUpload Name: mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4 and status: Accepted 2024/04/16 22:21:15 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4", "generateName": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "eb55834b-c4ed-4c4e-bfb3-986d65e5f8c7", "resourceVersion": "36702", "generation": 2, "creationTimestamp": "2024-04-16T22:20:50Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-7b05df9f-6ee7-4044-924a-39e934328f2e.fd8b0382-3522-48907dbe9", "velero.io/backup-name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "velero.io/pvc-uid": "fd8b0382-3522-4892-8165-6cfd3340ac75" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:50Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:50Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"7b05df9f-6ee7-4044-924a-39e934328f2e\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-1-t6r95", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:20:50Z", "progress": {} } } backup phase: WaitingForPluginOperationsPartiallyFailed DataUpload mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r phase: Accepted DataUpload Name: mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r and status: Accepted 2024/04/16 22:21:35 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r", "generateName": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "5ca0cddc-a596-41eb-a2ff-09e6b727a1f8", "resourceVersion": "36597", "generation": 2, "creationTimestamp": "2024-04-16T22:20:45Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-7b05df9f-6ee7-4044-924a-39e934328f2e.98637d77-9f13-4eecb4b68", "velero.io/backup-name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "velero.io/pvc-uid": "98637d77-9f13-4eee-af45-a04adbe2a80b" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:45Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:45Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"7b05df9f-6ee7-4044-924a-39e934328f2e\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-fhgd9", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:20:45Z", "progress": {} } } DataUpload mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4 phase: Failed DataUpload Name: mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4 and status: Failed 2024/04/16 22:21:35 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4", "generateName": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "eb55834b-c4ed-4c4e-bfb3-986d65e5f8c7", "resourceVersion": "37341", "generation": 3, "creationTimestamp": "2024-04-16T22:20:50Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-7b05df9f-6ee7-4044-924a-39e934328f2e.fd8b0382-3522-48907dbe9", "velero.io/backup-name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "velero.io/pvc-uid": "fd8b0382-3522-4892-8165-6cfd3340ac75" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:20:50Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"7b05df9f-6ee7-4044-924a-39e934328f2e\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:21:24Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-1-t6r95", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-8c2c5bc3-7c83-4b44-a49f-6d6abfefa63d: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:20:50Z", "completionTimestamp": "2024-04-16T22:21:24Z", "progress": {} } } backup phase: PartiallyFailed STEP: Verify backup mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:21:55.802 2024/04/16 22:21:55 Run velero describe on the backup 2024/04/16 22:21:56 Exec stderr: "" 2024/04/16 22:21:56 Name: mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: PartiallyFailed (run `velero backup logs mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5` for more information) Errors: Velero: Cluster: Namespaces: Namespaces: Included: test-oadp-439 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-04-16 22:20:35 +0000 UTC Completed: 2024-04-16 22:21:49 +0000 UTC Expiration: 2024-05-16 22:20:35 +0000 UTC Total items to be backed up: 50 Items backed up: 50 Backup Item Operations: Operation for persistentvolumeclaims test-oadp-439/mysql: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-7b05df9f-6ee7-4044-924a-39e934328f2e.98637d77-9f13-4eecb4b68 Items to Update: datauploads.velero.io openshift-adp/mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-jnk8r Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-8d1e890d-b06e-4b97-a838-4cdade3a851b: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:20:45 +0000 UTC Started: 2024-04-16 22:20:45 +0000 UTC Updated: 2024-04-16 22:21:41 +0000 UTC Operation for persistentvolumeclaims test-oadp-439/mysql-1: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-7b05df9f-6ee7-4044-924a-39e934328f2e.fd8b0382-3522-48907dbe9 Items to Update: datauploads.velero.io openshift-adp/mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-k4nz4 Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-8c2c5bc3-7c83-4b44-a49f-6d6abfefa63d: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:20:50 +0000 UTC Started: 2024-04-16 22:20:50 +0000 UTC Updated: 2024-04-16 22:21:24 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - test-oadp-439/mysql apps/v1/ReplicaSet: - test-oadp-439/mysql-7f77d47fd7 authorization.openshift.io/v1/RoleBinding: - test-oadp-439/admin - test-oadp-439/system:deployers - test-oadp-439/system:image-builders - test-oadp-439/system:image-pullers discovery.k8s.io/v1/EndpointSlice: - test-oadp-439/mysql-tg2ph operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-439/volsync-product.v0.9.1 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-439/admin - test-oadp-439/system:deployers - test-oadp-439/system:image-builders - test-oadp-439/system:image-pullers v1/ConfigMap: - test-oadp-439/kube-root-ca.crt - test-oadp-439/openshift-service-ca.crt v1/Endpoints: - test-oadp-439/mysql v1/Event: - test-oadp-439/mysql-1.17c6e2bcd1010750 - test-oadp-439/mysql-1.17c6e2bcd6b425b3 - test-oadp-439/mysql-1.17c6e2bcd6bd708b - test-oadp-439/mysql-1.17c6e2bdd7141ce9 - test-oadp-439/mysql-7f77d47fd7-h7cwv.17c6e2be012d3975 - test-oadp-439/mysql-7f77d47fd7-h7cwv.17c6e2be672515fc - test-oadp-439/mysql-7f77d47fd7-h7cwv.17c6e2be698e3ce7 - test-oadp-439/mysql-7f77d47fd7-h7cwv.17c6e2bfc00aae09 - test-oadp-439/mysql-7f77d47fd7-h7cwv.17c6e2bfc147fd02 - test-oadp-439/mysql-7f77d47fd7-h7cwv.17c6e2bfc677d4c5 - test-oadp-439/mysql-7f77d47fd7-h7cwv.17c6e2bfc753e9af - test-oadp-439/mysql-7f77d47fd7.17c6e2bcd506fa3e - test-oadp-439/mysql.17c6e2bccf36fb78 - test-oadp-439/mysql.17c6e2bcd39cb30c - test-oadp-439/mysql.17c6e2bcd61f63f0 - test-oadp-439/mysql.17c6e2bcd62a8acc - test-oadp-439/mysql.17c6e2bdd723df77 v1/Namespace: - test-oadp-439 v1/PersistentVolume: - pvc-98637d77-9f13-4eee-af45-a04adbe2a80b - pvc-fd8b0382-3522-4892-8165-6cfd3340ac75 v1/PersistentVolumeClaim: - test-oadp-439/mysql - test-oadp-439/mysql-1 v1/Pod: - test-oadp-439/mysql-7f77d47fd7-h7cwv v1/Secret: - test-oadp-439/builder-dockercfg-qlcsx - test-oadp-439/builder-token-zldb5 - test-oadp-439/default-dockercfg-t8m65 - test-oadp-439/default-token-kjxjs - test-oadp-439/deployer-dockercfg-6klg5 - test-oadp-439/deployer-token-kxwrn - test-oadp-439/mysql v1/Service: - test-oadp-439/mysql v1/ServiceAccount: - test-oadp-439/builder - test-oadp-439/default - test-oadp-439/deployer Velero-Native Snapshots: 2024/04/16 22:21:56 { "metadata": { "name": "mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5", "namespace": "openshift-adp", "uid": "7b05df9f-6ee7-4044-924a-39e934328f2e", "resourceVersion": "37653", "generation": 8, "creationTimestamp": "2024-04-16T22:20:35Z", "labels": { "velero.io/storage-location": "ts-dpa-1" }, "annotations": { "velero.io/resource-timeout": "10m0s", "velero.io/source-cluster-k8s-gitversion": "v1.29.3+5fa1806", "velero.io/source-cluster-k8s-major-version": "1", "velero.io/source-cluster-k8s-minor-version": "29" }, "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:20:35Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:hooks": {}, "f:includedNamespaces": {}, "f:metadata": {}, "f:snapshotMoveData": {} }, "f:status": {} } }, { "manager": "velero-server", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:21:50Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:annotations": { ".": {}, "f:velero.io/resource-timeout": {}, "f:velero.io/source-cluster-k8s-gitversion": {}, "f:velero.io/source-cluster-k8s-major-version": {}, "f:velero.io/source-cluster-k8s-minor-version": {} }, "f:labels": { ".": {}, "f:velero.io/storage-location": {} } }, "f:spec": { "f:csiSnapshotTimeout": {}, "f:defaultVolumesToFsBackup": {}, "f:itemOperationTimeout": {}, "f:storageLocation": {}, "f:ttl": {} }, "f:status": { "f:backupItemOperationsAttempted": {}, "f:backupItemOperationsFailed": {}, "f:completionTimestamp": {}, "f:errors": {}, "f:expiration": {}, "f:formatVersion": {}, "f:phase": {}, "f:progress": { ".": {}, "f:itemsBackedUp": {}, "f:totalItems": {} }, "f:startTimestamp": {}, "f:version": {} } } } ] }, "spec": { "metadata": {}, "includedNamespaces": [ "test-oadp-439" ], "ttl": "720h0m0s", "hooks": {}, "storageLocation": "ts-dpa-1", "defaultVolumesToFsBackup": false, "csiSnapshotTimeout": "10m0s", "itemOperationTimeout": "4h0m0s", "snapshotMoveData": true }, "status": { "version": 1, "formatVersion": "1.1.0", "expiration": "2024-05-16T22:20:35Z", "phase": "PartiallyFailed", "startTimestamp": "2024-04-16T22:20:35Z", "completionTimestamp": "2024-04-16T22:21:49Z", "errors": 2, "progress": { "totalItems": 50, "itemsBackedUp": 50 }, "backupItemOperationsAttempted": 2, "backupItemOperationsFailed": 2 } } [FAILED] in [It] - /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 @ 04/16/24 22:21:56.374 2024/04/16 22:21:56 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 STEP: Get the failed spec name @ 04/16/24 22:21:56.374 2024/04/16 22:21:56 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-439][interop] MySQL application STEP: Create a folder for all must-gather files if it doesn't exists already @ 04/16/24 22:21:56.374 STEP: Create a folder for the failed spec if it doesn't exists already @ 04/16/24 22:21:56.374 STEP: Run must-gather because the spec failed @ 04/16/24 22:21:56.374 2024/04/16 22:21:56 [adm must-gather --dest-dir logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_MySQL_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1] STEP: Find must-gather folder and rename it to a shorter more readable name @ 04/16/24 22:22:07.362 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_MySQL_application/must-gather already exists, skipping renaming the folder 2024/04/16 22:22:07 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:22:26 2024-04-16 22:22:08,890 p=23321 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:22:08,890 p=23321 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:22:09,141 p=23321 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:22:09,141 p=23321 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:22:09,392 p=23321 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:22:09,393 p=23321 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:22:09,410 p=23321 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:22:09,410 p=23321 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:09,718 p=23321 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:22:09,718 p=23321 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:09,747 p=23321 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:22:09,747 p=23321 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:09,772 p=23321 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:22:09,773 p=23321 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:09,784 p=23321 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:22:10,346 p=23321 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:22:10,347 p=23321 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:26,281 p=23321 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** 2024-04-16 22:22:26,281 p=23321 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:22:26,589 p=23321 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:22:26,589 p=23321 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:22:26 Cleaning setup resources for the backup 2024/04/16 22:22:26 Setting new default StorageClass 'gp3-csi' 2024/04/16 22:22:26 Checking default storage class count Run the command: oc get sc 2024/04/16 22:22:26 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 31m gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 31m 2024/04/16 22:22:26 Deleting VolumeSnapshotClass 'example-snapclass' Attempt #2 Failed. Retrying ↺ @ 04/16/24 22:22:26.878 2024/04/16 22:22:26 Delete all downloadrequest mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-2fd9184d-598b-4316-beeb-c2ea318d2714 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-3abb67dc-e87f-4852-9840-6222cc55a369 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-956136ea-50f8-49e9-8857-fb97d9c2932e mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-e7ed29cd-ac87-4ab3-8361-96ee6720fa26 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-054bf3e9-4c8a-4de1-946c-0c03b29c404c mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-0f4622b9-c6b8-4b17-a1fb-1cbb00949b36 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-6610d232-8eff-472b-b57b-41cdef53561c mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-6b4f3834-e3d0-48b7-84ce-24bb3669df0a mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-735636fd-ce57-45c2-a2a6-a15abe6ac490 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-93bf3093-3fc2-4da7-b1c8-d26e18fe3eb7 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-a7517a92-f281-4116-9316-4b7db7d31f85 STEP: Create DPA CR @ 04/16/24 22:22:27.154 2024/04/16 22:22:27 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "f3bf5677-ffa6-4675-a189-eb771e6fd337", "resourceVersion": "38047", "generation": 1, "creationTimestamp": "2024-04-16T22:22:27Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:22:27Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:22:27.232 2024/04/16 22:22:27 Waiting for velero pod to be running 2024/04/16 22:22:27 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:22:27.302 2024/04/16 22:22:27 Snapclass 'example-snapclass' doesn't exist, creating 2024/04/16 22:22:27 Setting new default StorageClass 'gp2-csi' 2024/04/16 22:22:27 Checking default storage class count Run the command: oc get sc 2024/04/16 22:22:27 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 31m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 31m 2024/04/16 22:22:27 Checking for correct number of running NodeAgent pods... 2024/04/16 22:22:27 pod: node-agent-d9mfw is not yet running with status: {Pending [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:22:27 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:22:27 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:22:27 +0000 UTC ContainersNotReady containers with unready status: [node-agent]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:22:27 +0000 UTC ContainersNotReady containers with unready status: [node-agent]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:22:27 +0000 UTC }] 10.0.106.243 [] 2024-04-16 22:22:27 +0000 UTC [] [{node-agent {&ContainerStateWaiting{Reason:ContainerCreating,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc00039a9ba}] Burstable []} STEP: Installing application for case mysql @ 04/16/24 22:22:32.698 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-439] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). FAILED - RETRYING: [localhost]: Check pod status (28 retries left). FAILED - RETRYING: [localhost]: Check pod status (27 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=19  changed=10  unreachable=0 failed=0 skipped=5  rescued=0 ignored=0 2024/04/16 22:23:04 2024-04-16 22:22:34,227 p=23565 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:22:34,228 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:22:34,479 p=23565 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:22:34,479 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:22:34,726 p=23565 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:22:34,726 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:22:34,743 p=23565 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:22:34,743 p=23565 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:35,044 p=23565 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:22:35,045 p=23565 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:35,072 p=23565 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:22:35,073 p=23565 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:35,097 p=23565 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:22:35,097 p=23565 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:35,109 p=23565 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:22:35,665 p=23565 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:22:35,665 p=23565 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:36,517 p=23565 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-439] *** 2024-04-16 22:22:36,518 p=23565 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:22:36,941 p=23565 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-04-16 22:22:36,941 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:22:37,959 p=23565 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-04-16 22:22:37,960 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:01,212 p=23565 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-04-16 22:23:01,213 p=23565 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:23:01,862 p=23565 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-04-16 22:23:01,862 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:02,222 p=23565 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:23:02,222 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:02,886 p=23565 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-04-16 22:23:02,886 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:03,782 p=23565 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-04-16 22:23:03,782 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:04,403 p=23565 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-04-16 22:23:04,404 p=23565 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:04,512 p=23565 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:23:04,512 p=23565 u=1009100000 n=ansible | localhost : ok=19 changed=10 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:23:04.57 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2024/04/16 22:23:10 2024-04-16 22:23:06,110 p=24040 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:23:06,110 p=24040 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:06,354 p=24040 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:23:06,354 p=24040 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:06,604 p=24040 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:23:06,604 p=24040 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:06,619 p=24040 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:23:06,619 p=24040 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:23:06,931 p=24040 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:23:06,931 p=24040 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:23:06,960 p=24040 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:23:06,961 p=24040 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:23:06,988 p=24040 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:23:06,989 p=24040 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:23:07,000 p=24040 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:23:07,556 p=24040 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:23:07,556 p=24040 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:23:08,716 p=24040 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-04-16 22:23:08,716 p=24040 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:23:09,084 p=24040 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:23:09,084 p=24040 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:09,596 p=24040 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-04-16 22:23:09,596 p=24040 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:10,229 p=24040 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-04-16 22:23:10,229 p=24040 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:23:10,248 p=24040 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:23:10,248 p=24040 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 STEP: Creating backup mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5 @ 04/16/24 22:23:10.305 2024/04/16 22:23:10 Wait until backup mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5 is completed backup phase: WaitingForPluginOperations DataUpload mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc phase: Accepted DataUpload Name: mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc and status: Accepted 2024/04/16 22:23:30 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc", "generateName": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "fdba7baa-a322-4ac7-bd6c-3af8059e5465", "resourceVersion": "38725", "generation": 2, "creationTimestamp": "2024-04-16T22:23:19Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-6dbe21ab-a456-43e3-a0a1-764d67d95b27.9ad339d8-24fa-47d898518", "velero.io/backup-name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "velero.io/pvc-uid": "9ad339d8-24fa-47d4-b9e4-0c5760f26659" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"6dbe21ab-a456-43e3-a0a1-764d67d95b27\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-6752q", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:23:19Z", "progress": {} } } DataUpload mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr phase: Accepted DataUpload Name: mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr and status: Accepted 2024/04/16 22:23:30 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr", "generateName": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "cc328e65-05dd-448b-95d0-6567c41f35b5", "resourceVersion": "38819", "generation": 2, "creationTimestamp": "2024-04-16T22:23:24Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-6dbe21ab-a456-43e3-a0a1-764d67d95b27.1317a045-5b4c-46cb18099", "velero.io/backup-name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "velero.io/pvc-uid": "1317a045-5b4c-46c8-b987-66c14a61b9b7" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:24Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:24Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"6dbe21ab-a456-43e3-a0a1-764d67d95b27\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-1-sswcx", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:23:24Z", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc phase: Accepted DataUpload Name: mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc and status: Accepted 2024/04/16 22:23:50 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc", "generateName": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "fdba7baa-a322-4ac7-bd6c-3af8059e5465", "resourceVersion": "38725", "generation": 2, "creationTimestamp": "2024-04-16T22:23:19Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-6dbe21ab-a456-43e3-a0a1-764d67d95b27.9ad339d8-24fa-47d898518", "velero.io/backup-name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "velero.io/pvc-uid": "9ad339d8-24fa-47d4-b9e4-0c5760f26659" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"6dbe21ab-a456-43e3-a0a1-764d67d95b27\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-6752q", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:23:19Z", "progress": {} } } DataUpload mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr phase: Accepted DataUpload Name: mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr and status: Accepted 2024/04/16 22:23:50 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr", "generateName": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "cc328e65-05dd-448b-95d0-6567c41f35b5", "resourceVersion": "38819", "generation": 2, "creationTimestamp": "2024-04-16T22:23:24Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-6dbe21ab-a456-43e3-a0a1-764d67d95b27.1317a045-5b4c-46cb18099", "velero.io/backup-name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "velero.io/pvc-uid": "1317a045-5b4c-46c8-b987-66c14a61b9b7" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:24Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:24Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"6dbe21ab-a456-43e3-a0a1-764d67d95b27\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-1-sswcx", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:23:24Z", "progress": {} } } backup phase: WaitingForPluginOperationsPartiallyFailed DataUpload mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc phase: Failed DataUpload Name: mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc and status: Failed 2024/04/16 22:24:10 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc", "generateName": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "fdba7baa-a322-4ac7-bd6c-3af8059e5465", "resourceVersion": "39348", "generation": 3, "creationTimestamp": "2024-04-16T22:23:19Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-6dbe21ab-a456-43e3-a0a1-764d67d95b27.9ad339d8-24fa-47d898518", "velero.io/backup-name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "velero.io/pvc-uid": "9ad339d8-24fa-47d4-b9e4-0c5760f26659" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:19Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"6dbe21ab-a456-43e3-a0a1-764d67d95b27\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:54Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-6752q", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-38700408-6034-4643-9cbb-6b16f7b02fae: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:23:19Z", "completionTimestamp": "2024-04-16T22:23:54Z", "progress": {} } } DataUpload mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr phase: Accepted DataUpload Name: mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr and status: Accepted 2024/04/16 22:24:10 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr", "generateName": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "cc328e65-05dd-448b-95d0-6567c41f35b5", "resourceVersion": "38819", "generation": 2, "creationTimestamp": "2024-04-16T22:23:24Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-6dbe21ab-a456-43e3-a0a1-764d67d95b27.1317a045-5b4c-46cb18099", "velero.io/backup-name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "velero.io/pvc-uid": "1317a045-5b4c-46c8-b987-66c14a61b9b7" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:24Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:23:24Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"6dbe21ab-a456-43e3-a0a1-764d67d95b27\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-mysql-1-sswcx", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "mysql-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-439", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:23:24Z", "progress": {} } } backup phase: PartiallyFailed STEP: Verify backup mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:24:30.505 2024/04/16 22:24:30 Run velero describe on the backup 2024/04/16 22:24:31 Exec stderr: "" 2024/04/16 22:24:31 Name: mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: PartiallyFailed (run `velero backup logs mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5` for more information) Errors: Velero: Cluster: Namespaces: Namespaces: Included: test-oadp-439 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-04-16 22:23:10 +0000 UTC Completed: 2024-04-16 22:24:28 +0000 UTC Expiration: 2024-05-16 22:23:10 +0000 UTC Total items to be backed up: 51 Items backed up: 51 Backup Item Operations: Operation for persistentvolumeclaims test-oadp-439/mysql: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-6dbe21ab-a456-43e3-a0a1-764d67d95b27.9ad339d8-24fa-47d898518 Items to Update: datauploads.velero.io openshift-adp/mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-6r6jc Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-38700408-6034-4643-9cbb-6b16f7b02fae: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:23:19 +0000 UTC Started: 2024-04-16 22:23:19 +0000 UTC Updated: 2024-04-16 22:23:54 +0000 UTC Operation for persistentvolumeclaims test-oadp-439/mysql-1: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-6dbe21ab-a456-43e3-a0a1-764d67d95b27.1317a045-5b4c-46cb18099 Items to Update: datauploads.velero.io openshift-adp/mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-r7kgr Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-f9db9726-2392-4f5f-8172-e73657f8ec3f: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:23:24 +0000 UTC Started: 2024-04-16 22:23:24 +0000 UTC Updated: 2024-04-16 22:24:27 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - test-oadp-439/mysql apps/v1/ReplicaSet: - test-oadp-439/mysql-7f77d47fd7 authorization.openshift.io/v1/RoleBinding: - test-oadp-439/admin - test-oadp-439/system:deployers - test-oadp-439/system:image-builders - test-oadp-439/system:image-pullers discovery.k8s.io/v1/EndpointSlice: - test-oadp-439/mysql-9qpw8 operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-439/volsync-product.v0.9.1 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-439/admin - test-oadp-439/system:deployers - test-oadp-439/system:image-builders - test-oadp-439/system:image-pullers v1/ConfigMap: - test-oadp-439/kube-root-ca.crt - test-oadp-439/openshift-service-ca.crt v1/Endpoints: - test-oadp-439/mysql v1/Event: - test-oadp-439/mysql-1.17c6e2df81e3b1c5 - test-oadp-439/mysql-1.17c6e2df8751ef22 - test-oadp-439/mysql-1.17c6e2df87600701 - test-oadp-439/mysql-1.17c6e2e088492be6 - test-oadp-439/mysql-7f77d47fd7-b2lm9.17c6e2e0b1584e98 - test-oadp-439/mysql-7f77d47fd7-b2lm9.17c6e2e139b37a83 - test-oadp-439/mysql-7f77d47fd7-b2lm9.17c6e2e13c2450d0 - test-oadp-439/mysql-7f77d47fd7-b2lm9.17c6e2e1f3444525 - test-oadp-439/mysql-7f77d47fd7-b2lm9.17c6e2e1f48a472e - test-oadp-439/mysql-7f77d47fd7-b2lm9.17c6e2e38c03954b - test-oadp-439/mysql-7f77d47fd7-b2lm9.17c6e2e3926a413d - test-oadp-439/mysql-7f77d47fd7-b2lm9.17c6e2e39342c755 - test-oadp-439/mysql-7f77d47fd7.17c6e2df85f211b8 - test-oadp-439/mysql.17c6e2df7fc1eddc - test-oadp-439/mysql.17c6e2df84bb53be - test-oadp-439/mysql.17c6e2df87249b9d - test-oadp-439/mysql.17c6e2df872a1475 - test-oadp-439/mysql.17c6e2e08869f10e v1/Namespace: - test-oadp-439 v1/PersistentVolume: - pvc-1317a045-5b4c-46c8-b987-66c14a61b9b7 - pvc-9ad339d8-24fa-47d4-b9e4-0c5760f26659 v1/PersistentVolumeClaim: - test-oadp-439/mysql - test-oadp-439/mysql-1 v1/Pod: - test-oadp-439/mysql-7f77d47fd7-b2lm9 v1/Secret: - test-oadp-439/builder-dockercfg-7w7jc - test-oadp-439/builder-token-b45ds - test-oadp-439/default-dockercfg-w8xq5 - test-oadp-439/default-token-flw8j - test-oadp-439/deployer-dockercfg-sz7t6 - test-oadp-439/deployer-token-dkm44 - test-oadp-439/mysql v1/Service: - test-oadp-439/mysql v1/ServiceAccount: - test-oadp-439/builder - test-oadp-439/default - test-oadp-439/deployer Velero-Native Snapshots: 2024/04/16 22:24:31 { "metadata": { "name": "mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5", "namespace": "openshift-adp", "uid": "6dbe21ab-a456-43e3-a0a1-764d67d95b27", "resourceVersion": "39533", "generation": 8, "creationTimestamp": "2024-04-16T22:23:10Z", "labels": { "velero.io/storage-location": "ts-dpa-1" }, "annotations": { "velero.io/resource-timeout": "10m0s", "velero.io/source-cluster-k8s-gitversion": "v1.29.3+5fa1806", "velero.io/source-cluster-k8s-major-version": "1", "velero.io/source-cluster-k8s-minor-version": "29" }, "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:23:10Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:hooks": {}, "f:includedNamespaces": {}, "f:metadata": {}, "f:snapshotMoveData": {} }, "f:status": {} } }, { "manager": "velero-server", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:24:28Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:annotations": { ".": {}, "f:velero.io/resource-timeout": {}, "f:velero.io/source-cluster-k8s-gitversion": {}, "f:velero.io/source-cluster-k8s-major-version": {}, "f:velero.io/source-cluster-k8s-minor-version": {} }, "f:labels": { ".": {}, "f:velero.io/storage-location": {} } }, "f:spec": { "f:csiSnapshotTimeout": {}, "f:defaultVolumesToFsBackup": {}, "f:itemOperationTimeout": {}, "f:storageLocation": {}, "f:ttl": {} }, "f:status": { "f:backupItemOperationsAttempted": {}, "f:backupItemOperationsFailed": {}, "f:completionTimestamp": {}, "f:errors": {}, "f:expiration": {}, "f:formatVersion": {}, "f:phase": {}, "f:progress": { ".": {}, "f:itemsBackedUp": {}, "f:totalItems": {} }, "f:startTimestamp": {}, "f:version": {} } } } ] }, "spec": { "metadata": {}, "includedNamespaces": [ "test-oadp-439" ], "ttl": "720h0m0s", "hooks": {}, "storageLocation": "ts-dpa-1", "defaultVolumesToFsBackup": false, "csiSnapshotTimeout": "10m0s", "itemOperationTimeout": "4h0m0s", "snapshotMoveData": true }, "status": { "version": 1, "formatVersion": "1.1.0", "expiration": "2024-05-16T22:23:10Z", "phase": "PartiallyFailed", "startTimestamp": "2024-04-16T22:23:10Z", "completionTimestamp": "2024-04-16T22:24:28Z", "errors": 2, "progress": { "totalItems": 51, "itemsBackedUp": 51 }, "backupItemOperationsAttempted": 2, "backupItemOperationsFailed": 2 } } [FAILED] in [It] - /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 @ 04/16/24 22:24:31.151 2024/04/16 22:24:31 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 STEP: Get the failed spec name @ 04/16/24 22:24:31.151 2024/04/16 22:24:31 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-439][interop] MySQL application STEP: Create a folder for all must-gather files if it doesn't exists already @ 04/16/24 22:24:31.151 STEP: Create a folder for the failed spec if it doesn't exists already @ 04/16/24 22:24:31.151 STEP: Run must-gather because the spec failed @ 04/16/24 22:24:31.151 2024/04/16 22:24:31 [adm must-gather --dest-dir logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_MySQL_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1] STEP: Find must-gather folder and rename it to a shorter more readable name @ 04/16/24 22:24:42.085 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-439_interop_MySQL_application/must-gather already exists, skipping renaming the folder 2024/04/16 22:24:42 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:24:56 2024-04-16 22:24:43,646 p=24378 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:24:43,646 p=24378 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:24:43,904 p=24378 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:24:43,904 p=24378 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:24:44,151 p=24378 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:24:44,151 p=24378 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:24:44,166 p=24378 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:24:44,167 p=24378 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:24:44,470 p=24378 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:24:44,470 p=24378 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:24:44,499 p=24378 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:24:44,500 p=24378 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:24:44,525 p=24378 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:24:44,525 p=24378 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:24:44,536 p=24378 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:24:45,092 p=24378 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:24:45,093 p=24378 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:24:55,986 p=24378 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-439] *** 2024-04-16 22:24:55,986 p=24378 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:24:56,291 p=24378 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:24:56,291 p=24378 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:24:56 Cleaning setup resources for the backup 2024/04/16 22:24:56 Setting new default StorageClass 'gp3-csi' 2024/04/16 22:24:56 Checking default storage class count Run the command: oc get sc 2024/04/16 22:24:56 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 34m gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 34m 2024/04/16 22:24:56 Deleting VolumeSnapshotClass 'example-snapclass' • [FAILED] [474.294 seconds] [datamover] DataMover: Backup/Restore stateful application with CSI  [It] [tc-id:OADP-439][interop] MySQL application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:114 [FAILED] Unexpected error: <*errors.errorString | 0xc00075d290>: backup phase is: PartiallyFailed; expected: Completed validation errors: [] velero failure logs: velero container contains "level=error" in line#156: time="2024-04-16T22:22:57Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1)" controller=backup-storage-location logSource="/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:194" { s: "backup phase is: PartiallyFailed; expected: Completed\nvalidation errors: []\nvelero failure logs: velero container contains \"level=error\" in line#156: time=\"2024-04-16T22:22:57Z\" level=error msg=\"Current BackupStorageLocations available/unavailable/unknown: 0/0/1)\" controller=backup-storage-location logSource=\"/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:194\"\n", } occurred In [It] at: /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 @ 04/16/24 22:24:31.151 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ [datamover] DataMover: Backup/Restore stateful application with CSI  [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:130 2024/04/16 22:24:57 Delete all downloadrequest mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-23c652df-31ff-4d3a-ba06-f9b8d31a7435 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-3f5d1d1d-a670-49b9-a75b-fb78e38fcd2d mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-735da542-4192-4f82-ac74-005f60ee15e3 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-85c75e1d-12bc-4919-acc1-bf64804fd40a mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-21da8e1b-2b9d-41ce-b6e6-e57530690c34 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-3e287799-8cc4-4605-bbfb-53e7f25a8859 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-5720bb0f-7c2e-496d-b608-fbe1a081aff0 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-64cfc5ee-3bc5-421d-a785-af229a8e0968 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-167bd570-8a09-4ebe-bfab-1418f7c22a0b mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-3e5db9e6-b636-4782-994a-4d4f12637db6 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-610de9a7-8473-4068-b60e-a2a7ee7e4fea mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-897ef686-b0f5-48bf-945f-441ca7edc4f0 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-a6e8f9a5-881f-4a3a-8b23-dc77ef7cf010 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-bd1bf1a8-b52a-461f-95f0-67de8d548a67 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-f78b4e91-7394-42e3-967b-8abacd5a3ed1 STEP: Create DPA CR @ 04/16/24 22:24:57.971 2024/04/16 22:24:58 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "34c32800-07ff-4660-8b81-5edc09815ea2", "resourceVersion": "39986", "generation": 1, "creationTimestamp": "2024-04-16T22:24:57Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:24:57Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:24:58.048 2024/04/16 22:24:58 Waiting for velero pod to be running 2024/04/16 22:25:03 pod: velero-5778f4ffcb-9chh5 is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:24:59 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:25:02 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:24:58 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:24:58 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:24:58 +0000 UTC }] 10.0.20.65 10.131.0.25 [{10.131.0.25}] 2024-04-16 22:24:58 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:24:59 +0000 UTC,FinishedAt:2024-04-16 22:24:59 +0000 UTC,ContainerID:cri-o://a0b98f620651bd2e7cae90379a83e94d3c7a14cd0b03ca4a38a627cae5ded635,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://a0b98f620651bd2e7cae90379a83e94d3c7a14cd0b03ca4a38a627cae5ded635 0xc000dba479} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:24:59 +0000 UTC,FinishedAt:2024-04-16 22:25:00 +0000 UTC,ContainerID:cri-o://7e6d7d2ee653f834851a1d480b4068aa6fab0d036b9959a2a34cef178515d1f8,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:716cec49a663d87eff7eaf9ddc948d2861645d601a2e15605cb660c9f98ab489 cri-o://7e6d7d2ee653f834851a1d480b4068aa6fab0d036b9959a2a34cef178515d1f8 0xc000dba489} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:25:00 +0000 UTC,FinishedAt:2024-04-16 22:25:00 +0000 UTC,ContainerID:cri-o://e039620e05bbbb88dbeed2728852b7ccc6d023a2241dfc0d18c608ec46b22c9a,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f cri-o://e039620e05bbbb88dbeed2728852b7ccc6d023a2241dfc0d18c608ec46b22c9a 0xc000dba499} {velero-plugin-for-csi {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:25:01 +0000 UTC,FinishedAt:2024-04-16 22:25:02 +0000 UTC,ContainerID:cri-o://88251321d0ef7f3a0d703cb911f4251b8f935fcb3f14eaebde4a9f007baaf909,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:c8979074fe11a5d1d0f5ecb0328fabb752f01103a9c03cf34e8df06bcad27f29 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:9833ca3b1b53eccc9de4e117bef53bb94d5be02d89bed222ef80dcd221bc1b18 cri-o://88251321d0ef7f3a0d703cb911f4251b8f935fcb3f14eaebde4a9f007baaf909 0xc000dba4a9}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc000dba4bf}] Burstable []} 2024/04/16 22:25:08 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:25:08.143 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/04/16 22:25:08 The 'openshift-storage' namespace does not exist 2024/04/16 22:25:08 Using default CSI driver based on infrastructure: ebs.csi.aws.com 2024/04/16 22:25:08 Snapclass 'example-snapclass' doesn't exist, creating 2024/04/16 22:25:08 Setting new default StorageClass 'gp2-csi' 2024/04/16 22:25:08 Checking default storage class count Run the command: oc get sc 2024/04/16 22:25:08 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 34m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 34m 2024/04/16 22:25:08 Checking for correct number of running NodeAgent pods... STEP: Installing application for case cassandra-e2e @ 04/16/24 22:25:08.604 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pods status (30 retries left). FAILED - RETRYING: [localhost]: Check pods status (29 retries left). FAILED - RETRYING: [localhost]: Check pods status (28 retries left). FAILED - RETRYING: [localhost]: Check pods status (27 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (30 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (29 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (28 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (27 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (26 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (25 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (24 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (23 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (22 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (21 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (20 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (19 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (18 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (17 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (16 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=18  changed=9  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/04/16 22:28:07 2024-04-16 22:25:10,186 p=24641 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:25:10,186 p=24641 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:25:10,442 p=24641 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:25:10,442 p=24641 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:25:10,695 p=24641 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:25:10,696 p=24641 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:25:10,714 p=24641 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:25:10,714 p=24641 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:25:11,025 p=24641 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:25:11,025 p=24641 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:25:11,054 p=24641 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:25:11,055 p=24641 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:25:11,080 p=24641 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:25:11,080 p=24641 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:25:11,091 p=24641 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:25:11,661 p=24641 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:25:11,661 p=24641 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:25:12,529 p=24641 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** 2024-04-16 22:25:12,530 p=24641 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:25:12,949 p=24641 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** 2024-04-16 22:25:12,949 p=24641 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:25:13,285 p=24641 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** 2024-04-16 22:25:13,285 p=24641 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:25:14,138 p=24641 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** 2024-04-16 22:25:14,138 p=24641 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:25:14,835 p=24641 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** 2024-04-16 22:25:14,835 p=24641 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:25:38,105 p=24641 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-04-16 22:25:38,105 p=24641 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:28:03,735 p=24641 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-04-16 22:28:03,735 p=24641 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:28:07,020 p=24641 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** 2024-04-16 22:28:07,020 p=24641 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:28:07,157 p=24641 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:28:07,157 p=24641 u=1009100000 n=ansible | localhost : ok=18 changed=9 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:28:07.223 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=15  changed=7  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/04/16 22:28:18 2024-04-16 22:28:09,828 p=25539 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:28:09,828 p=25539 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:28:10,079 p=25539 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:28:10,079 p=25539 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:28:10,338 p=25539 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:28:10,339 p=25539 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:28:10,355 p=25539 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:28:10,355 p=25539 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:28:10,670 p=25539 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:28:10,670 p=25539 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:28:10,701 p=25539 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:28:10,701 p=25539 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:28:10,726 p=25539 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:28:10,726 p=25539 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:28:10,739 p=25539 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:28:11,303 p=25539 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:28:11,303 p=25539 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:28:12,456 p=25539 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-04-16 22:28:12,456 p=25539 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:28:16,637 p=25539 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-04-16 22:28:16,637 p=25539 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:28:17,768 p=25539 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** 2024-04-16 22:28:17,768 p=25539 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:28:18,296 p=25539 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** 2024-04-16 22:28:18,297 p=25539 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:28:18,484 p=25539 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** 2024-04-16 22:28:18,484 p=25539 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:28:18,499 p=25539 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:28:18,499 p=25539 u=1009100000 n=ansible | localhost : ok=15 changed=7 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Creating backup cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5 @ 04/16/24 22:28:18.552 2024/04/16 22:28:18 Wait until backup cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5 is completed backup phase: InProgress DataUpload cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-h9xzz phase: Accepted DataUpload Name: cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-h9xzz and status: Accepted 2024/04/16 22:28:38 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-h9xzz", "generateName": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "00f75756-caa9-4f56-85a4-7040cc6a0aa9", "resourceVersion": "41550", "generation": 2, "creationTimestamp": "2024-04-16T22:28:28Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-45d1a36b-9cf8-4d84-a4d0-240965662636.e30ee1ec-8f28-4dcd1c535", "velero.io/backup-name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "velero.io/pvc-uid": "e30ee1ec-8f28-4dc8-8112-16633530dc46" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:28Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:28Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"45d1a36b-9cf8-4d84-a4d0-240965662636\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-l9zsz", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:28:28Z", "progress": {} } } DataUpload cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-t4524 phase: Accepted DataUpload Name: cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-t4524 and status: Accepted 2024/04/16 22:28:38 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-t4524", "generateName": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "84145b90-ddd2-4dba-9027-eec6d5982af7", "resourceVersion": "41664", "generation": 2, "creationTimestamp": "2024-04-16T22:28:33Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-45d1a36b-9cf8-4d84-a4d0-240965662636.2691a445-d376-48678b13a", "velero.io/backup-name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "velero.io/pvc-uid": "2691a445-d376-4867-a95d-ebb80ec0fc71" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"45d1a36b-9cf8-4d84-a4d0-240965662636\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-h7cvq", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:28:33Z", "progress": {} } } DataUpload cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-2hxrz phase: Accepted DataUpload Name: cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-2hxrz and status: Accepted 2024/04/16 22:28:38 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-2hxrz", "generateName": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "755f48da-dd39-4e7f-b5c0-81c88e9d12f9", "resourceVersion": "41764", "generation": 2, "creationTimestamp": "2024-04-16T22:28:38Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-45d1a36b-9cf8-4d84-a4d0-240965662636.f13d5598-8aff-4d7b63445", "velero.io/backup-name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "velero.io/pvc-uid": "f13d5598-8aff-4d73-aa72-cc9792a8a790" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:38Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:38Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"45d1a36b-9cf8-4d84-a4d0-240965662636\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-vkkdl", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:28:38Z", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-h9xzz phase: Failed DataUpload Name: cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-h9xzz and status: Failed 2024/04/16 22:28:58 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-h9xzz", "generateName": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "00f75756-caa9-4f56-85a4-7040cc6a0aa9", "resourceVersion": "42059", "generation": 3, "creationTimestamp": "2024-04-16T22:28:28Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-45d1a36b-9cf8-4d84-a4d0-240965662636.e30ee1ec-8f28-4dcd1c535", "velero.io/backup-name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "velero.io/pvc-uid": "e30ee1ec-8f28-4dc8-8112-16633530dc46" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:28Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"45d1a36b-9cf8-4d84-a4d0-240965662636\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:56Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-l9zsz", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-8dbb8136-8647-49a0-8f51-f32e910d01d8: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:28:28Z", "completionTimestamp": "2024-04-16T22:28:56Z", "progress": {} } } DataUpload cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-t4524 phase: Failed DataUpload Name: cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-t4524 and status: Failed 2024/04/16 22:28:58 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-t4524", "generateName": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "84145b90-ddd2-4dba-9027-eec6d5982af7", "resourceVersion": "42049", "generation": 3, "creationTimestamp": "2024-04-16T22:28:33Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-45d1a36b-9cf8-4d84-a4d0-240965662636.2691a445-d376-48678b13a", "velero.io/backup-name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "velero.io/pvc-uid": "2691a445-d376-4867-a95d-ebb80ec0fc71" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:33Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"45d1a36b-9cf8-4d84-a4d0-240965662636\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:55Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-h7cvq", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-91d25007-5f13-47ed-b1c5-3d67025150c3: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:28:33Z", "completionTimestamp": "2024-04-16T22:28:55Z", "progress": {} } } DataUpload cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-2hxrz phase: Failed DataUpload Name: cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-2hxrz and status: Failed 2024/04/16 22:28:58 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-2hxrz", "generateName": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "755f48da-dd39-4e7f-b5c0-81c88e9d12f9", "resourceVersion": "42063", "generation": 3, "creationTimestamp": "2024-04-16T22:28:38Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-45d1a36b-9cf8-4d84-a4d0-240965662636.f13d5598-8aff-4d7b63445", "velero.io/backup-name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "velero.io/pvc-uid": "f13d5598-8aff-4d73-aa72-cc9792a8a790" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:38Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"45d1a36b-9cf8-4d84-a4d0-240965662636\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:28:56Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-vkkdl", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-f73c548a-8f2a-408c-bc1d-ef470dfa4790: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:28:38Z", "completionTimestamp": "2024-04-16T22:28:56Z", "progress": {} } } backup phase: PartiallyFailed STEP: Verify backup cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:29:18.772 2024/04/16 22:29:18 Run velero describe on the backup 2024/04/16 22:29:19 Exec stderr: "" 2024/04/16 22:29:19 Name: cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: PartiallyFailed (run `velero backup logs cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5` for more information) Errors: Velero: Cluster: Namespaces: Namespaces: Included: test-oadp-440 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-04-16 22:28:18 +0000 UTC Completed: 2024-04-16 22:29:04 +0000 UTC Expiration: 2024-05-16 22:28:18 +0000 UTC Total items to be backed up: 80 Items backed up: 80 Backup Item Operations: Operation for persistentvolumeclaims test-oadp-440/cassandra-data-cassandra-0: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-45d1a36b-9cf8-4d84-a4d0-240965662636.e30ee1ec-8f28-4dcd1c535 Items to Update: datauploads.velero.io openshift-adp/cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-h9xzz Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-8dbb8136-8647-49a0-8f51-f32e910d01d8: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:28:28 +0000 UTC Started: 2024-04-16 22:28:28 +0000 UTC Updated: 2024-04-16 22:28:56 +0000 UTC Operation for persistentvolumeclaims test-oadp-440/cassandra-data-cassandra-1: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-45d1a36b-9cf8-4d84-a4d0-240965662636.2691a445-d376-48678b13a Items to Update: datauploads.velero.io openshift-adp/cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-t4524 Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-91d25007-5f13-47ed-b1c5-3d67025150c3: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:28:33 +0000 UTC Started: 2024-04-16 22:28:33 +0000 UTC Updated: 2024-04-16 22:28:55 +0000 UTC Operation for persistentvolumeclaims test-oadp-440/cassandra-data-cassandra-2: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-45d1a36b-9cf8-4d84-a4d0-240965662636.f13d5598-8aff-4d7b63445 Items to Update: datauploads.velero.io openshift-adp/cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-2hxrz Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-f73c548a-8f2a-408c-bc1d-ef470dfa4790: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:28:38 +0000 UTC Started: 2024-04-16 22:28:38 +0000 UTC Updated: 2024-04-16 22:28:56 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/ControllerRevision: - test-oadp-440/cassandra-58bc85766c apps/v1/StatefulSet: - test-oadp-440/cassandra authorization.openshift.io/v1/RoleBinding: - test-oadp-440/admin - test-oadp-440/system:deployers - test-oadp-440/system:image-builders - test-oadp-440/system:image-pullers - test-oadp-440/system:openshift:scc:privileged discovery.k8s.io/v1/EndpointSlice: - test-oadp-440/cassandra-lv6gs operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-440/volsync-product.v0.9.1 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-440/admin - test-oadp-440/system:deployers - test-oadp-440/system:image-builders - test-oadp-440/system:image-pullers - test-oadp-440/system:openshift:scc:privileged v1/ConfigMap: - test-oadp-440/kube-root-ca.crt - test-oadp-440/openshift-service-ca.crt v1/Endpoints: - test-oadp-440/cassandra v1/Event: - test-oadp-440/cassandra-0.17c6e30538bb8e52 - test-oadp-440/cassandra-0.17c6e305ca5d54ce - test-oadp-440/cassandra-0.17c6e3073f7a4f78 - test-oadp-440/cassandra-0.17c6e30740e6dd6e - test-oadp-440/cassandra-0.17c6e30838c43a46 - test-oadp-440/cassandra-0.17c6e3083fc9aa2f - test-oadp-440/cassandra-0.17c6e3084099da32 - test-oadp-440/cassandra-1.17c6e3096f40f2c2 - test-oadp-440/cassandra-1.17c6e309d371b038 - test-oadp-440/cassandra-1.17c6e30a353b3f74 - test-oadp-440/cassandra-1.17c6e30a3683ca9f - test-oadp-440/cassandra-1.17c6e30b34fb93c2 - test-oadp-440/cassandra-1.17c6e30b3c72a1ea - test-oadp-440/cassandra-1.17c6e30b3d415205 - test-oadp-440/cassandra-2.17c6e30ca450e4a3 - test-oadp-440/cassandra-2.17c6e30d2eff3e08 - test-oadp-440/cassandra-2.17c6e30d9f3461ee - test-oadp-440/cassandra-2.17c6e30da0a87438 - test-oadp-440/cassandra-2.17c6e30e9c710a60 - test-oadp-440/cassandra-2.17c6e30ea3db15c3 - test-oadp-440/cassandra-2.17c6e30ea4b4a663 - test-oadp-440/cassandra-2.17c6e312648f8f63 - test-oadp-440/cassandra-2.17c6e31774d47132 - test-oadp-440/cassandra-2.17c6e31b11f65dbe - test-oadp-440/cassandra-data-cassandra-0.17c6e3040ce09991 - test-oadp-440/cassandra-data-cassandra-0.17c6e3040e656790 - test-oadp-440/cassandra-data-cassandra-0.17c6e3040e6f38f7 - test-oadp-440/cassandra-data-cassandra-0.17c6e3051032549f - test-oadp-440/cassandra-data-cassandra-1.17c6e30843b716df - test-oadp-440/cassandra-data-cassandra-1.17c6e308450f7441 - test-oadp-440/cassandra-data-cassandra-1.17c6e308451afd97 - test-oadp-440/cassandra-data-cassandra-1.17c6e30945f6b87a - test-oadp-440/cassandra-data-cassandra-2.17c6e30b78637231 - test-oadp-440/cassandra-data-cassandra-2.17c6e30b7a1fbf9a - test-oadp-440/cassandra-data-cassandra-2.17c6e30b7a2cc203 - test-oadp-440/cassandra-data-cassandra-2.17c6e30c7bd94d34 - test-oadp-440/cassandra.17c6e3040cd8386e - test-oadp-440/cassandra.17c6e3040d6a6c99 - test-oadp-440/cassandra.17c6e30843b24354 - test-oadp-440/cassandra.17c6e3084451d8a9 - test-oadp-440/cassandra.17c6e30b7857ef34 - test-oadp-440/cassandra.17c6e30b79002888 v1/Namespace: - test-oadp-440 v1/PersistentVolume: - pvc-2691a445-d376-4867-a95d-ebb80ec0fc71 - pvc-e30ee1ec-8f28-4dc8-8112-16633530dc46 - pvc-f13d5598-8aff-4d73-aa72-cc9792a8a790 v1/PersistentVolumeClaim: - test-oadp-440/cassandra-data-cassandra-0 - test-oadp-440/cassandra-data-cassandra-1 - test-oadp-440/cassandra-data-cassandra-2 v1/Pod: - test-oadp-440/cassandra-0 - test-oadp-440/cassandra-1 - test-oadp-440/cassandra-2 v1/Secret: - test-oadp-440/builder-dockercfg-gw4hg - test-oadp-440/builder-token-jh2x7 - test-oadp-440/default-dockercfg-2ldpc - test-oadp-440/default-token-f5jlq - test-oadp-440/deployer-dockercfg-6nl72 - test-oadp-440/deployer-token-gpcf7 v1/Service: - test-oadp-440/cassandra v1/ServiceAccount: - test-oadp-440/builder - test-oadp-440/default - test-oadp-440/deployer Velero-Native Snapshots: 2024/04/16 22:29:19 { "metadata": { "name": "cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5", "namespace": "openshift-adp", "uid": "45d1a36b-9cf8-4d84-a4d0-240965662636", "resourceVersion": "42105", "generation": 9, "creationTimestamp": "2024-04-16T22:28:18Z", "labels": { "velero.io/storage-location": "ts-dpa-1" }, "annotations": { "velero.io/resource-timeout": "10m0s", "velero.io/source-cluster-k8s-gitversion": "v1.29.3+5fa1806", "velero.io/source-cluster-k8s-major-version": "1", "velero.io/source-cluster-k8s-minor-version": "29" }, "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:28:18Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:hooks": {}, "f:includedNamespaces": {}, "f:metadata": {}, "f:snapshotMoveData": {} }, "f:status": {} } }, { "manager": "velero-server", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:29:04Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:annotations": { ".": {}, "f:velero.io/resource-timeout": {}, "f:velero.io/source-cluster-k8s-gitversion": {}, "f:velero.io/source-cluster-k8s-major-version": {}, "f:velero.io/source-cluster-k8s-minor-version": {} }, "f:labels": { ".": {}, "f:velero.io/storage-location": {} } }, "f:spec": { "f:csiSnapshotTimeout": {}, "f:defaultVolumesToFsBackup": {}, "f:itemOperationTimeout": {}, "f:storageLocation": {}, "f:ttl": {} }, "f:status": { "f:backupItemOperationsAttempted": {}, "f:backupItemOperationsFailed": {}, "f:completionTimestamp": {}, "f:errors": {}, "f:expiration": {}, "f:formatVersion": {}, "f:phase": {}, "f:progress": { ".": {}, "f:itemsBackedUp": {}, "f:totalItems": {} }, "f:startTimestamp": {}, "f:version": {} } } } ] }, "spec": { "metadata": {}, "includedNamespaces": [ "test-oadp-440" ], "ttl": "720h0m0s", "hooks": {}, "storageLocation": "ts-dpa-1", "defaultVolumesToFsBackup": false, "csiSnapshotTimeout": "10m0s", "itemOperationTimeout": "4h0m0s", "snapshotMoveData": true }, "status": { "version": 1, "formatVersion": "1.1.0", "expiration": "2024-05-16T22:28:18Z", "phase": "PartiallyFailed", "startTimestamp": "2024-04-16T22:28:18Z", "completionTimestamp": "2024-04-16T22:29:04Z", "errors": 3, "progress": { "totalItems": 80, "itemsBackedUp": 80 }, "backupItemOperationsAttempted": 3, "backupItemOperationsFailed": 3 } } [FAILED] in [It] - /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 @ 04/16/24 22:29:19.361 2024/04/16 22:29:19 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 STEP: Get the failed spec name @ 04/16/24 22:29:19.361 2024/04/16 22:29:19 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application STEP: Create a folder for all must-gather files if it doesn't exists already @ 04/16/24 22:29:19.361 STEP: Create a folder for the failed spec if it doesn't exists already @ 04/16/24 22:29:19.361 2024/04/16 22:29:19 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application does not exists, creating new folder with the name: logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application STEP: Run must-gather because the spec failed @ 04/16/24 22:29:19.361 2024/04/16 22:29:19 [adm must-gather --dest-dir logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1] STEP: Find must-gather folder and rename it to a shorter more readable name @ 04/16/24 22:29:30.409 2024/04/16 22:29:30 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:29:54 2024-04-16 22:29:31,951 p=25879 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:29:31,952 p=25879 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:29:32,203 p=25879 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:29:32,203 p=25879 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:29:32,457 p=25879 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:29:32,458 p=25879 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:29:32,473 p=25879 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:29:32,473 p=25879 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:29:32,773 p=25879 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:29:32,773 p=25879 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:29:32,805 p=25879 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:29:32,806 p=25879 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:29:32,830 p=25879 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:29:32,830 p=25879 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:29:32,843 p=25879 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:29:33,405 p=25879 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:29:33,405 p=25879 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:29:54,352 p=25879 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** 2024-04-16 22:29:54,352 p=25879 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:29:54,651 p=25879 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:29:54,651 p=25879 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:29:54 Cleaning setup resources for the backup 2024/04/16 22:29:54 Setting new default StorageClass 'gp3-csi' 2024/04/16 22:29:54 Checking default storage class count Run the command: oc get sc 2024/04/16 22:29:54 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 39m gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 39m 2024/04/16 22:29:54 Deleting VolumeSnapshotClass 'example-snapclass' Attempt #1 Failed. Retrying ↺ @ 04/16/24 22:29:54.952 2024/04/16 22:29:55 Delete all downloadrequest cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-00f2509e-b3e0-4662-96d5-2906ae0977ed cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-1009a59b-fb90-4e6c-abf3-5b022c8a4293 cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-2bf27e8b-ed91-49a2-9187-9601d944eb8f cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-4cf3e809-8f42-4d74-9627-913117aa9dd4 cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-7260a969-e4c3-4c11-8ebb-db3bf2ed17fd cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-78e3f684-0325-4ca2-a046-ef3e40b934b7 cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-bddf07ec-e07a-4701-a344-a6ad19a241b3 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-672df615-300d-4d98-9981-641696adbbda mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-6e1968e8-24ab-4df1-a379-c67e6861c144 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-b4cd086f-53c5-44dc-adcb-1af50c3e113b mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-c93a2b71-3055-44b4-832d-210ce6fd652d mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-705dcbc1-d5cc-4b32-90c4-047709eafc57 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-8c8587d7-a868-4400-ad7f-ade0496b43b6 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-d3593e88-4357-4a1a-b4ef-0c7af4577100 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-defcbe51-5177-4963-b3d8-a12de92dea48 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-842c4033-bcfb-4181-b613-879c63570329 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-ba7dd1f3-6460-4101-b559-b639d0ffab77 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-c42d2681-b6dc-4644-8b0c-8dccf13bc9f2 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-da221546-0989-4be9-ac71-21e44daf7824 STEP: Create DPA CR @ 04/16/24 22:29:55.635 2024/04/16 22:29:55 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "ada3d29b-2301-47bf-aece-8e9e429eab40", "resourceVersion": "42742", "generation": 1, "creationTimestamp": "2024-04-16T22:29:55Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:29:55Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:29:55.714 2024/04/16 22:29:55 Waiting for velero pod to be running 2024/04/16 22:29:55 pod: velero-5778f4ffcb-9chh5 is not yet running with status: {Succeeded [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:29:55 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:25:02 +0000 UTC PodCompleted } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:29:55 +0000 UTC PodCompleted } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:29:55 +0000 UTC PodCompleted } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:24:58 +0000 UTC }] 10.0.20.65 [] 2024-04-16 22:24:58 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:24:59 +0000 UTC,FinishedAt:2024-04-16 22:24:59 +0000 UTC,ContainerID:cri-o://a0b98f620651bd2e7cae90379a83e94d3c7a14cd0b03ca4a38a627cae5ded635,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://a0b98f620651bd2e7cae90379a83e94d3c7a14cd0b03ca4a38a627cae5ded635 0xc0012c95f9} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:24:59 +0000 UTC,FinishedAt:2024-04-16 22:25:00 +0000 UTC,ContainerID:cri-o://7e6d7d2ee653f834851a1d480b4068aa6fab0d036b9959a2a34cef178515d1f8,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:716cec49a663d87eff7eaf9ddc948d2861645d601a2e15605cb660c9f98ab489 cri-o://7e6d7d2ee653f834851a1d480b4068aa6fab0d036b9959a2a34cef178515d1f8 0xc0012c9609} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:25:00 +0000 UTC,FinishedAt:2024-04-16 22:25:00 +0000 UTC,ContainerID:cri-o://e039620e05bbbb88dbeed2728852b7ccc6d023a2241dfc0d18c608ec46b22c9a,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f cri-o://e039620e05bbbb88dbeed2728852b7ccc6d023a2241dfc0d18c608ec46b22c9a 0xc0012c9619} {velero-plugin-for-csi {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:25:01 +0000 UTC,FinishedAt:2024-04-16 22:25:02 +0000 UTC,ContainerID:cri-o://88251321d0ef7f3a0d703cb911f4251b8f935fcb3f14eaebde4a9f007baaf909,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:c8979074fe11a5d1d0f5ecb0328fabb752f01103a9c03cf34e8df06bcad27f29 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:9833ca3b1b53eccc9de4e117bef53bb94d5be02d89bed222ef80dcd221bc1b18 cri-o://88251321d0ef7f3a0d703cb911f4251b8f935fcb3f14eaebde4a9f007baaf909 0xc0012c9629}] [{velero {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:25:02 +0000 UTC,FinishedAt:2024-04-16 22:29:55 +0000 UTC,ContainerID:cri-o://87a2f370d77695612fa6a181e275ba9178561b4f2c0ad1b39a59c5fc3356f1d3,}} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:02f194caab292aea9bcbb22556fca8e2d7d893a629e6e56097501501fbff7ba7 cri-o://87a2f370d77695612fa6a181e275ba9178561b4f2c0ad1b39a59c5fc3356f1d3 0xc0012c9639}] Burstable []} 2024/04/16 22:30:00 pod: velero-5778f4ffcb-qk848 is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:29:57 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:30:00 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:29:55 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:29:55 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:29:55 +0000 UTC }] 10.0.20.65 10.131.0.28 [{10.131.0.28}] 2024-04-16 22:29:55 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:29:56 +0000 UTC,FinishedAt:2024-04-16 22:29:56 +0000 UTC,ContainerID:cri-o://a01d69e7f4539eee874515e1a27a663452bed9814c4e4480a6148a4985e15378,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://a01d69e7f4539eee874515e1a27a663452bed9814c4e4480a6148a4985e15378 0xc0012c9da9} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:29:57 +0000 UTC,FinishedAt:2024-04-16 22:29:57 +0000 UTC,ContainerID:cri-o://76d530788bec60341fcff6f915a42b833f7fc3e4e0ed20b5f50a927aeafd1ced,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:716cec49a663d87eff7eaf9ddc948d2861645d601a2e15605cb660c9f98ab489 cri-o://76d530788bec60341fcff6f915a42b833f7fc3e4e0ed20b5f50a927aeafd1ced 0xc0012c9db9} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:29:58 +0000 UTC,FinishedAt:2024-04-16 22:29:58 +0000 UTC,ContainerID:cri-o://3139d4b07d916a6a0f0c81a92058cf9fc75da76ee4fa9e7146dfc2071f370534,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f cri-o://3139d4b07d916a6a0f0c81a92058cf9fc75da76ee4fa9e7146dfc2071f370534 0xc0012c9dc9} {velero-plugin-for-csi {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:29:59 +0000 UTC,FinishedAt:2024-04-16 22:29:59 +0000 UTC,ContainerID:cri-o://ca632592dbfa972a8b38fb973a70cf323edabd5e247322e8d8dd67caed63a084,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:c8979074fe11a5d1d0f5ecb0328fabb752f01103a9c03cf34e8df06bcad27f29 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:9833ca3b1b53eccc9de4e117bef53bb94d5be02d89bed222ef80dcd221bc1b18 cri-o://ca632592dbfa972a8b38fb973a70cf323edabd5e247322e8d8dd67caed63a084 0xc0012c9f09}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc0012c9f1f}] Burstable []} 2024/04/16 22:30:05 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:30:05.802 2024/04/16 22:30:05 Snapclass 'example-snapclass' doesn't exist, creating 2024/04/16 22:30:05 Setting new default StorageClass 'gp2-csi' 2024/04/16 22:30:05 Checking default storage class count Run the command: oc get sc 2024/04/16 22:30:06 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 39m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 39m 2024/04/16 22:30:06 Checking for correct number of running NodeAgent pods... STEP: Installing application for case cassandra-e2e @ 04/16/24 22:30:06.111 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pods status (30 retries left). FAILED - RETRYING: [localhost]: Check pods status (29 retries left). FAILED - RETRYING: [localhost]: Check pods status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (30 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (29 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (28 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (27 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (26 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (25 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (24 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (23 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (22 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (21 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (20 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (19 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (18 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (17 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (16 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (15 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=18  changed=9  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/04/16 22:33:09 2024-04-16 22:30:07,656 p=26122 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:30:07,656 p=26122 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:30:07,902 p=26122 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:30:07,902 p=26122 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:30:08,164 p=26122 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:30:08,164 p=26122 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:30:08,180 p=26122 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:30:08,180 p=26122 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:30:08,473 p=26122 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:30:08,473 p=26122 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:30:08,502 p=26122 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:30:08,502 p=26122 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:30:08,527 p=26122 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:30:08,527 p=26122 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:30:08,538 p=26122 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:30:09,085 p=26122 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:30:09,085 p=26122 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:30:09,930 p=26122 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** 2024-04-16 22:30:09,930 p=26122 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:30:10,439 p=26122 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** 2024-04-16 22:30:10,439 p=26122 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:30:10,798 p=26122 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** 2024-04-16 22:30:10,798 p=26122 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:30:11,635 p=26122 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** 2024-04-16 22:30:11,635 p=26122 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:30:12,326 p=26122 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** 2024-04-16 22:30:12,326 p=26122 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:30:29,874 p=26122 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-04-16 22:30:29,874 p=26122 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:33:05,339 p=26122 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-04-16 22:33:05,339 p=26122 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:33:09,729 p=26122 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** 2024-04-16 22:33:09,730 p=26122 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:33:09,862 p=26122 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:33:09,862 p=26122 u=1009100000 n=ansible | localhost : ok=18 changed=9 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:33:09.915 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=15  changed=6  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/04/16 22:33:20 2024-04-16 22:33:11,457 p=27047 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:33:11,458 p=27047 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:33:11,712 p=27047 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:33:11,712 p=27047 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:33:11,961 p=27047 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:33:11,961 p=27047 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:33:11,979 p=27047 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:33:11,979 p=27047 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:33:12,295 p=27047 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:33:12,295 p=27047 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:33:12,324 p=27047 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:33:12,324 p=27047 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:33:12,348 p=27047 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:33:12,348 p=27047 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:33:12,359 p=27047 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:33:12,947 p=27047 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:33:12,947 p=27047 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:33:14,129 p=27047 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-04-16 22:33:14,129 p=27047 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:33:18,725 p=27047 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-04-16 22:33:18,725 p=27047 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:33:19,850 p=27047 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** 2024-04-16 22:33:19,851 p=27047 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:33:20,427 p=27047 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** 2024-04-16 22:33:20,428 p=27047 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:33:20,622 p=27047 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** 2024-04-16 22:33:20,622 p=27047 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:33:20,634 p=27047 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:33:20,635 p=27047 u=1009100000 n=ansible | localhost : ok=15 changed=6 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Creating backup cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5 @ 04/16/24 22:33:20.691 2024/04/16 22:33:20 Wait until backup cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5 is completed backup phase: InProgress DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7 phase: Accepted DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7 and status: Accepted 2024/04/16 22:33:40 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "1eacc9c8-c6ee-41e0-85b9-ce8d6f60851b", "resourceVersion": "44319", "generation": 2, "creationTimestamp": "2024-04-16T22:33:30Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.0685d294-921b-493646ae3", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "0685d294-921b-493d-9f48-89fe4665a5d1" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-pnqqw", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:33:30Z", "progress": {} } } DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w phase: Accepted DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w and status: Accepted 2024/04/16 22:33:40 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "93ce789b-40f7-4972-8be4-410af50f1421", "resourceVersion": "44419", "generation": 2, "creationTimestamp": "2024-04-16T22:33:35Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.319b54d1-e784-4c2033963", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "319b54d1-e784-4c2d-87fb-8b684ec50997" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:35Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:35Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-dzpvt", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:33:35Z", "progress": {} } } DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b phase: Accepted DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b and status: Accepted 2024/04/16 22:33:40 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "04a6aca3-a204-4ff8-a414-23864b948abd", "resourceVersion": "44528", "generation": 2, "creationTimestamp": "2024-04-16T22:33:40Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.4cfe7730-805d-48d08e014", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "4cfe7730-805d-48dc-9df5-121577441e9b" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:40Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:40Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-j9h29", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:33:40Z", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7 phase: Accepted DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7 and status: Accepted 2024/04/16 22:34:00 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "1eacc9c8-c6ee-41e0-85b9-ce8d6f60851b", "resourceVersion": "44319", "generation": 2, "creationTimestamp": "2024-04-16T22:33:30Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.0685d294-921b-493646ae3", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "0685d294-921b-493d-9f48-89fe4665a5d1" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-pnqqw", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:33:30Z", "progress": {} } } DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w phase: Accepted DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w and status: Accepted 2024/04/16 22:34:00 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "93ce789b-40f7-4972-8be4-410af50f1421", "resourceVersion": "44419", "generation": 2, "creationTimestamp": "2024-04-16T22:33:35Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.319b54d1-e784-4c2033963", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "319b54d1-e784-4c2d-87fb-8b684ec50997" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:35Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:35Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-dzpvt", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:33:35Z", "progress": {} } } DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b phase: Accepted DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b and status: Accepted 2024/04/16 22:34:00 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "04a6aca3-a204-4ff8-a414-23864b948abd", "resourceVersion": "44528", "generation": 2, "creationTimestamp": "2024-04-16T22:33:40Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.4cfe7730-805d-48d08e014", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "4cfe7730-805d-48dc-9df5-121577441e9b" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:40Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:40Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-j9h29", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:33:40Z", "progress": {} } } backup phase: WaitingForPluginOperationsPartiallyFailed DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7 phase: Failed DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7 and status: Failed 2024/04/16 22:34:20 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "1eacc9c8-c6ee-41e0-85b9-ce8d6f60851b", "resourceVersion": "45100", "generation": 3, "creationTimestamp": "2024-04-16T22:33:30Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.0685d294-921b-493646ae3", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "0685d294-921b-493d-9f48-89fe4665a5d1" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:34:12Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-pnqqw", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-f748731f-0cf7-4322-9084-c2489ecb51b4: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:33:30Z", "completionTimestamp": "2024-04-16T22:34:12Z", "progress": {} } } DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w phase: Accepted DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w and status: Accepted 2024/04/16 22:34:20 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "93ce789b-40f7-4972-8be4-410af50f1421", "resourceVersion": "44419", "generation": 2, "creationTimestamp": "2024-04-16T22:33:35Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.319b54d1-e784-4c2033963", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "319b54d1-e784-4c2d-87fb-8b684ec50997" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:35Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:35Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-dzpvt", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:33:35Z", "progress": {} } } DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b phase: Accepted DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b and status: Accepted 2024/04/16 22:34:20 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "04a6aca3-a204-4ff8-a414-23864b948abd", "resourceVersion": "44528", "generation": 2, "creationTimestamp": "2024-04-16T22:33:40Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.4cfe7730-805d-48d08e014", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "4cfe7730-805d-48dc-9df5-121577441e9b" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:40Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:40Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-j9h29", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:33:40Z", "progress": {} } } backup phase: WaitingForPluginOperationsPartiallyFailed DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7 phase: Failed DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7 and status: Failed 2024/04/16 22:34:40 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "1eacc9c8-c6ee-41e0-85b9-ce8d6f60851b", "resourceVersion": "45100", "generation": 3, "creationTimestamp": "2024-04-16T22:33:30Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.0685d294-921b-493646ae3", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "0685d294-921b-493d-9f48-89fe4665a5d1" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:30Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:34:12Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-pnqqw", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-f748731f-0cf7-4322-9084-c2489ecb51b4: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:33:30Z", "completionTimestamp": "2024-04-16T22:34:12Z", "progress": {} } } DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w phase: Failed DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w and status: Failed 2024/04/16 22:34:40 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "93ce789b-40f7-4972-8be4-410af50f1421", "resourceVersion": "45237", "generation": 3, "creationTimestamp": "2024-04-16T22:33:35Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.319b54d1-e784-4c2033963", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "319b54d1-e784-4c2d-87fb-8b684ec50997" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:35Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:34:37Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-dzpvt", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-b94dab66-8261-4ee2-92a8-52cbcc99ef67: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:33:35Z", "completionTimestamp": "2024-04-16T22:34:37Z", "progress": {} } } DataUpload cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b phase: Accepted DataUpload Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b and status: Accepted 2024/04/16 22:34:40 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b", "generateName": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "04a6aca3-a204-4ff8-a414-23864b948abd", "resourceVersion": "44528", "generation": 2, "creationTimestamp": "2024-04-16T22:33:40Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-96e25279-1ec8-4cd0-aee6-134ec041501b.4cfe7730-805d-48d08e014", "velero.io/backup-name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "velero.io/pvc-uid": "4cfe7730-805d-48dc-9df5-121577441e9b" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:40Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:33:40Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"96e25279-1ec8-4cd0-aee6-134ec041501b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-j9h29", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:33:40Z", "progress": {} } } backup phase: PartiallyFailed STEP: Verify backup cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:35:00.967 2024/04/16 22:35:01 Run velero describe on the backup 2024/04/16 22:35:01 Exec stderr: "" 2024/04/16 22:35:01 Name: cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: PartiallyFailed (run `velero backup logs cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5` for more information) Errors: Velero: Cluster: Namespaces: Namespaces: Included: test-oadp-440 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-04-16 22:33:20 +0000 UTC Completed: 2024-04-16 22:34:44 +0000 UTC Expiration: 2024-05-16 22:33:20 +0000 UTC Total items to be backed up: 80 Items backed up: 80 Backup Item Operations: Operation for persistentvolumeclaims test-oadp-440/cassandra-data-cassandra-0: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-96e25279-1ec8-4cd0-aee6-134ec041501b.0685d294-921b-493646ae3 Items to Update: datauploads.velero.io openshift-adp/cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dpjv7 Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-f748731f-0cf7-4322-9084-c2489ecb51b4: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:33:30 +0000 UTC Started: 2024-04-16 22:33:30 +0000 UTC Updated: 2024-04-16 22:34:12 +0000 UTC Operation for persistentvolumeclaims test-oadp-440/cassandra-data-cassandra-1: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-96e25279-1ec8-4cd0-aee6-134ec041501b.319b54d1-e784-4c2033963 Items to Update: datauploads.velero.io openshift-adp/cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-qgk2w Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-b94dab66-8261-4ee2-92a8-52cbcc99ef67: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:33:35 +0000 UTC Started: 2024-04-16 22:33:35 +0000 UTC Updated: 2024-04-16 22:34:37 +0000 UTC Operation for persistentvolumeclaims test-oadp-440/cassandra-data-cassandra-2: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-96e25279-1ec8-4cd0-aee6-134ec041501b.4cfe7730-805d-48d08e014 Items to Update: datauploads.velero.io openshift-adp/cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-z9d2b Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-b8fdb9af-b9d9-4bff-935f-05f8b36a5569: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:33:40 +0000 UTC Started: 2024-04-16 22:33:40 +0000 UTC Updated: 2024-04-16 22:34:42 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/ControllerRevision: - test-oadp-440/cassandra-58bc85766c apps/v1/StatefulSet: - test-oadp-440/cassandra authorization.openshift.io/v1/RoleBinding: - test-oadp-440/admin - test-oadp-440/system:deployers - test-oadp-440/system:image-builders - test-oadp-440/system:image-pullers - test-oadp-440/system:openshift:scc:privileged discovery.k8s.io/v1/EndpointSlice: - test-oadp-440/cassandra-644pn operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-440/volsync-product.v0.9.1 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-440/admin - test-oadp-440/system:deployers - test-oadp-440/system:image-builders - test-oadp-440/system:image-pullers - test-oadp-440/system:openshift:scc:privileged v1/ConfigMap: - test-oadp-440/kube-root-ca.crt - test-oadp-440/openshift-service-ca.crt v1/Endpoints: - test-oadp-440/cassandra v1/Event: - test-oadp-440/cassandra-0.17c6e34a7b66569d - test-oadp-440/cassandra-0.17c6e34b087b10ca - test-oadp-440/cassandra-0.17c6e34c169b1cc3 - test-oadp-440/cassandra-0.17c6e34c17e535fa - test-oadp-440/cassandra-0.17c6e34c2d923d2c - test-oadp-440/cassandra-0.17c6e34c343a7937 - test-oadp-440/cassandra-0.17c6e34c350c0371 - test-oadp-440/cassandra-1.17c6e34d90d20ab4 - test-oadp-440/cassandra-1.17c6e34df3ed4dc3 - test-oadp-440/cassandra-1.17c6e34f3f0a19ed - test-oadp-440/cassandra-1.17c6e34f404a0b0d - test-oadp-440/cassandra-1.17c6e34f52eca727 - test-oadp-440/cassandra-1.17c6e34f59e351ad - test-oadp-440/cassandra-1.17c6e34f5ae4e132 - test-oadp-440/cassandra-2.17c6e3508e3e966f - test-oadp-440/cassandra-2.17c6e350f588960f - test-oadp-440/cassandra-2.17c6e35271d7b014 - test-oadp-440/cassandra-2.17c6e352733dee56 - test-oadp-440/cassandra-2.17c6e354714c43c1 - test-oadp-440/cassandra-2.17c6e35478ff89d8 - test-oadp-440/cassandra-2.17c6e35479e3c450 - test-oadp-440/cassandra-2.17c6e35870ba4e5d - test-oadp-440/cassandra-2.17c6e35d7c0865e1 - test-oadp-440/cassandra-2.17c6e36075057ff0 - test-oadp-440/cassandra-data-cassandra-0.17c6e3494fc14dfd - test-oadp-440/cassandra-data-cassandra-0.17c6e349515e7fbd - test-oadp-440/cassandra-data-cassandra-0.17c6e34951657a1c - test-oadp-440/cassandra-data-cassandra-0.17c6e34a53388ac8 - test-oadp-440/cassandra-data-cassandra-1.17c6e34c6543dc61 - test-oadp-440/cassandra-data-cassandra-1.17c6e34c66a7917b - test-oadp-440/cassandra-data-cassandra-1.17c6e34c66a827d1 - test-oadp-440/cassandra-data-cassandra-1.17c6e34d68d099b2 - test-oadp-440/cassandra-data-cassandra-2.17c6e34f62c0822b - test-oadp-440/cassandra-data-cassandra-2.17c6e34f641ceb33 - test-oadp-440/cassandra-data-cassandra-2.17c6e34f6421783e - test-oadp-440/cassandra-data-cassandra-2.17c6e3506669ae23 - test-oadp-440/cassandra.17c6e3494fb7592b - test-oadp-440/cassandra.17c6e3495045db40 - test-oadp-440/cassandra.17c6e34c653c2689 - test-oadp-440/cassandra.17c6e34c65d80efd - test-oadp-440/cassandra.17c6e34f62aadf31 - test-oadp-440/cassandra.17c6e34f633899f9 v1/Namespace: - test-oadp-440 v1/PersistentVolume: - pvc-0685d294-921b-493d-9f48-89fe4665a5d1 - pvc-319b54d1-e784-4c2d-87fb-8b684ec50997 - pvc-4cfe7730-805d-48dc-9df5-121577441e9b v1/PersistentVolumeClaim: - test-oadp-440/cassandra-data-cassandra-0 - test-oadp-440/cassandra-data-cassandra-1 - test-oadp-440/cassandra-data-cassandra-2 v1/Pod: - test-oadp-440/cassandra-0 - test-oadp-440/cassandra-1 - test-oadp-440/cassandra-2 v1/Secret: - test-oadp-440/builder-dockercfg-nd5t9 - test-oadp-440/builder-token-ng4n5 - test-oadp-440/default-dockercfg-9x2s4 - test-oadp-440/default-token-mvjkn - test-oadp-440/deployer-dockercfg-jnlrv - test-oadp-440/deployer-token-b8jtr v1/Service: - test-oadp-440/cassandra v1/ServiceAccount: - test-oadp-440/builder - test-oadp-440/default - test-oadp-440/deployer Velero-Native Snapshots: 2024/04/16 22:35:01 { "metadata": { "name": "cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5", "namespace": "openshift-adp", "uid": "96e25279-1ec8-4cd0-aee6-134ec041501b", "resourceVersion": "45293", "generation": 10, "creationTimestamp": "2024-04-16T22:33:20Z", "labels": { "velero.io/storage-location": "ts-dpa-1" }, "annotations": { "velero.io/resource-timeout": "10m0s", "velero.io/source-cluster-k8s-gitversion": "v1.29.3+5fa1806", "velero.io/source-cluster-k8s-major-version": "1", "velero.io/source-cluster-k8s-minor-version": "29" }, "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:33:20Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:hooks": {}, "f:includedNamespaces": {}, "f:metadata": {}, "f:snapshotMoveData": {} }, "f:status": {} } }, { "manager": "velero-server", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:34:44Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:annotations": { ".": {}, "f:velero.io/resource-timeout": {}, "f:velero.io/source-cluster-k8s-gitversion": {}, "f:velero.io/source-cluster-k8s-major-version": {}, "f:velero.io/source-cluster-k8s-minor-version": {} }, "f:labels": { ".": {}, "f:velero.io/storage-location": {} } }, "f:spec": { "f:csiSnapshotTimeout": {}, "f:defaultVolumesToFsBackup": {}, "f:itemOperationTimeout": {}, "f:storageLocation": {}, "f:ttl": {} }, "f:status": { "f:backupItemOperationsAttempted": {}, "f:backupItemOperationsFailed": {}, "f:completionTimestamp": {}, "f:errors": {}, "f:expiration": {}, "f:formatVersion": {}, "f:phase": {}, "f:progress": { ".": {}, "f:itemsBackedUp": {}, "f:totalItems": {} }, "f:startTimestamp": {}, "f:version": {} } } } ] }, "spec": { "metadata": {}, "includedNamespaces": [ "test-oadp-440" ], "ttl": "720h0m0s", "hooks": {}, "storageLocation": "ts-dpa-1", "defaultVolumesToFsBackup": false, "csiSnapshotTimeout": "10m0s", "itemOperationTimeout": "4h0m0s", "snapshotMoveData": true }, "status": { "version": 1, "formatVersion": "1.1.0", "expiration": "2024-05-16T22:33:20Z", "phase": "PartiallyFailed", "startTimestamp": "2024-04-16T22:33:20Z", "completionTimestamp": "2024-04-16T22:34:44Z", "errors": 3, "progress": { "totalItems": 80, "itemsBackedUp": 80 }, "backupItemOperationsAttempted": 3, "backupItemOperationsFailed": 3 } } [FAILED] in [It] - /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 @ 04/16/24 22:35:01.615 2024/04/16 22:35:01 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 STEP: Get the failed spec name @ 04/16/24 22:35:01.615 2024/04/16 22:35:01 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application STEP: Create a folder for all must-gather files if it doesn't exists already @ 04/16/24 22:35:01.616 STEP: Create a folder for the failed spec if it doesn't exists already @ 04/16/24 22:35:01.616 STEP: Run must-gather because the spec failed @ 04/16/24 22:35:01.616 2024/04/16 22:35:01 [adm must-gather --dest-dir logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1] STEP: Find must-gather folder and rename it to a shorter more readable name @ 04/16/24 22:35:19.716 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application/must-gather already exists, skipping renaming the folder 2024/04/16 22:35:19 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:36:14 2024-04-16 22:35:21,258 p=27387 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:35:21,259 p=27387 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:35:21,511 p=27387 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:35:21,511 p=27387 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:35:21,765 p=27387 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:35:21,765 p=27387 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:35:21,783 p=27387 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:35:21,783 p=27387 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:35:22,086 p=27387 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:35:22,086 p=27387 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:35:22,114 p=27387 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:35:22,115 p=27387 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:35:22,141 p=27387 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:35:22,141 p=27387 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:35:22,152 p=27387 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:35:22,709 p=27387 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:35:22,709 p=27387 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:36:13,802 p=27387 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** 2024-04-16 22:36:13,802 p=27387 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:36:14,142 p=27387 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:36:14,143 p=27387 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:36:14 Cleaning setup resources for the backup 2024/04/16 22:36:14 Setting new default StorageClass 'gp3-csi' 2024/04/16 22:36:14 Checking default storage class count Run the command: oc get sc 2024/04/16 22:36:14 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 45m gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 45m 2024/04/16 22:36:14 Deleting VolumeSnapshotClass 'example-snapclass' Attempt #2 Failed. Retrying ↺ @ 04/16/24 22:36:14.478 2024/04/16 22:36:14 Delete all downloadrequest cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-08ee3bf9-c0a9-4e82-8c47-da62352fd3be cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-6481e11e-0403-4cff-b260-fe0a97b89d18 cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-89877f24-252a-4136-ba63-a905436830f3 cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-b7b30783-86e9-4773-ae14-ca2f5b6a92f8 cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-6afa1ce9-3ab6-4a1f-a64a-586f620ca6a1 cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-7ced49bf-ef15-4b05-a075-595b982d787c cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-89aa7dde-dc8e-4a2a-8eaf-a741c0f8b2f0 cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-9a57d9dc-83ee-48c8-b683-58fc3c2ac2a7 cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-9d081bbc-798e-4cdb-ac70-fca3c7a6c50a cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-9e6b9cc6-0f84-48ab-a094-84861649cdd2 cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-ff76e624-68a1-4483-8d90-2f0d3d489d7e mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-3e19ac05-74d6-4de6-81d8-405d59804efe mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-4bb5240b-8af1-47bb-a6ff-29773b7407b0 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-681b4eb3-8a17-411d-a909-dc5b6951e874 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-7738bd79-46a0-4a7f-9469-55580ed349bc mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-d19194c9-93f8-4206-b146-5d31c2909c2d mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-d209a44a-5e08-4d9e-9fab-38120168f3f8 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-e63fdf5d-4b2b-490b-abfc-80594c7485b3 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-f05c3388-e3c5-494d-bc81-6a91674dbf39 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-4439d254-87fe-4175-9e12-aa0c2b54b2b0 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-72960572-dc1c-4a6c-990f-789122ff86cd mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-9668fc4f-c34d-4b26-add2-ba3f15099d46 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-df7a5b50-c78a-4200-965b-003165c627ef STEP: Create DPA CR @ 04/16/24 22:36:15.195 2024/04/16 22:36:15 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "9636804b-cc90-493e-85d1-18c38a97b0f8", "resourceVersion": "46179", "generation": 1, "creationTimestamp": "2024-04-16T22:36:15Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:36:15Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:36:15.272 2024/04/16 22:36:15 Waiting for velero pod to be running 2024/04/16 22:36:20 pod: velero-5778f4ffcb-nw6x8 is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:36:17 +0000 UTC } {Initialized False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:36:15 +0000 UTC ContainersNotInitialized containers with incomplete status: [velero-plugin-for-csi]} {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:36:15 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:36:15 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:36:15 +0000 UTC }] 10.0.20.65 10.131.0.31 [{10.131.0.31}] 2024-04-16 22:36:15 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:36:16 +0000 UTC,FinishedAt:2024-04-16 22:36:16 +0000 UTC,ContainerID:cri-o://b66bfd166a48eeb56e31176e4bb84d8235f9023fcd60339953a6867ddcd8eb58,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://b66bfd166a48eeb56e31176e4bb84d8235f9023fcd60339953a6867ddcd8eb58 0xc000f87669} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:36:17 +0000 UTC,FinishedAt:2024-04-16 22:36:17 +0000 UTC,ContainerID:cri-o://dc24fb7ca5b24bac78aa171757ece0ab01d4e2f2435f5507580f0a7f5fc1d255,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:716cec49a663d87eff7eaf9ddc948d2861645d601a2e15605cb660c9f98ab489 cri-o://dc24fb7ca5b24bac78aa171757ece0ab01d4e2f2435f5507580f0a7f5fc1d255 0xc000f87679} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:36:18 +0000 UTC,FinishedAt:2024-04-16 22:36:18 +0000 UTC,ContainerID:cri-o://0cc5c6a5581a2bd4a78cb289c0786cdb3e4c3de26bee7883c9b48511078c517b,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f cri-o://0cc5c6a5581a2bd4a78cb289c0786cdb3e4c3de26bee7883c9b48511078c517b 0xc000f87689} {velero-plugin-for-csi {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:c8979074fe11a5d1d0f5ecb0328fabb752f01103a9c03cf34e8df06bcad27f29 0xc000f8768a}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc000f876a6}] Burstable []} 2024/04/16 22:36:25 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:36:25.357 2024/04/16 22:36:25 Snapclass 'example-snapclass' doesn't exist, creating 2024/04/16 22:36:25 Setting new default StorageClass 'gp2-csi' 2024/04/16 22:36:25 Checking default storage class count Run the command: oc get sc 2024/04/16 22:36:25 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 45m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 45m 2024/04/16 22:36:25 Checking for correct number of running NodeAgent pods... STEP: Installing application for case cassandra-e2e @ 04/16/24 22:36:25.656 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pods status (30 retries left). FAILED - RETRYING: [localhost]: Check pods status (29 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (30 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (29 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (28 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (27 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (26 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (25 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (24 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (23 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (22 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (21 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (20 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (19 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (18 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (17 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (16 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=18  changed=9  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/04/16 22:39:16 2024-04-16 22:36:27,345 p=27631 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:36:27,345 p=27631 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:36:27,623 p=27631 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:36:27,623 p=27631 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:36:27,904 p=27631 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:36:27,904 p=27631 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:36:27,920 p=27631 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:36:27,920 p=27631 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:36:28,261 p=27631 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:36:28,261 p=27631 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:36:28,289 p=27631 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:36:28,290 p=27631 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:36:28,317 p=27631 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:36:28,317 p=27631 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:36:28,328 p=27631 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:36:28,905 p=27631 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:36:28,905 p=27631 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:36:29,838 p=27631 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** 2024-04-16 22:36:29,838 p=27631 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:36:30,292 p=27631 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** 2024-04-16 22:36:30,293 p=27631 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:36:30,668 p=27631 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** 2024-04-16 22:36:30,668 p=27631 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:36:31,588 p=27631 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** 2024-04-16 22:36:31,588 p=27631 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:36:32,329 p=27631 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** 2024-04-16 22:36:32,330 p=27631 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:36:44,341 p=27631 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-04-16 22:36:44,342 p=27631 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:39:12,721 p=27631 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-04-16 22:39:12,721 p=27631 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:39:16,519 p=27631 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** 2024-04-16 22:39:16,519 p=27631 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:39:16,643 p=27631 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:39:16,643 p=27631 u=1009100000 n=ansible | localhost : ok=18 changed=9 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:39:16.689 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=15  changed=6  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/04/16 22:39:27 2024-04-16 22:39:18,144 p=28501 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:39:18,145 p=28501 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:39:18,384 p=28501 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:39:18,384 p=28501 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:39:18,612 p=28501 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:39:18,613 p=28501 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:39:18,628 p=28501 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:39:18,628 p=28501 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:39:18,914 p=28501 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:39:18,914 p=28501 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:39:18,941 p=28501 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:39:18,942 p=28501 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:39:18,966 p=28501 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:39:18,967 p=28501 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:39:18,979 p=28501 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:39:19,537 p=28501 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:39:19,537 p=28501 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:39:20,571 p=28501 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-04-16 22:39:20,572 p=28501 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:39:25,324 p=28501 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-04-16 22:39:25,325 p=28501 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:39:26,565 p=28501 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** 2024-04-16 22:39:26,566 p=28501 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:39:27,065 p=28501 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** 2024-04-16 22:39:27,066 p=28501 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:39:27,246 p=28501 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** 2024-04-16 22:39:27,246 p=28501 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:39:27,260 p=28501 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:39:27,260 p=28501 u=1009100000 n=ansible | localhost : ok=15 changed=6 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Creating backup cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5 @ 04/16/24 22:39:27.308 2024/04/16 22:39:27 Wait until backup cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5 is completed backup phase: InProgress DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg phase: Accepted DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg and status: Accepted 2024/04/16 22:39:47 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "4e2f524a-1b3b-48b4-9259-dd72b3188d5b", "resourceVersion": "47642", "generation": 2, "creationTimestamp": "2024-04-16T22:39:36Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.d8efc1a2-3db1-4a4603f5e", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "d8efc1a2-3db1-4a4d-a161-3f805ba4525f" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:36Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:36Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-99nfx", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:39:36Z", "progress": {} } } DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd phase: Accepted DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd and status: Accepted 2024/04/16 22:39:47 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "24c62850-8d51-44ef-8572-9d6dc61e9f6e", "resourceVersion": "47751", "generation": 2, "creationTimestamp": "2024-04-16T22:39:42Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.72be8977-059b-4f28f2318", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "72be8977-059b-4f29-910b-0b945aa007bd" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:42Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:42Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-bbz6t", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:39:42Z", "progress": {} } } DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh phase: Accepted DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh and status: Accepted 2024/04/16 22:39:47 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "f1ab6932-6b86-48d2-8fec-8aa8c67b25ff", "resourceVersion": "47866", "generation": 2, "creationTimestamp": "2024-04-16T22:39:47Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.55207899-3eed-4b69de365", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "55207899-3eed-4b65-9640-cbd00f928184" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:47Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:47Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-dm2df", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:39:47Z", "progress": {} } } backup phase: WaitingForPluginOperations DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg phase: Accepted DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg and status: Accepted 2024/04/16 22:40:07 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "4e2f524a-1b3b-48b4-9259-dd72b3188d5b", "resourceVersion": "47642", "generation": 2, "creationTimestamp": "2024-04-16T22:39:36Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.d8efc1a2-3db1-4a4603f5e", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "d8efc1a2-3db1-4a4d-a161-3f805ba4525f" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:36Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:36Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-99nfx", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:39:36Z", "progress": {} } } DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd phase: Failed DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd and status: Failed 2024/04/16 22:40:07 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "24c62850-8d51-44ef-8572-9d6dc61e9f6e", "resourceVersion": "48222", "generation": 3, "creationTimestamp": "2024-04-16T22:39:42Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.72be8977-059b-4f28f2318", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "72be8977-059b-4f29-910b-0b945aa007bd" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:42Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:40:06Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-bbz6t", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-bab19c42-310b-4769-b5ff-3ba073b78964: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:39:42Z", "completionTimestamp": "2024-04-16T22:40:06Z", "progress": {} } } DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh phase: Accepted DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh and status: Accepted 2024/04/16 22:40:07 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "f1ab6932-6b86-48d2-8fec-8aa8c67b25ff", "resourceVersion": "47866", "generation": 2, "creationTimestamp": "2024-04-16T22:39:47Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.55207899-3eed-4b69de365", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "55207899-3eed-4b65-9640-cbd00f928184" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:47Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:47Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-dm2df", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:39:47Z", "progress": {} } } backup phase: WaitingForPluginOperationsPartiallyFailed DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg phase: Failed DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg and status: Failed 2024/04/16 22:40:27 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "4e2f524a-1b3b-48b4-9259-dd72b3188d5b", "resourceVersion": "48299", "generation": 3, "creationTimestamp": "2024-04-16T22:39:36Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.d8efc1a2-3db1-4a4603f5e", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "d8efc1a2-3db1-4a4d-a161-3f805ba4525f" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:36Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:40:13Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-99nfx", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-926bf6f4-cd42-4eca-a563-8288b37542bd: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:39:36Z", "completionTimestamp": "2024-04-16T22:40:13Z", "progress": {} } } DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd phase: Failed DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd and status: Failed 2024/04/16 22:40:27 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "24c62850-8d51-44ef-8572-9d6dc61e9f6e", "resourceVersion": "48222", "generation": 3, "creationTimestamp": "2024-04-16T22:39:42Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.72be8977-059b-4f28f2318", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "72be8977-059b-4f29-910b-0b945aa007bd" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:42Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:40:06Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-bbz6t", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-bab19c42-310b-4769-b5ff-3ba073b78964: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:39:42Z", "completionTimestamp": "2024-04-16T22:40:06Z", "progress": {} } } DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh phase: Accepted DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh and status: Accepted 2024/04/16 22:40:27 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "f1ab6932-6b86-48d2-8fec-8aa8c67b25ff", "resourceVersion": "47866", "generation": 2, "creationTimestamp": "2024-04-16T22:39:47Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.55207899-3eed-4b69de365", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "55207899-3eed-4b65-9640-cbd00f928184" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:47Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:47Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-dm2df", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:39:47Z", "progress": {} } } backup phase: WaitingForPluginOperationsPartiallyFailed DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg phase: Failed DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg and status: Failed 2024/04/16 22:40:47 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "4e2f524a-1b3b-48b4-9259-dd72b3188d5b", "resourceVersion": "48299", "generation": 3, "creationTimestamp": "2024-04-16T22:39:36Z", "labels": { "velero.io/accepted-by": "ip-10-0-20-65.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.d8efc1a2-3db1-4a4603f5e", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "d8efc1a2-3db1-4a4d-a161-3f805ba4525f" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:36Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:40:13Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-0-99nfx", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-0", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-926bf6f4-cd42-4eca-a563-8288b37542bd: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:39:36Z", "completionTimestamp": "2024-04-16T22:40:13Z", "progress": {} } } DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd phase: Failed DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd and status: Failed 2024/04/16 22:40:47 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "24c62850-8d51-44ef-8572-9d6dc61e9f6e", "resourceVersion": "48222", "generation": 3, "creationTimestamp": "2024-04-16T22:39:42Z", "labels": { "velero.io/accepted-by": "ip-10-0-106-243.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.72be8977-059b-4f28f2318", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "72be8977-059b-4f29-910b-0b945aa007bd" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "managedFields": [ { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:42Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } }, { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:40:06Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:completionTimestamp": {}, "f:message": {}, "f:phase": {}, "f:startTimestamp": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-1-bbz6t", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-1", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Failed", "message": "error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-bab19c42-310b-4769-b5ff-3ba073b78964: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil", "startTimestamp": "2024-04-16T22:39:42Z", "completionTimestamp": "2024-04-16T22:40:06Z", "progress": {} } } DataUpload cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh phase: Accepted DataUpload Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh and status: Accepted 2024/04/16 22:40:47 { "kind": "DataUpload", "apiVersion": "velero.io/v2alpha1", "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh", "generateName": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-", "namespace": "openshift-adp", "uid": "f1ab6932-6b86-48d2-8fec-8aa8c67b25ff", "resourceVersion": "47866", "generation": 2, "creationTimestamp": "2024-04-16T22:39:47Z", "labels": { "velero.io/accepted-by": "ip-10-0-89-31.us-east-2.compute.internal", "velero.io/async-operation-id": "du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.55207899-3eed-4b69de365", "velero.io/backup-name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "velero.io/backup-uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "velero.io/pvc-uid": "55207899-3eed-4b65-9640-cbd00f928184" }, "ownerReferences": [ { "apiVersion": "velero.io/v1", "kind": "Backup", "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "controller": true } ], "finalizers": [ "velero.io/data-upload-download-finalizer" ], "managedFields": [ { "manager": "node-agent-server", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:47Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:finalizers": { ".": {}, "v:\"velero.io/data-upload-download-finalizer\"": {} }, "f:labels": { "f:velero.io/accepted-by": {} } }, "f:status": { "f:phase": {}, "f:startTimestamp": {} } } }, { "manager": "velero-plugin-for-csi", "operation": "Update", "apiVersion": "velero.io/v2alpha1", "time": "2024-04-16T22:39:47Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:generateName": {}, "f:labels": { ".": {}, "f:velero.io/async-operation-id": {}, "f:velero.io/backup-name": {}, "f:velero.io/backup-uid": {}, "f:velero.io/pvc-uid": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b\"}": {} } }, "f:spec": { ".": {}, "f:backupStorageLocation": {}, "f:csiSnapshot": { ".": {}, "f:snapshotClass": {}, "f:storageClass": {}, "f:volumeSnapshot": {} }, "f:operationTimeout": {}, "f:snapshotType": {}, "f:sourceNamespace": {}, "f:sourcePVC": {} }, "f:status": { ".": {}, "f:progress": {} } } } ] }, "spec": { "snapshotType": "CSI", "csiSnapshot": { "volumeSnapshot": "velero-cassandra-data-cassandra-2-dm2df", "storageClass": "gp2-csi", "snapshotClass": "" }, "sourcePVC": "cassandra-data-cassandra-2", "backupStorageLocation": "ts-dpa-1", "sourceNamespace": "test-oadp-440", "operationTimeout": "10m0s" }, "status": { "phase": "Accepted", "startTimestamp": "2024-04-16T22:39:47Z", "progress": {} } } backup phase: PartiallyFailed STEP: Verify backup cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:41:07.601 2024/04/16 22:41:07 Run velero describe on the backup 2024/04/16 22:41:08 Exec stderr: "" 2024/04/16 22:41:08 Name: cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: PartiallyFailed (run `velero backup logs cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5` for more information) Errors: Velero: Cluster: Namespaces: Namespaces: Included: test-oadp-440 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-04-16 22:39:27 +0000 UTC Completed: 2024-04-16 22:40:54 +0000 UTC Expiration: 2024-05-16 22:39:27 +0000 UTC Total items to be backed up: 80 Items backed up: 80 Backup Item Operations: Operation for persistentvolumeclaims test-oadp-440/cassandra-data-cassandra-0: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.d8efc1a2-3db1-4a4603f5e Items to Update: datauploads.velero.io openshift-adp/cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-sx5cg Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-926bf6f4-cd42-4eca-a563-8288b37542bd: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:39:36 +0000 UTC Started: 2024-04-16 22:39:36 +0000 UTC Updated: 2024-04-16 22:40:13 +0000 UTC Operation for persistentvolumeclaims test-oadp-440/cassandra-data-cassandra-1: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.72be8977-059b-4f28f2318 Items to Update: datauploads.velero.io openshift-adp/cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-7s7hd Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-bab19c42-310b-4769-b5ff-3ba073b78964: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:39:42 +0000 UTC Started: 2024-04-16 22:39:42 +0000 UTC Updated: 2024-04-16 22:40:06 +0000 UTC Operation for persistentvolumeclaims test-oadp-440/cassandra-data-cassandra-2: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b.55207899-3eed-4b69de365 Items to Update: datauploads.velero.io openshift-adp/cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fmksh Phase: Failed Operation Error: error to expose snapshot: error to remove protect from volume snapshot content: error to update VolumeSnapshotContent snapcontent-b461b893-3397-42bf-9a67-fa972baa9f3e: admission webhook "volumesnapshotclasses.snapshot.storage.k8s.io" denied the request: Spec.SourceVolumeMode is immutable but was changed from Filesystem to nil Progress description: Failed Created: 2024-04-16 22:39:47 +0000 UTC Started: 2024-04-16 22:39:47 +0000 UTC Updated: 2024-04-16 22:40:49 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/ControllerRevision: - test-oadp-440/cassandra-58bc85766c apps/v1/StatefulSet: - test-oadp-440/cassandra authorization.openshift.io/v1/RoleBinding: - test-oadp-440/admin - test-oadp-440/system:deployers - test-oadp-440/system:image-builders - test-oadp-440/system:image-pullers - test-oadp-440/system:openshift:scc:privileged discovery.k8s.io/v1/EndpointSlice: - test-oadp-440/cassandra-9lsbq operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-440/volsync-product.v0.9.1 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-440/admin - test-oadp-440/system:deployers - test-oadp-440/system:image-builders - test-oadp-440/system:image-pullers - test-oadp-440/system:openshift:scc:privileged v1/ConfigMap: - test-oadp-440/kube-root-ca.crt - test-oadp-440/openshift-service-ca.crt v1/Endpoints: - test-oadp-440/cassandra v1/Event: - test-oadp-440/cassandra-0.17c6e3a2f51515fa - test-oadp-440/cassandra-0.17c6e3a380b10969 - test-oadp-440/cassandra-0.17c6e3a424ebf729 - test-oadp-440/cassandra-0.17c6e3a42638cac8 - test-oadp-440/cassandra-0.17c6e3a4488fd8f6 - test-oadp-440/cassandra-0.17c6e3a44fe0db88 - test-oadp-440/cassandra-0.17c6e3a450af5479 - test-oadp-440/cassandra-1.17c6e3a58697fa05 - test-oadp-440/cassandra-1.17c6e3a5ed9b2ab9 - test-oadp-440/cassandra-1.17c6e3a6da3d8409 - test-oadp-440/cassandra-1.17c6e3a6db90bf0c - test-oadp-440/cassandra-1.17c6e3a6ef6e1708 - test-oadp-440/cassandra-1.17c6e3a6f66a8887 - test-oadp-440/cassandra-1.17c6e3a6f736e138 - test-oadp-440/cassandra-2.17c6e3a84ed27655 - test-oadp-440/cassandra-2.17c6e3a8dc7be630 - test-oadp-440/cassandra-2.17c6e3aa79765e62 - test-oadp-440/cassandra-2.17c6e3aa7adc76ac - test-oadp-440/cassandra-2.17c6e3aa8e995d07 - test-oadp-440/cassandra-2.17c6e3aa962a9f31 - test-oadp-440/cassandra-2.17c6e3aa97035d75 - test-oadp-440/cassandra-2.17c6e3ae52c92530 - test-oadp-440/cassandra-2.17c6e3b39d883021 - test-oadp-440/cassandra-2.17c6e3b6d456bbf3 - test-oadp-440/cassandra-data-cassandra-0.17c6e3a1c8ac9b7b - test-oadp-440/cassandra-data-cassandra-0.17c6e3a1cb14212e - test-oadp-440/cassandra-data-cassandra-0.17c6e3a1cb165c79 - test-oadp-440/cassandra-data-cassandra-0.17c6e3a2cdca15b6 - test-oadp-440/cassandra-data-cassandra-1.17c6e3a45aeb9a6b - test-oadp-440/cassandra-data-cassandra-1.17c6e3a45c6ca07e - test-oadp-440/cassandra-data-cassandra-1.17c6e3a45c79fff1 - test-oadp-440/cassandra-data-cassandra-1.17c6e3a55efdbbf3 - test-oadp-440/cassandra-data-cassandra-2.17c6e3a72357c830 - test-oadp-440/cassandra-data-cassandra-2.17c6e3a724ba9a56 - test-oadp-440/cassandra-data-cassandra-2.17c6e3a724c00160 - test-oadp-440/cassandra-data-cassandra-2.17c6e3a826afc913 - test-oadp-440/cassandra.17c6e3a1c89d0452 - test-oadp-440/cassandra.17c6e3a1c9a3ec23 - test-oadp-440/cassandra.17c6e3a45addf6cd - test-oadp-440/cassandra.17c6e3a45b8f1b6a - test-oadp-440/cassandra.17c6e3a723444a03 - test-oadp-440/cassandra.17c6e3a723df8357 v1/Namespace: - test-oadp-440 v1/PersistentVolume: - pvc-55207899-3eed-4b65-9640-cbd00f928184 - pvc-72be8977-059b-4f29-910b-0b945aa007bd - pvc-d8efc1a2-3db1-4a4d-a161-3f805ba4525f v1/PersistentVolumeClaim: - test-oadp-440/cassandra-data-cassandra-0 - test-oadp-440/cassandra-data-cassandra-1 - test-oadp-440/cassandra-data-cassandra-2 v1/Pod: - test-oadp-440/cassandra-0 - test-oadp-440/cassandra-1 - test-oadp-440/cassandra-2 v1/Secret: - test-oadp-440/builder-dockercfg-8ph9b - test-oadp-440/builder-token-fqfdk - test-oadp-440/default-dockercfg-ttvb2 - test-oadp-440/default-token-t7vh6 - test-oadp-440/deployer-dockercfg-pbnnv - test-oadp-440/deployer-token-swl8h v1/Service: - test-oadp-440/cassandra v1/ServiceAccount: - test-oadp-440/builder - test-oadp-440/default - test-oadp-440/deployer Velero-Native Snapshots: 2024/04/16 22:41:08 { "metadata": { "name": "cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5", "namespace": "openshift-adp", "uid": "e3b27bbd-283f-4bd3-b7e6-7a3bc14ae08b", "resourceVersion": "48523", "generation": 10, "creationTimestamp": "2024-04-16T22:39:27Z", "labels": { "velero.io/storage-location": "ts-dpa-1" }, "annotations": { "velero.io/resource-timeout": "10m0s", "velero.io/source-cluster-k8s-gitversion": "v1.29.3+5fa1806", "velero.io/source-cluster-k8s-major-version": "1", "velero.io/source-cluster-k8s-minor-version": "29" }, "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:39:27Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:hooks": {}, "f:includedNamespaces": {}, "f:metadata": {}, "f:snapshotMoveData": {} }, "f:status": {} } }, { "manager": "velero-server", "operation": "Update", "apiVersion": "velero.io/v1", "time": "2024-04-16T22:40:54Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:annotations": { ".": {}, "f:velero.io/resource-timeout": {}, "f:velero.io/source-cluster-k8s-gitversion": {}, "f:velero.io/source-cluster-k8s-major-version": {}, "f:velero.io/source-cluster-k8s-minor-version": {} }, "f:labels": { ".": {}, "f:velero.io/storage-location": {} } }, "f:spec": { "f:csiSnapshotTimeout": {}, "f:defaultVolumesToFsBackup": {}, "f:itemOperationTimeout": {}, "f:storageLocation": {}, "f:ttl": {} }, "f:status": { "f:backupItemOperationsAttempted": {}, "f:backupItemOperationsFailed": {}, "f:completionTimestamp": {}, "f:errors": {}, "f:expiration": {}, "f:formatVersion": {}, "f:phase": {}, "f:progress": { ".": {}, "f:itemsBackedUp": {}, "f:totalItems": {} }, "f:startTimestamp": {}, "f:version": {} } } } ] }, "spec": { "metadata": {}, "includedNamespaces": [ "test-oadp-440" ], "ttl": "720h0m0s", "hooks": {}, "storageLocation": "ts-dpa-1", "defaultVolumesToFsBackup": false, "csiSnapshotTimeout": "10m0s", "itemOperationTimeout": "4h0m0s", "snapshotMoveData": true }, "status": { "version": 1, "formatVersion": "1.1.0", "expiration": "2024-05-16T22:39:27Z", "phase": "PartiallyFailed", "startTimestamp": "2024-04-16T22:39:27Z", "completionTimestamp": "2024-04-16T22:40:54Z", "errors": 3, "progress": { "totalItems": 80, "itemsBackedUp": 80 }, "backupItemOperationsAttempted": 3, "backupItemOperationsFailed": 3 } } [FAILED] in [It] - /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 @ 04/16/24 22:41:08.229 2024/04/16 22:41:08 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 STEP: Get the failed spec name @ 04/16/24 22:41:08.23 2024/04/16 22:41:08 The failed spec name is: [datamover] DataMover: Backup/Restore stateful application with CSI [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application STEP: Create a folder for all must-gather files if it doesn't exists already @ 04/16/24 22:41:08.23 STEP: Create a folder for the failed spec if it doesn't exists already @ 04/16/24 22:41:08.23 STEP: Run must-gather because the spec failed @ 04/16/24 22:41:08.23 2024/04/16 22:41:08 [adm must-gather --dest-dir logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application --image registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1] STEP: Find must-gather folder and rename it to a shorter more readable name @ 04/16/24 22:41:19.363 The folder logs/It_datamover_DataMover_Backup_Restore_stateful_application_with_CSI_tc-id_OADP-440_interop_bug-id_OADP-2742_Cassandra_application/must-gather already exists, skipping renaming the folder 2024/04/16 22:41:19 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:41:43 2024-04-16 22:41:20,916 p=28845 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:41:20,916 p=28845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:41:21,155 p=28845 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:41:21,155 p=28845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:41:21,400 p=28845 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:41:21,400 p=28845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:41:21,418 p=28845 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:41:21,418 p=28845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:21,719 p=28845 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:41:21,719 p=28845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:21,751 p=28845 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:41:21,751 p=28845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:21,776 p=28845 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:41:21,776 p=28845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:21,788 p=28845 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:41:22,345 p=28845 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:41:22,345 p=28845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:43,332 p=28845 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-440] *** 2024-04-16 22:41:43,332 p=28845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:41:43,617 p=28845 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:41:43,617 p=28845 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:41:43 Cleaning setup resources for the backup 2024/04/16 22:41:43 Setting new default StorageClass 'gp3-csi' 2024/04/16 22:41:43 Checking default storage class count Run the command: oc get sc 2024/04/16 22:41:43 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 51m gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 51m 2024/04/16 22:41:43 Deleting VolumeSnapshotClass 'example-snapclass' • [FAILED] [1007.305 seconds] [datamover] DataMover: Backup/Restore stateful application with CSI  [It] [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:130 [FAILED] Unexpected error: <*errors.errorString | 0xc000dd9e30>: backup phase is: PartiallyFailed; expected: Completed validation errors: [] velero failure logs: velero container contains "level=error" in line#157: time="2024-04-16T22:36:43Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1)" controller=backup-storage-location logSource="/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:194" { s: "backup phase is: PartiallyFailed; expected: Completed\nvalidation errors: []\nvelero failure logs: velero container contains \"level=error\" in line#157: time=\"2024-04-16T22:36:43Z\" level=error msg=\"Current BackupStorageLocations available/unavailable/unknown: 0/0/1)\" controller=backup-storage-location logSource=\"/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:194\"\n", } occurred In [It] at: /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 @ 04/16/24 22:41:08.229 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ SSSSSSSSSSSSSS ------------------------------ [skip-disconnected] Restore hooks tests Successful Init hook [tc-id:OADP-164][interop] [bug-id:OADP-1076] MySQL app with Restic /alabama/cspi/e2e/hooks/restore_hooks.go:129 2024/04/16 22:41:44 Delete all downloadrequest cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-5691aa2c-aa30-42c6-b3dc-49286b1faabc cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-878dad04-e46e-408b-95c1-23983a8d568f cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-9f681f01-1d6b-480b-a2b9-7448de92315f cassandra-e2e-2825bb83-fc40-11ee-99f6-0a580a8231b5-a32ed843-6004-489b-92aa-8d8faed02674 cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-45cd5010-8448-4135-b71c-c3cf2e016a2d cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-49547db0-8b12-4340-8aa4-8ef4d2e70689 cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-69107780-eb89-45c5-ac3b-08cfd08baa8d cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-967693c0-ea8b-4951-a2f3-fcd1002cf281 cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-e5eae791-17f6-4809-8d95-c768d6eb0b21 cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-f44f5803-4e8a-488a-9e85-aa029e8b3a8e cassandra-e2e-bbb4dbab-fc41-11ee-99f6-0a580a8231b5-fd3b2b45-0d66-4f59-bd59-2a93374ab4ac cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-8fd27342-3bfe-4fda-885e-7785428cb407 cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-cec64bea-c179-4e57-823f-3a9e5e913f80 cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-dbcaf0ad-6916-464b-b735-26ce2c9886c7 cassandra-e2e-d985dc95-fc40-11ee-99f6-0a580a8231b5-fe117d7e-b421-4b51-af60-7c433a724734 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-61e8abc8-e01a-4c58-846d-9dcc855d6030 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-9118ac58-625f-4dee-ac79-d2cb86407fe3 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-a2266bfb-b5b8-4583-9bd8-c8068889b980 mysql-0cd0b312-fc3f-11ee-99f6-0a580a8231b5-c88eb5a6-19d8-4664-bf27-406e3eee36c7 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-00e02203-6495-482c-a985-8c923170292b mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-19c4ce51-d985-4a5c-91af-c8007f105199 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-ba25241d-7d1e-4b0a-9dca-5d1428643778 mysql-757ef7d3-fc3f-11ee-99f6-0a580a8231b5-fcd9c7dc-f81d-4c9f-a469-280c14fed6f0 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-0e5c7310-73ec-4fc0-bae9-785cc9732ed2 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-11c4c457-4f11-4605-b70b-2c6abeedff2b mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-cb4662fe-3452-42fd-92b9-c2d08276d367 mysql-ce4726a8-fc3f-11ee-99f6-0a580a8231b5-e07d999e-c074-4838-8a52-efdc3fa920c1 STEP: Create DPA CR @ 04/16/24 22:41:44.82 2024/04/16 22:41:44 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "4abf920b-4703-4cbb-9652-ff43c3fe9cc0", "resourceVersion": "49143", "generation": 1, "creationTimestamp": "2024-04-16T22:41:44Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:41:44Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:41:44.9 2024/04/16 22:41:44 Waiting for velero pod to be running 2024/04/16 22:41:44 pod: velero-5778f4ffcb-nw6x8 is not yet running with status: {Succeeded [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:41:44 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:36:21 +0000 UTC PodCompleted } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:41:44 +0000 UTC PodCompleted } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:41:44 +0000 UTC PodCompleted } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:36:15 +0000 UTC }] 10.0.20.65 [] 2024-04-16 22:36:15 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:36:16 +0000 UTC,FinishedAt:2024-04-16 22:36:16 +0000 UTC,ContainerID:cri-o://b66bfd166a48eeb56e31176e4bb84d8235f9023fcd60339953a6867ddcd8eb58,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://b66bfd166a48eeb56e31176e4bb84d8235f9023fcd60339953a6867ddcd8eb58 0xc000963c69} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:36:17 +0000 UTC,FinishedAt:2024-04-16 22:36:17 +0000 UTC,ContainerID:cri-o://dc24fb7ca5b24bac78aa171757ece0ab01d4e2f2435f5507580f0a7f5fc1d255,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:716cec49a663d87eff7eaf9ddc948d2861645d601a2e15605cb660c9f98ab489 cri-o://dc24fb7ca5b24bac78aa171757ece0ab01d4e2f2435f5507580f0a7f5fc1d255 0xc000963c79} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:36:18 +0000 UTC,FinishedAt:2024-04-16 22:36:18 +0000 UTC,ContainerID:cri-o://0cc5c6a5581a2bd4a78cb289c0786cdb3e4c3de26bee7883c9b48511078c517b,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f cri-o://0cc5c6a5581a2bd4a78cb289c0786cdb3e4c3de26bee7883c9b48511078c517b 0xc000963cd9} {velero-plugin-for-csi {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:36:20 +0000 UTC,FinishedAt:2024-04-16 22:36:20 +0000 UTC,ContainerID:cri-o://b4551c2e951acb3c192cd07062e9b19544c11d8e373bffed696dcfbaebaa01ae,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:c8979074fe11a5d1d0f5ecb0328fabb752f01103a9c03cf34e8df06bcad27f29 registry.redhat.io/oadp/oadp-velero-plugin-for-csi-rhel9@sha256:9833ca3b1b53eccc9de4e117bef53bb94d5be02d89bed222ef80dcd221bc1b18 cri-o://b4551c2e951acb3c192cd07062e9b19544c11d8e373bffed696dcfbaebaa01ae 0xc000963ce9}] [{velero {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:36:22 +0000 UTC,FinishedAt:2024-04-16 22:41:44 +0000 UTC,ContainerID:cri-o://62eb9226622004238d150d0c937ee85057cc7837eea9cab5083a5d26f8377ce2,}} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:02f194caab292aea9bcbb22556fca8e2d7d893a629e6e56097501501fbff7ba7 cri-o://62eb9226622004238d150d0c937ee85057cc7837eea9cab5083a5d26f8377ce2 0xc000963cf9}] Burstable []} 2024/04/16 22:41:49 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:41:49.962 2024/04/16 22:41:49 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql-hooks-e2e @ 04/16/24 22:41:49.992 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-164] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). FAILED - RETRYING: [localhost]: Check pod status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=19  changed=10  unreachable=0 failed=0 skipped=5  rescued=0 ignored=0 2024/04/16 22:42:20 2024-04-16 22:41:51,445 p=29071 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:41:51,446 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:41:51,677 p=29071 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:41:51,678 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:41:51,902 p=29071 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:41:51,902 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:41:51,916 p=29071 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:41:51,916 p=29071 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:52,190 p=29071 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:41:52,190 p=29071 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:52,218 p=29071 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:41:52,218 p=29071 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:52,241 p=29071 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:41:52,241 p=29071 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:52,251 p=29071 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:41:52,780 p=29071 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:41:52,780 p=29071 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:53,586 p=29071 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-164] *** 2024-04-16 22:41:53,586 p=29071 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:41:53,971 p=29071 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-04-16 22:41:53,971 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:41:54,972 p=29071 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-04-16 22:41:54,972 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:12,479 p=29071 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-04-16 22:42:12,479 p=29071 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:42:13,080 p=29071 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-04-16 22:42:13,080 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:18,720 p=29071 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:42:18,721 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:19,356 p=29071 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-04-16 22:42:19,356 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:20,204 p=29071 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-04-16 22:42:20,204 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:20,802 p=29071 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-04-16 22:42:20,802 p=29071 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:20,897 p=29071 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:42:20,897 p=29071 u=1009100000 n=ansible | localhost : ok=19 changed=10 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:42:20.939 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2024/04/16 22:42:26 2024-04-16 22:42:22,352 p=29567 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:42:22,352 p=29567 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:22,570 p=29567 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:42:22,570 p=29567 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:22,798 p=29567 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:42:22,798 p=29567 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:22,813 p=29567 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:42:22,813 p=29567 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:42:23,103 p=29567 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:42:23,103 p=29567 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:42:23,131 p=29567 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:42:23,131 p=29567 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:42:23,154 p=29567 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:42:23,154 p=29567 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:42:23,164 p=29567 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:42:23,705 p=29567 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:42:23,705 p=29567 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:42:24,749 p=29567 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-04-16 22:42:24,749 p=29567 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:42:25,070 p=29567 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:42:25,070 p=29567 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:25,500 p=29567 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-04-16 22:42:25,500 p=29567 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:26,093 p=29567 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-04-16 22:42:26,093 p=29567 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:42:26,105 p=29567 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:42:26,105 p=29567 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 2024/04/16 22:42:26 ExtractTarGz: Create file /tmp/tempDir1672081118/world-db/world.sql 2024/04/16 22:42:26 2024/04/16 22:42:26 {{ } { } [{{ } {mysql test-oadp-164 a27611a2-03c9-40b7-ba0e-62ab4aaaad43 49397 0 2024-04-16 22:41:54 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-89-31.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-04-16 22:41:54 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 22:41:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 22:41:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 22:41:59 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-a27611a2-03c9-40b7-ba0e-62ab4aaaad43 0xc000dd96d0 0xc000dd96e0 nil nil} {Bound [ReadWriteOnce] map[storage:{{2147483648 0} {} 2Gi BinarySI}] [] map[] }} {{ } {mysql-1 test-oadp-164 aec31b68-949a-47cf-8e28-f6ef384ee5ad 49402 0 2024-04-16 22:41:54 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-89-31.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-04-16 22:41:54 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 22:41:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 22:41:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 22:41:59 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-aec31b68-949a-47cf-8e28-f6ef384ee5ad 0xc000dd9860 0xc000dd9870 nil nil} {Bound [ReadWriteOnce] map[storage:{{2147483648 0} {} 2Gi BinarySI}] [] map[] }}]} STEP: Creating backup mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 @ 04/16/24 22:42:26.842 2024/04/16 22:42:26 Wait until backup mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 is completed backup phase: InProgress backup phase: InProgress backup phase: InProgress backup phase: Completed 2024/04/16 22:43:46 Verify the PodVolumeBackup is completed successfully and BackupRepository type is matching with DPA.nodeAgent.uploaderType 2024/04/16 22:43:46 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql creationTimestamp: "2024-04-16T22:42:34Z" generateName: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5- generation: 4 labels: velero.io/backup-name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 velero.io/backup-uid: e82be37d-27e1-4d9f-8244-81e06df6a4e3 velero.io/pvc-uid: a27611a2-03c9-40b7-ba0e-62ab4aaaad43 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"e82be37d-27e1-4d9f-8244-81e06df6a4e3"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T22:42:34Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T22:42:37Z" name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5-56zts namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 uid: e82be37d-27e1-4d9f-8244-81e06df6a4e3 resourceVersion: "49615" uid: 769cf3e7-c075-4529-9f2c-4fb960567142 spec: backupStorageLocation: ts-dpa-1 node: ip-10-0-89-31.us-east-2.compute.internal pod: kind: Pod name: mysql-7f77d47fd7-894c7 namespace: test-oadp-164 uid: 027262f9-e32f-4bcd-b807-5fa3cb860e23 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-164 tags: backup: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 backup-uid: e82be37d-27e1-4d9f-8244-81e06df6a4e3 ns: test-oadp-164 pod: mysql-7f77d47fd7-894c7 pod-uid: 027262f9-e32f-4bcd-b807-5fa3cb860e23 pvc-uid: a27611a2-03c9-40b7-ba0e-62ab4aaaad43 volume: mysql-data uploaderType: restic volume: mysql-data status: completionTimestamp: "2024-04-16T22:42:37Z" path: /host_pods/027262f9-e32f-4bcd-b807-5fa3cb860e23/volumes/kubernetes.io~csi/pvc-a27611a2-03c9-40b7-ba0e-62ab4aaaad43/mount phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 snapshotID: 1245c0e3 startTimestamp: "2024-04-16T22:42:34Z" 2024/04/16 22:43:46 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql-1 creationTimestamp: "2024-04-16T22:42:34Z" generateName: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5- generation: 4 labels: velero.io/backup-name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 velero.io/backup-uid: e82be37d-27e1-4d9f-8244-81e06df6a4e3 velero.io/pvc-uid: aec31b68-949a-47cf-8e28-f6ef384ee5ad managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"e82be37d-27e1-4d9f-8244-81e06df6a4e3"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T22:42:34Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T22:43:38Z" name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5-qkh4t namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 uid: e82be37d-27e1-4d9f-8244-81e06df6a4e3 resourceVersion: "49996" uid: 884b0ae9-bf38-4b5f-b1d6-eda57c519846 spec: backupStorageLocation: ts-dpa-1 node: ip-10-0-89-31.us-east-2.compute.internal pod: kind: Pod name: mysql-7f77d47fd7-894c7 namespace: test-oadp-164 uid: 027262f9-e32f-4bcd-b807-5fa3cb860e23 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-164 tags: backup: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 backup-uid: e82be37d-27e1-4d9f-8244-81e06df6a4e3 ns: test-oadp-164 pod: mysql-7f77d47fd7-894c7 pod-uid: 027262f9-e32f-4bcd-b807-5fa3cb860e23 pvc-uid: aec31b68-949a-47cf-8e28-f6ef384ee5ad volume: mysql-data1 uploaderType: restic volume: mysql-data1 status: completionTimestamp: "2024-04-16T22:43:38Z" path: /host_pods/027262f9-e32f-4bcd-b807-5fa3cb860e23/volumes/kubernetes.io~csi/pvc-aec31b68-949a-47cf-8e28-f6ef384ee5ad/mount phase: Completed progress: bytesDone: 105256269 totalBytes: 105256269 snapshotID: be06a34c startTimestamp: "2024-04-16T22:43:35Z" STEP: Verify backup mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:43:46.983 2024/04/16 22:43:47 Backup for case mysql-hooks-e2e succeeded STEP: Delete the appplication resources mysql-hooks-e2e @ 04/16/24 22:43:47.055 STEP: Cleanup Application for case mysql-hooks-e2e @ 04/16/24 22:43:47.055 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-164] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:44:06 2024-04-16 22:43:48,583 p=29873 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:43:48,584 p=29873 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:43:48,827 p=29873 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:43:48,827 p=29873 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:43:49,081 p=29873 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:43:49,081 p=29873 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:43:49,100 p=29873 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:43:49,100 p=29873 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:43:49,398 p=29873 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:43:49,399 p=29873 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:43:49,428 p=29873 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:43:49,428 p=29873 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:43:49,454 p=29873 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:43:49,454 p=29873 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:43:49,470 p=29873 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:43:50,044 p=29873 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:43:50,044 p=29873 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:44:05,985 p=29873 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-164] *** 2024-04-16 22:44:05,985 p=29873 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:44:06,255 p=29873 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:44:06,255 p=29873 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:44:06 Creating restore mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 for case mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 STEP: Create restore mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 from backup mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 @ 04/16/24 22:44:06.304 2024/04/16 22:44:06 Wait until restore mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 is complete restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: Completed 2024/04/16 22:45:46 Verify the PodVolumeBackup and PodVolumeRestore count is equal 2024/04/16 22:45:46 Verify the PodVolumeRestore is completed sucessfully and uploaderType is matching 2024/04/16 22:45:46 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-04-16T22:44:23Z" generateName: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5- generation: 5 labels: velero.io/pod-uid: fb9edef1-ea3b-4743-aa30-5b079edba99e velero.io/pvc-uid: 73c8b449-49db-4915-b9b0-b850a30ba8bc velero.io/restore-name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 velero.io/restore-uid: 5c69649b-a098-4247-9489-91a27b17e1d0 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"5c69649b-a098-4247-9489-91a27b17e1d0"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T22:44:23Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T22:45:38Z" name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5-9xzrd namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 uid: 5c69649b-a098-4247-9489-91a27b17e1d0 resourceVersion: "50992" uid: c08d5fcd-eb6f-4cd7-a182-2b10fa295622 spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-7f77d47fd7-894c7 namespace: test-oadp-164 uid: fb9edef1-ea3b-4743-aa30-5b079edba99e repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-164 snapshotID: be06a34c sourceNamespace: test-oadp-164 uploaderType: restic volume: mysql-data1 status: completionTimestamp: "2024-04-16T22:45:38Z" phase: Completed progress: bytesDone: 105256269 totalBytes: 105256269 startTimestamp: "2024-04-16T22:45:35Z" 2024/04/16 22:45:46 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-04-16T22:44:23Z" generateName: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5- generation: 5 labels: velero.io/pod-uid: fb9edef1-ea3b-4743-aa30-5b079edba99e velero.io/pvc-uid: b096063d-d7aa-4bb8-bc10-9ddaae46d179 velero.io/restore-name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 velero.io/restore-uid: 5c69649b-a098-4247-9489-91a27b17e1d0 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"5c69649b-a098-4247-9489-91a27b17e1d0"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T22:44:23Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T22:44:37Z" name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5-xf7xs namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5 uid: 5c69649b-a098-4247-9489-91a27b17e1d0 resourceVersion: "50539" uid: a6c7f601-ca35-4f79-89c2-6e8946672f7a spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-7f77d47fd7-894c7 namespace: test-oadp-164 uid: fb9edef1-ea3b-4743-aa30-5b079edba99e repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-164 snapshotID: 1245c0e3 sourceNamespace: test-oadp-164 uploaderType: restic volume: mysql-data status: completionTimestamp: "2024-04-16T22:44:37Z" phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 startTimestamp: "2024-04-16T22:44:35Z" STEP: Verify restore mysql-hooks-e2e-801f956f-fc42-11ee-99f6-0a580a8231b5has completed successfully @ 04/16/24 22:45:46.573 STEP: Verify Application restore @ 04/16/24 22:45:46.587 STEP: Verify Application deployment for case mysql-hooks-e2e @ 04/16/24 22:45:46.587 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2024/04/16 22:45:52 2024-04-16 22:45:48,346 p=30077 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:45:48,346 p=30077 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:45:48,629 p=30077 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:45:48,630 p=30077 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:45:48,905 p=30077 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:45:48,906 p=30077 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:45:48,925 p=30077 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:45:48,925 p=30077 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:49,284 p=30077 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:45:49,284 p=30077 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:49,320 p=30077 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:45:49,320 p=30077 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:49,349 p=30077 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:45:49,350 p=30077 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:49,364 p=30077 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:45:49,972 p=30077 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:45:49,972 p=30077 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:51,299 p=30077 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-04-16 22:45:51,299 p=30077 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:51,696 p=30077 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:45:51,696 p=30077 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:45:52,178 p=30077 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-04-16 22:45:52,178 p=30077 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:45:52,794 p=30077 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-04-16 22:45:52,795 p=30077 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:45:52,807 p=30077 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:45:52,808 p=30077 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 2024/04/16 22:45:53 stderr: ERROR 1049 (42000): Unknown database 'world' 2024/04/16 22:45:53 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 2024/04/16 22:45:53 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-164] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:46:12 2024-04-16 22:45:54,624 p=30377 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:45:54,624 p=30377 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:45:54,878 p=30377 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:45:54,878 p=30377 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:45:55,107 p=30377 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:45:55,108 p=30377 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:45:55,122 p=30377 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:45:55,122 p=30377 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:55,435 p=30377 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:45:55,436 p=30377 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:55,474 p=30377 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:45:55,474 p=30377 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:55,501 p=30377 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:45:55,501 p=30377 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:45:55,514 p=30377 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:45:56,051 p=30377 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:45:56,051 p=30377 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:11,989 p=30377 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-164] *** 2024-04-16 22:46:11,989 p=30377 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:46:12,311 p=30377 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:46:12,311 p=30377 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:46:12 Cleaning setup resources for the backup • [268.472 seconds] ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ Incremental backup restore tests Incremental restore pod count [tc-id:OADP-165][interop] Todolist app with CSI - policy: update /alabama/cspi/e2e/incremental_restore/backup_restore_incremental.go:94 2024/04/16 22:46:12 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 04/16/24 22:46:12.411 2024/04/16 22:46:12 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "69bb9b4d-6a74-48b7-91c3-3dec007e07ea", "resourceVersion": "51281", "generation": 1, "creationTimestamp": "2024-04-16T22:46:12Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:46:12Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:46:12.525 2024/04/16 22:46:12 Waiting for velero pod to be running 2024/04/16 22:46:12 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Installing application for case todolist-backup @ 04/16/24 22:46:12.557 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check namespace todolist-mariadb-csi-policy-update] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Create namespace todolist-mariadb-csi-policy-update] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Ensure namespace todolist-mariadb-csi-policy-update is present] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Deploy todolist-mysql application] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check mysql pod status (30 retries left). FAILED - RETRYING: [localhost]: Check mysql pod status (29 retries left). FAILED - RETRYING: [localhost]: Check mysql pod status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Check todolist pod status (30 retries left). FAILED - RETRYING: [localhost]: Check todolist pod status (29 retries left). FAILED - RETRYING: [localhost]: Check todolist pod status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until service is ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Add additional items todo list] *** changed: [localhost] Pausing for 30 seconds TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait for 30 seconds] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=20  changed=8  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/04/16 22:47:13 2024-04-16 22:46:14,045 p=30584 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:46:14,046 p=30584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:46:14,276 p=30584 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:46:14,277 p=30584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:46:14,528 p=30584 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:46:14,529 p=30584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:46:14,545 p=30584 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:46:14,546 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:14,835 p=30584 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:46:14,835 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:14,862 p=30584 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:46:14,862 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:14,886 p=30584 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:46:14,886 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:14,896 p=30584 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:46:15,433 p=30584 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:46:15,433 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:16,291 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check namespace todolist-mariadb-csi-policy-update] *** 2024-04-16 22:46:16,291 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:16,698 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Create namespace todolist-mariadb-csi-policy-update] *** 2024-04-16 22:46:16,699 p=30584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:46:17,342 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Ensure namespace todolist-mariadb-csi-policy-update is present] *** 2024-04-16 22:46:17,342 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:18,513 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Deploy todolist-mysql application] *** 2024-04-16 22:46:18,514 p=30584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:46:30,010 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod status] *** 2024-04-16 22:46:30,011 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:41,549 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod status] *** 2024-04-16 22:46:41,549 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:46:41,879 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until service is ready for connections] *** 2024-04-16 22:46:41,880 p=30584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:46:42,205 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** 2024-04-16 22:46:42,205 p=30584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:46:42,642 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Add additional items todo list] *** 2024-04-16 22:46:42,643 p=30584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:46:42,668 p=30584 u=1009100000 n=ansible | Pausing for 30 seconds 2024-04-16 22:47:12,674 p=30584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait for 30 seconds] *** 2024-04-16 22:47:12,675 p=30584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:12,932 p=30584 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:47:12,932 p=30584 u=1009100000 n=ansible | localhost : ok=20 changed=8 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:47:13.022 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=17  changed=5  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/04/16 22:47:23 2024-04-16 22:47:14,921 p=31052 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:47:14,921 p=31052 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:47:15,237 p=31052 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:47:15,237 p=31052 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:47:15,581 p=31052 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:47:15,582 p=31052 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:47:15,605 p=31052 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:47:15,605 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:15,976 p=31052 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:47:15,976 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:16,011 p=31052 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:47:16,011 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:16,044 p=31052 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:47:16,044 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:16,061 p=31052 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:47:16,702 p=31052 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:47:16,702 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:18,243 p=31052 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** 2024-04-16 22:47:18,243 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:18,706 p=31052 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** 2024-04-16 22:47:18,707 p=31052 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:47:19,647 p=31052 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** 2024-04-16 22:47:19,647 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:20,070 p=31052 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** 2024-04-16 22:47:20,070 p=31052 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:47:21,915 p=31052 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** 2024-04-16 22:47:21,916 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:22,550 p=31052 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** 2024-04-16 22:47:22,551 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:22,983 p=31052 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** 2024-04-16 22:47:22,984 p=31052 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:47:23,003 p=31052 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:47:23,003 p=31052 u=1009100000 n=ansible | localhost : ok=17 changed=5 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:47:23.1 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/04/16 22:47:23 The 'openshift-storage' namespace does not exist 2024/04/16 22:47:23 Using default CSI driver based on infrastructure: ebs.csi.aws.com 2024/04/16 22:47:23 Snapclass 'example-snapclass' doesn't exist, creating 2024/04/16 22:47:23 Setting new default StorageClass 'gp2-csi' 2024/04/16 22:47:23 Checking default storage class count Run the command: oc get sc 2024/04/16 22:47:23 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 56m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 56m 2024/04/16 22:47:23 {{ } { } [{{ } {mysql todolist-mariadb-csi-policy-update ec222cf7-1696-436e-b82c-60a3ef759f56 51517 0 2024-04-16 22:46:18 +0000 UTC map[app:mysql] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-20-65.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-04-16 22:46:18 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 22:46:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 22:46:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 22:46:22 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{1073741824 0} {} 1Gi BinarySI}]} pvc-ec222cf7-1696-436e-b82c-60a3ef759f56 0xc00075d2f0 0xc00075d300 nil nil} {Bound [ReadWriteOnce] map[storage:{{1073741824 0} {} 1Gi BinarySI}] [] map[] }}]} STEP: Creating backup todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 @ 04/16/24 22:47:23.558 2024/04/16 22:47:23 Wait until backup todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 is completed backup phase: WaitingForPluginOperations backup phase: Completed 2024/04/16 22:48:03 Verify the PVC and VSC count is matching 2024/04/16 22:48:03 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2024/04/16 22:48:03 Run velero describe on the backup 2024/04/16 22:48:03 [./velero describe backup todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 -n openshift-adp --details] 2024/04/16 22:48:04 Exec stderr: "" 2024/04/16 22:48:04 Name: todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: Completed Namespaces: Included: todolist-mariadb-csi-policy-update Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-04-16 22:47:23 +0000 UTC Completed: 2024-04-16 22:47:53 +0000 UTC Expiration: 2024-05-16 22:47:23 +0000 UTC Total items to be backed up: 68 Items backed up: 68 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io todolist-mariadb-csi-policy-update/velero-mysql-blz54: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: todolist-mariadb-csi-policy-update/velero-mysql-blz54/2024-04-16T22:47:33Z Items to Update: volumesnapshots.snapshot.storage.k8s.io todolist-mariadb-csi-policy-update/velero-mysql-blz54 Phase: Completed Created: 2024-04-16 22:47:33 +0000 UTC Started: 2024-04-16 22:47:33 +0000 UTC Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-631b9d99-ecad-4c29-9a93-525e2348ef85: Backup Item Action Plugin: velero.io/csi-volumesnapshotcontent-backupper Operation ID: snapcontent-631b9d99-ecad-4c29-9a93-525e2348ef85/2024-04-16T22:47:33Z Items to Update: volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-631b9d99-ecad-4c29-9a93-525e2348ef85 Phase: Completed Created: 2024-04-16 22:47:33 +0000 UTC Started: 2024-04-16 22:47:33 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist apps/v1/ReplicaSet: - todolist-mariadb-csi-policy-update/mysql-849798ff64 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b authorization.openshift.io/v1/RoleBinding: - todolist-mariadb-csi-policy-update/admin - todolist-mariadb-csi-policy-update/system:deployers - todolist-mariadb-csi-policy-update/system:image-builders - todolist-mariadb-csi-policy-update/system:image-pullers discovery.k8s.io/v1/EndpointSlice: - todolist-mariadb-csi-policy-update/mysql-gj6k9 - todolist-mariadb-csi-policy-update/todolist-8t7x2 operators.coreos.com/v1alpha1/ClusterServiceVersion: - todolist-mariadb-csi-policy-update/volsync-product.v0.9.1 rbac.authorization.k8s.io/v1/RoleBinding: - todolist-mariadb-csi-policy-update/admin - todolist-mariadb-csi-policy-update/system:deployers - todolist-mariadb-csi-policy-update/system:image-builders - todolist-mariadb-csi-policy-update/system:image-pullers route.openshift.io/v1/Route: - todolist-mariadb-csi-policy-update/todolist-route security.openshift.io/v1/SecurityContextConstraints: - todolist-mariadb-csi-policy-update-scc snapshot.storage.k8s.io/v1/VolumeSnapshot: - todolist-mariadb-csi-policy-update/velero-mysql-blz54 snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-631b9d99-ecad-4c29-9a93-525e2348ef85 v1/ConfigMap: - todolist-mariadb-csi-policy-update/kube-root-ca.crt - todolist-mariadb-csi-policy-update/openshift-service-ca.crt v1/Endpoints: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist v1/Event: - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42b677ce942 - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42bcc020aeb - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42c0877c88a - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42c0a237964 - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42c0e54936d - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42c0f485f7f - todolist-mariadb-csi-policy-update/mysql-849798ff64.17c6e42a3c2b8118 - todolist-mariadb-csi-policy-update/mysql.17c6e42a360b0a32 - todolist-mariadb-csi-policy-update/mysql.17c6e42a3b1ca6d8 - todolist-mariadb-csi-policy-update/mysql.17c6e42a3d6cb937 - todolist-mariadb-csi-policy-update/mysql.17c6e42a3d7be7e8 - todolist-mariadb-csi-policy-update/mysql.17c6e42b40a7bdb7 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42a41d5ce70 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42a66291423 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42a677af08b - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42bcfe383a0 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42bd5bac24b - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42bd69f2a32 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42e62212a10 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42ea84e4a82 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42eaea001d1 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42eaf943e39 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b.17c6e42a415a0c8b - todolist-mariadb-csi-policy-update/todolist.17c6e42a40a44df7 v1/Namespace: - todolist-mariadb-csi-policy-update v1/PersistentVolume: - pvc-ec222cf7-1696-436e-b82c-60a3ef759f56 v1/PersistentVolumeClaim: - todolist-mariadb-csi-policy-update/mysql v1/Pod: - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb v1/Secret: - todolist-mariadb-csi-policy-update/builder-dockercfg-2222k - todolist-mariadb-csi-policy-update/builder-token-nbgnn - todolist-mariadb-csi-policy-update/default-dockercfg-d5ztt - todolist-mariadb-csi-policy-update/default-token-nn6kx - todolist-mariadb-csi-policy-update/deployer-dockercfg-lql22 - todolist-mariadb-csi-policy-update/deployer-token-7hz2d - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa-dockercfg-l42p5 - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa-token-th8cq v1/Service: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist v1/ServiceAccount: - todolist-mariadb-csi-policy-update/builder - todolist-mariadb-csi-policy-update/default - todolist-mariadb-csi-policy-update/deployer - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa Velero-Native Snapshots: STEP: Verify backup todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:48:04.166 2024/04/16 22:48:04 Backup for case todolist-backup succeeded STEP: Scale application @ 04/16/24 22:48:04.24 2024/04/16 22:48:04 Scaling deployment 'todolist' to 2 replicas 2024/04/16 22:48:04 Deployment updated successfully 2024/04/16 22:48:04 number of running pods: 1 2024/04/16 22:48:09 number of running pods: 1 2024/04/16 22:48:14 number of running pods: 1 2024/04/16 22:48:19 number of running pods: 1 2024/04/16 22:48:24 Application reached target number of replicas: 2 STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:48:24.391 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/04/16 22:48:24 The 'openshift-storage' namespace does not exist 2024/04/16 22:48:24 Using default CSI driver based on infrastructure: ebs.csi.aws.com 2024/04/16 22:48:24 Snapclass 'example-snapclass' already exists, skip creating 2024/04/16 22:48:24 Setting new default StorageClass 'gp2-csi' 2024/04/16 22:48:24 Checking default storage class count Skipping creation of StorageClass The current StorageClass: gp2-csi matches the new StorageClass: gp2-csi 2024/04/16 22:48:24 {{ } { } [{{ } {mysql todolist-mariadb-csi-policy-update ec222cf7-1696-436e-b82c-60a3ef759f56 52348 0 2024-04-16 22:46:18 +0000 UTC map[app:mysql] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-20-65.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-04-16 22:46:18 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 22:46:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 22:46:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 22:46:22 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{1073741824 0} {} 1Gi BinarySI}]} pvc-ec222cf7-1696-436e-b82c-60a3ef759f56 0xc001a0faf0 0xc001a0fb00 nil nil} {Bound [ReadWriteOnce] map[storage:{{1073741824 0} {} 1Gi BinarySI}] [] map[] }}]} STEP: Creating backup todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 @ 04/16/24 22:48:24.727 2024/04/16 22:48:24 Wait until backup todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 is completed backup phase: InProgress backup phase: WaitingForPluginOperations backup phase: WaitingForPluginOperations backup phase: Completed 2024/04/16 22:49:24 Verify the PVC and VSC count is matching 2024/04/16 22:49:24 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2024/04/16 22:49:24 Run velero describe on the backup 2024/04/16 22:49:24 [./velero describe backup todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 -n openshift-adp --details] 2024/04/16 22:49:25 Exec stderr: "" 2024/04/16 22:49:25 Name: todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: Completed Namespaces: Included: todolist-mariadb-csi-policy-update Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-04-16 22:48:24 +0000 UTC Completed: 2024-04-16 22:49:13 +0000 UTC Expiration: 2024-05-16 22:48:24 +0000 UTC Total items to be backed up: 84 Items backed up: 84 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io todolist-mariadb-csi-policy-update/velero-mysql-w5nkj: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: todolist-mariadb-csi-policy-update/velero-mysql-w5nkj/2024-04-16T22:48:34Z Items to Update: volumesnapshots.snapshot.storage.k8s.io todolist-mariadb-csi-policy-update/velero-mysql-w5nkj Phase: Completed Created: 2024-04-16 22:48:34 +0000 UTC Started: 2024-04-16 22:48:34 +0000 UTC Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-96929ae5-74ec-432f-907e-c4334baf9de1: Backup Item Action Plugin: velero.io/csi-volumesnapshotcontent-backupper Operation ID: snapcontent-96929ae5-74ec-432f-907e-c4334baf9de1/2024-04-16T22:48:34Z Items to Update: volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-96929ae5-74ec-432f-907e-c4334baf9de1 Phase: Completed Created: 2024-04-16 22:48:34 +0000 UTC Started: 2024-04-16 22:48:34 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist apps/v1/ReplicaSet: - todolist-mariadb-csi-policy-update/mysql-849798ff64 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b authorization.openshift.io/v1/RoleBinding: - todolist-mariadb-csi-policy-update/admin - todolist-mariadb-csi-policy-update/system:deployers - todolist-mariadb-csi-policy-update/system:image-builders - todolist-mariadb-csi-policy-update/system:image-pullers discovery.k8s.io/v1/EndpointSlice: - todolist-mariadb-csi-policy-update/mysql-gj6k9 - todolist-mariadb-csi-policy-update/todolist-8t7x2 operators.coreos.com/v1alpha1/ClusterServiceVersion: - todolist-mariadb-csi-policy-update/volsync-product.v0.9.1 rbac.authorization.k8s.io/v1/RoleBinding: - todolist-mariadb-csi-policy-update/admin - todolist-mariadb-csi-policy-update/system:deployers - todolist-mariadb-csi-policy-update/system:image-builders - todolist-mariadb-csi-policy-update/system:image-pullers route.openshift.io/v1/Route: - todolist-mariadb-csi-policy-update/todolist-route security.openshift.io/v1/SecurityContextConstraints: - todolist-mariadb-csi-policy-update-scc snapshot.storage.k8s.io/v1/VolumeSnapshot: - todolist-mariadb-csi-policy-update/velero-mysql-w5nkj snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-96929ae5-74ec-432f-907e-c4334baf9de1 v1/ConfigMap: - todolist-mariadb-csi-policy-update/kube-root-ca.crt - todolist-mariadb-csi-policy-update/openshift-service-ca.crt v1/Endpoints: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist v1/Event: - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42b677ce942 - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42bcc020aeb - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42c0877c88a - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42c0a237964 - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42c0e54936d - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz.17c6e42c0f485f7f - todolist-mariadb-csi-policy-update/mysql-849798ff64.17c6e42a3c2b8118 - todolist-mariadb-csi-policy-update/mysql.17c6e42a360b0a32 - todolist-mariadb-csi-policy-update/mysql.17c6e42a3b1ca6d8 - todolist-mariadb-csi-policy-update/mysql.17c6e42a3d6cb937 - todolist-mariadb-csi-policy-update/mysql.17c6e42a3d7be7e8 - todolist-mariadb-csi-policy-update/mysql.17c6e42b40a7bdb7 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42a41d5ce70 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42a66291423 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42a677af08b - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42bcfe383a0 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42bd5bac24b - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42bd69f2a32 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42e62212a10 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42ea84e4a82 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42eaea001d1 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb.17c6e42eaf943e39 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e442eab040bf - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e4430ec2da04 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e4430ff61902 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e4444e6c2f81 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e44453fd7607 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e44454d78608 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e446c1b44a2d - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e44704a260ae - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e4470a8ce925 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx.17c6e4470b764cc0 - todolist-mariadb-csi-policy-update/todolist-fdc764b8b.17c6e42a415a0c8b - todolist-mariadb-csi-policy-update/todolist-fdc764b8b.17c6e442ea1dc1bb - todolist-mariadb-csi-policy-update/todolist.17c6e42a40a44df7 - todolist-mariadb-csi-policy-update/todolist.17c6e442e84a4c80 - todolist-mariadb-csi-policy-update/velero-mysql-blz54.17c6e43a7971ae57 - todolist-mariadb-csi-policy-update/velero-mysql-blz54.17c6e43a90f44760 - todolist-mariadb-csi-policy-update/velero-mysql-blz54.17c6e43f976c1640 v1/Namespace: - todolist-mariadb-csi-policy-update v1/PersistentVolume: - pvc-ec222cf7-1696-436e-b82c-60a3ef759f56 v1/PersistentVolumeClaim: - todolist-mariadb-csi-policy-update/mysql v1/Pod: - todolist-mariadb-csi-policy-update/mysql-849798ff64-gztlz - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-m2xmb - todolist-mariadb-csi-policy-update/todolist-fdc764b8b-rtnlx v1/Secret: - todolist-mariadb-csi-policy-update/builder-dockercfg-2222k - todolist-mariadb-csi-policy-update/builder-token-nbgnn - todolist-mariadb-csi-policy-update/default-dockercfg-d5ztt - todolist-mariadb-csi-policy-update/default-token-nn6kx - todolist-mariadb-csi-policy-update/deployer-dockercfg-lql22 - todolist-mariadb-csi-policy-update/deployer-token-7hz2d - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa-dockercfg-l42p5 - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa-token-th8cq v1/Service: - todolist-mariadb-csi-policy-update/mysql - todolist-mariadb-csi-policy-update/todolist v1/ServiceAccount: - todolist-mariadb-csi-policy-update/builder - todolist-mariadb-csi-policy-update/default - todolist-mariadb-csi-policy-update/deployer - todolist-mariadb-csi-policy-update/todolist-mariadb-csi-policy-update-sa Velero-Native Snapshots: STEP: Verify backup todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:49:25.328 2024/04/16 22:49:25 Backup for case todolist-backup succeeded STEP: Cleanup application and restore 1st backup @ 04/16/24 22:49:25.437 STEP: Delete the appplication resources todolist-backup @ 04/16/24 22:49:25.437 STEP: Cleanup Application for case todolist-backup @ 04/16/24 22:49:25.437 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove namespace todolist-mariadb-csi-policy-update] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove todolist-mariadb-csi-policy-update SCC] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=12  changed=5  unreachable=0 failed=0 skipped=17  rescued=0 ignored=0 2024/04/16 22:49:47 2024-04-16 22:49:27,000 p=31399 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:49:27,001 p=31399 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:49:27,273 p=31399 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:49:27,273 p=31399 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:49:27,521 p=31399 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:49:27,521 p=31399 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:49:27,539 p=31399 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:49:27,540 p=31399 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:49:27,866 p=31399 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:49:27,867 p=31399 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:49:27,904 p=31399 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:49:27,904 p=31399 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:49:27,940 p=31399 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:49:27,940 p=31399 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:49:27,957 p=31399 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:49:28,671 p=31399 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:49:28,671 p=31399 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:49:44,949 p=31399 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove namespace todolist-mariadb-csi-policy-update] *** 2024-04-16 22:49:44,949 p=31399 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:49:46,591 p=31399 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove todolist-mariadb-csi-policy-update SCC] *** 2024-04-16 22:49:46,591 p=31399 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:49:46,993 p=31399 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:49:46,993 p=31399 u=1009100000 n=ansible | localhost : ok=12 changed=5 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0 2024/04/16 22:49:47 Creating restore todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 for case todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 STEP: Create restore todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 from backup todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 @ 04/16/24 22:49:47.043 2024/04/16 22:49:47 Wait until restore todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5 is complete restore phase: InProgress restore phase: InProgress restore phase: Completed STEP: Verify restore todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5has completed successfully @ 04/16/24 22:50:07.114 STEP: Verify Application restore @ 04/16/24 22:50:07.128 STEP: Verify Application deployment for case todolist-backup @ 04/16/24 22:50:07.128 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] FAILED - RETRYING: [localhost]: Check mysql pod is running (30 retries left). FAILED - RETRYING: [localhost]: Check mysql pod is running (29 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check todolist pod is running (30 retries left). FAILED - RETRYING: [localhost]: Check todolist pod is running (29 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=17  changed=5  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/04/16 22:50:37 2024-04-16 22:50:08,684 p=31620 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:50:08,685 p=31620 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:50:08,917 p=31620 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:50:08,917 p=31620 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:50:09,183 p=31620 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:50:09,183 p=31620 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:50:09,199 p=31620 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:50:09,199 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:09,526 p=31620 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:50:09,526 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:09,555 p=31620 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:50:09,556 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:09,582 p=31620 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:50:09,582 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:09,594 p=31620 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:50:10,142 p=31620 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:50:10,142 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:22,585 p=31620 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** 2024-04-16 22:50:22,585 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:22,946 p=31620 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** 2024-04-16 22:50:22,946 p=31620 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:50:34,895 p=31620 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** 2024-04-16 22:50:34,895 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:35,232 p=31620 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** 2024-04-16 22:50:35,232 p=31620 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:50:36,844 p=31620 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** 2024-04-16 22:50:36,844 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:37,282 p=31620 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** 2024-04-16 22:50:37,283 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:37,609 p=31620 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** 2024-04-16 22:50:37,609 p=31620 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:37,626 p=31620 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:50:37,627 p=31620 u=1009100000 n=ansible | localhost : ok=17 changed=5 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/04/16 22:50:37 Application reached target number of replicas: 1 STEP: Restore 2nd backup with existingRessourcePolicy: update @ 04/16/24 22:50:37.706 2024/04/16 22:50:37 Creating restore todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 for case todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 STEP: Create restore todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 from backup todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 @ 04/16/24 22:50:37.706 2024/04/16 22:50:37 Wait until restore todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5 is complete restore phase: InProgress restore phase: InProgress restore phase: Completed STEP: Verify restore todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5has completed successfully @ 04/16/24 22:50:57.783 STEP: Verify Application restore @ 04/16/24 22:50:57.798 STEP: Verify Application deployment for case todolist-backup @ 04/16/24 22:50:57.798 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=17  changed=5  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/04/16 22:51:05 2024-04-16 22:50:59,219 p=32002 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:50:59,219 p=32002 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:50:59,456 p=32002 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:50:59,456 p=32002 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:50:59,678 p=32002 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:50:59,678 p=32002 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:50:59,693 p=32002 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:50:59,693 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:59,968 p=32002 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:50:59,969 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:50:59,997 p=32002 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:50:59,997 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:00,022 p=32002 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:51:00,023 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:00,034 p=32002 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:51:00,559 p=32002 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:51:00,559 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:01,634 p=32002 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check mysql pod is running] *** 2024-04-16 22:51:01,634 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:01,964 p=32002 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until mysql service ready for connections] *** 2024-04-16 22:51:01,964 p=32002 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:02,609 p=32002 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Check todolist pod is running] *** 2024-04-16 22:51:02,609 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:02,933 p=32002 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Wait until todolist API server starts] *** 2024-04-16 22:51:02,934 p=32002 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:04,457 p=32002 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Obtain todolist route] *** 2024-04-16 22:51:04,457 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:04,938 p=32002 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find 1st database item] *** 2024-04-16 22:51:04,939 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:05,260 p=32002 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Find the string in incomplete items] *** 2024-04-16 22:51:05,260 p=32002 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:05,272 p=32002 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:51:05,272 p=32002 u=1009100000 n=ansible | localhost : ok=17 changed=5 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/04/16 22:51:05 number of running pods: 1 2024/04/16 22:51:10 Application reached target number of replicas: 2 2024/04/16 22:51:10 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 2024/04/16 22:51:10 Cleaning setup resources for the backup 2024/04/16 22:51:10 Setting new default StorageClass 'gp3-csi' 2024/04/16 22:51:10 Checking default storage class count Run the command: oc get sc 2024/04/16 22:51:10 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 60m gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 60m 2024/04/16 22:51:10 Deleting VolumeSnapshotClass 'example-snapclass' 2024/04/16 22:51:10 Cleaning setup resources for the backup 2024/04/16 22:51:10 Setting new default StorageClass 'gp3-csi' 2024/04/16 22:51:10 Checking default storage class count Skipping creation of StorageClass The current StorageClass: gp3-csi matches the new StorageClass: gp3-csi 2024/04/16 22:51:10 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove namespace todolist-mariadb-csi-policy-update] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove todolist-mariadb-csi-policy-update SCC] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=12  changed=5  unreachable=0 failed=0 skipped=17  rescued=0 ignored=0 2024/04/16 22:51:32 2024-04-16 22:51:12,233 p=32350 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:51:12,233 p=32350 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:12,513 p=32350 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:51:12,513 p=32350 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:12,811 p=32350 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:51:12,812 p=32350 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:12,830 p=32350 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:51:12,831 p=32350 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:13,188 p=32350 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:51:13,189 p=32350 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:13,230 p=32350 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:51:13,230 p=32350 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:13,268 p=32350 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:51:13,268 p=32350 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:13,280 p=32350 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:51:13,927 p=32350 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:51:13,928 p=32350 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:30,083 p=32350 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove namespace todolist-mariadb-csi-policy-update] *** 2024-04-16 22:51:30,084 p=32350 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:31,864 p=32350 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-todolist-mariadb : Remove todolist-mariadb-csi-policy-update SCC] *** 2024-04-16 22:51:31,865 p=32350 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:32,431 p=32350 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:51:32,431 p=32350 u=1009100000 n=ansible | localhost : ok=12 changed=5 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0 • [320.145 seconds] ------------------------------ SSSSSSSSSSSSSSSSS ------------------------------ Backup hooks tests Pre exec hook [tc-id:OADP-92][interop] Cassandra app with Restic /alabama/cspi/e2e/hooks/backup_hooks.go:109 2024/04/16 22:51:32 Delete all downloadrequest todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5-838e0aa2-8df6-406b-9e54-68365837cc61 todolist-backup-4a159fcd-fc43-11ee-99f6-0a580a8231b5-c35ab49c-73bc-4d12-9280-39983c3e68dd todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5-946f0b7e-dd57-45dc-88cc-0b55bc3d1064 todolist-backup-6e9de923-fc43-11ee-99f6-0a580a8231b5-95675f0c-c808-43c4-9d7d-6e5988f632a8 STEP: Create DPA CR @ 04/16/24 22:51:32.66 2024/04/16 22:51:32 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "946bfc0a-c49a-4690-aa1a-e566487dc395", "resourceVersion": "54723", "generation": 1, "creationTimestamp": "2024-04-16T22:51:32Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:51:32Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:51:32.756 2024/04/16 22:51:32 Waiting for velero pod to be running 2024/04/16 22:51:32 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:51:32.821 2024/04/16 22:51:32 Checking for correct number of running NodeAgent pods... 2024/04/16 22:51:32 pod: node-agent-2pccb is not yet running with status: {Pending [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:51:32 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:51:32 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:51:32 +0000 UTC ContainersNotReady containers with unready status: [node-agent]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:51:32 +0000 UTC ContainersNotReady containers with unready status: [node-agent]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:51:32 +0000 UTC }] 10.0.106.243 [] 2024-04-16 22:51:32 +0000 UTC [] [{node-agent {&ContainerStateWaiting{Reason:ContainerCreating,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc0006e91da}] Burstable []} STEP: Installing application for case cassandra-hooks-e2e @ 04/16/24 22:51:37.924 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pods status (30 retries left). FAILED - RETRYING: [localhost]: Check pods status (29 retries left). FAILED - RETRYING: [localhost]: Check pods status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (30 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (29 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (28 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (27 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (26 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (25 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (24 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (23 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (22 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (21 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (20 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (19 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (18 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (17 retries left). FAILED - RETRYING: [localhost]: Wait until all cassandra node are ready (Status=Up and State=Normal) (16 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=18  changed=9  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/04/16 22:54:35 2024-04-16 22:51:39,842 p=32561 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:51:39,843 p=32561 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:40,145 p=32561 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:51:40,145 p=32561 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:40,464 p=32561 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:51:40,464 p=32561 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:40,486 p=32561 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:51:40,487 p=32561 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:40,870 p=32561 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:51:40,870 p=32561 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:40,911 p=32561 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:51:40,911 p=32561 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:40,946 p=32561 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:51:40,947 p=32561 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:40,964 p=32561 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:51:41,611 p=32561 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:51:41,611 p=32561 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:42,683 p=32561 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check namespace] *** 2024-04-16 22:51:42,683 p=32561 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:51:43,183 p=32561 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create namespace] *** 2024-04-16 22:51:43,183 p=32561 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:43,591 p=32561 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Add scc privileged to service account] *** 2024-04-16 22:51:43,591 p=32561 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:44,685 p=32561 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a service object required to provide network identity] *** 2024-04-16 22:51:44,685 p=32561 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:51:45,552 p=32561 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Create a statefulset with the existing yaml] *** 2024-04-16 22:51:45,553 p=32561 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:52:03,578 p=32561 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-04-16 22:52:03,578 p=32561 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:54:31,330 p=32561 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-04-16 22:54:31,331 p=32561 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:54:35,327 p=32561 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Populate the DB with some sample data] *** 2024-04-16 22:54:35,327 p=32561 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:54:35,490 p=32561 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:54:35,490 p=32561 u=1009100000 n=ansible | localhost : ok=18 changed=9 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:54:35.566 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=15  changed=6  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/04/16 22:54:48 2024-04-16 22:54:37,365 p=33395 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:54:37,365 p=33395 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:54:37,670 p=33395 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:54:37,670 p=33395 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:54:37,972 p=33395 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:54:37,973 p=33395 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:54:37,995 p=33395 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:54:37,995 p=33395 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:54:38,376 p=33395 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:54:38,377 p=33395 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:54:38,416 p=33395 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:54:38,417 p=33395 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:54:38,453 p=33395 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:54:38,453 p=33395 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:54:38,470 p=33395 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:54:40,114 p=33395 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:54:40,114 p=33395 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:54:41,584 p=33395 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Check pods status] *** 2024-04-16 22:54:41,585 p=33395 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:54:46,138 p=33395 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Wait until all cassandra node are ready (Status=Up and State=Normal)] *** 2024-04-16 22:54:46,138 p=33395 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:54:47,323 p=33395 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data was populated] *** 2024-04-16 22:54:47,323 p=33395 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:54:47,991 p=33395 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : copy] *** 2024-04-16 22:54:47,992 p=33395 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:54:48,216 p=33395 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Verify Data Integrity] *** 2024-04-16 22:54:48,216 p=33395 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:54:48,232 p=33395 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:54:48,232 p=33395 u=1009100000 n=ansible | localhost : ok=15 changed=6 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Creating backup cassandra-hooks-e2e @ 04/16/24 22:54:48.309 2024/04/16 22:54:48 Wait until backup cassandra-hooks-e2e is completed backup phase: InProgress backup phase: InProgress backup phase: InProgress backup phase: Completed STEP: Verify backup cassandra-hooks-e2e has completed successfully @ 04/16/24 22:56:08.469 2024/04/16 22:56:08 Backup for case cassandra-hooks-e2e succeeded STEP: Verify pre-backup hooks were executed; verify that the cassandra app is quiesced and it's not possible to connect to the DB server @ 04/16/24 22:56:08.588 2024/04/16 22:56:08 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 2024/04/16 22:56:08 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-92] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:56:27 2024-04-16 22:56:10,307 p=33672 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:56:10,307 p=33672 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:56:10,543 p=33672 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:56:10,543 p=33672 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:56:10,766 p=33672 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:56:10,766 p=33672 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:56:10,781 p=33672 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:56:10,782 p=33672 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:11,053 p=33672 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:56:11,053 p=33672 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:11,079 p=33672 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:56:11,079 p=33672 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:11,101 p=33672 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:56:11,102 p=33672 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:11,112 p=33672 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:56:11,630 p=33672 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:56:11,630 p=33672 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:27,496 p=33672 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-cassandra : Remove namespace test-oadp-92] *** 2024-04-16 22:56:27,496 p=33672 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:56:27,762 p=33672 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:56:27,763 p=33672 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:56:27 Cleaning setup resources for the backup • [295.330 seconds] ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ Backup restore tests Application backup [tc-id:OADP-371] [bug-id:OADP-1077] [interop] [smoke] MySQL application with Restic [mr-check] /alabama/cspi/e2e/app_backup/backup_restore.go:48 2024/04/16 22:56:28 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 04/16/24 22:56:28.229 2024/04/16 22:56:28 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "c9d3a1de-6fde-4fb3-91f1-d975042d7b36", "resourceVersion": "56759", "generation": 1, "creationTimestamp": "2024-04-16T22:56:28Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T22:56:28Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 22:56:28.325 2024/04/16 22:56:28 Waiting for velero pod to be running 2024/04/16 22:56:28 pod: velero-664bfc68b4-xt9z9 is not yet running with status: {Succeeded [{PodReadyToStartContainers False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:56:28 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:51:37 +0000 UTC PodCompleted } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:56:28 +0000 UTC PodCompleted } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:56:28 +0000 UTC PodCompleted } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:51:32 +0000 UTC }] 10.0.20.65 [] 2024-04-16 22:51:32 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:51:33 +0000 UTC,FinishedAt:2024-04-16 22:51:34 +0000 UTC,ContainerID:cri-o://1043c9dfd7b48aad1e51084ae3f76492fd7eb2ed64ae97f669fe25710f5c4506,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://1043c9dfd7b48aad1e51084ae3f76492fd7eb2ed64ae97f669fe25710f5c4506 0xc000dcde59} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:51:35 +0000 UTC,FinishedAt:2024-04-16 22:51:35 +0000 UTC,ContainerID:cri-o://56be64d7abac38c4ce243f8b39c8c78fbebe6b56c3f5a22de097f5b4a3ab97f5,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:716cec49a663d87eff7eaf9ddc948d2861645d601a2e15605cb660c9f98ab489 cri-o://56be64d7abac38c4ce243f8b39c8c78fbebe6b56c3f5a22de097f5b4a3ab97f5 0xc000dcde69} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:51:36 +0000 UTC,FinishedAt:2024-04-16 22:51:36 +0000 UTC,ContainerID:cri-o://be3eeb0a65e0be76c7d5076f53b024bbba28b58fa495fc0ba7da8a90201eae61,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f cri-o://be3eeb0a65e0be76c7d5076f53b024bbba28b58fa495fc0ba7da8a90201eae61 0xc000dcde79}] [{velero {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:51:37 +0000 UTC,FinishedAt:2024-04-16 22:56:27 +0000 UTC,ContainerID:cri-o://723ab481b7b1ff76a13992b02f891f53ae138021aab482a08eebcb0cfaa2e028,}} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:02f194caab292aea9bcbb22556fca8e2d7d893a629e6e56097501501fbff7ba7 cri-o://723ab481b7b1ff76a13992b02f891f53ae138021aab482a08eebcb0cfaa2e028 0xc000dcde89}] Burstable []} 2024/04/16 22:56:33 pod: velero-664bfc68b4-b9kn9 is not yet running with status: {Pending [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:56:30 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:56:32 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:56:28 +0000 UTC ContainersNotReady containers with unready status: [velero]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:56:28 +0000 UTC ContainersNotReady containers with unready status: [velero]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-16 22:56:28 +0000 UTC }] 10.0.20.65 10.131.0.44 [{10.131.0.44}] 2024-04-16 22:56:28 +0000 UTC [{openshift-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:56:29 +0000 UTC,FinishedAt:2024-04-16 22:56:29 +0000 UTC,ContainerID:cri-o://c549147d5a0336c741a2e5c7870ebc44428ffff265ca9233cc55680ffa24fa3f,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 registry.redhat.io/oadp/oadp-velero-plugin-rhel9@sha256:36f396227846d8b849f364beddf65524c9817f7c9f7e9713db8d4b4e2729cde2 cri-o://c549147d5a0336c741a2e5c7870ebc44428ffff265ca9233cc55680ffa24fa3f 0xc0004b5679} {velero-plugin-for-aws {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:56:31 +0000 UTC,FinishedAt:2024-04-16 22:56:31 +0000 UTC,ContainerID:cri-o://f4a5d767b126e51c0d1bc130e7aa1091f07ab6eab0bcd29639a98f554ebaa245,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:8fa90687d6c3a6147858f6df87a63c3edd77531fff12091eb04df86a4f6811d4 registry.redhat.io/oadp/oadp-velero-plugin-for-aws-rhel9@sha256:716cec49a663d87eff7eaf9ddc948d2861645d601a2e15605cb660c9f98ab489 cri-o://f4a5d767b126e51c0d1bc130e7aa1091f07ab6eab0bcd29639a98f554ebaa245 0xc0004b5689} {kubevirt-velero-plugin {nil nil &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2024-04-16 22:56:32 +0000 UTC,FinishedAt:2024-04-16 22:56:32 +0000 UTC,ContainerID:cri-o://f6cd0682f8e172469d97390d97f60fabc34cf2e4f0a20406e60bb2fa766fc1f3,}} {nil nil nil} true 0 registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f registry.redhat.io/oadp/oadp-kubevirt-velero-plugin-rhel9@sha256:476fbe152797d15731a77e710b3c666317fbe9aeb998966309a021e2f1b5402f cri-o://f6cd0682f8e172469d97390d97f60fabc34cf2e4f0a20406e60bb2fa766fc1f3 0xc0004b5699}] [{velero {&ContainerStateWaiting{Reason:PodInitializing,Message:,} nil nil} {nil nil nil} false 0 registry.redhat.io/oadp/oadp-velero-rhel9@sha256:40810b794f6e67983466a687d1f320301e85dfb476671742ec89874dfa4e9117 0xc0004b56af}] Burstable []} 2024/04/16 22:56:38 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 22:56:38.427 2024/04/16 22:56:38 Checking for correct number of running NodeAgent pods... STEP: Installing application for case mysql @ 04/16/24 22:56:38.46 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-1077] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=19  changed=10  unreachable=0 failed=0 skipped=5  rescued=0 ignored=0 2024/04/16 22:57:05 2024-04-16 22:56:40,308 p=33880 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:56:40,309 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:56:40,618 p=33880 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:56:40,618 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:56:40,933 p=33880 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:56:40,934 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:56:40,953 p=33880 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:56:40,954 p=33880 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:41,313 p=33880 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:56:41,313 p=33880 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:41,355 p=33880 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:56:41,355 p=33880 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:41,386 p=33880 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:56:41,387 p=33880 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:41,399 p=33880 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:56:42,040 p=33880 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:56:42,041 p=33880 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:43,145 p=33880 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-1077] *** 2024-04-16 22:56:43,145 p=33880 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:43,629 p=33880 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-04-16 22:56:43,629 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:56:44,837 p=33880 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-04-16 22:56:44,837 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:56:56,906 p=33880 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-04-16 22:56:56,906 p=33880 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:56:57,485 p=33880 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-04-16 22:56:57,485 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:03,198 p=33880 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:57:03,198 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:03,941 p=33880 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-04-16 22:57:03,942 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:04,896 p=33880 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-04-16 22:57:04,896 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:05,565 p=33880 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-04-16 22:57:05,565 p=33880 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:05,688 p=33880 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:57:05,688 p=33880 u=1009100000 n=ansible | localhost : ok=19 changed=10 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 22:57:05.751 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2024/04/16 22:57:11 2024-04-16 22:57:07,236 p=34330 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:57:07,237 p=34330 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:07,464 p=34330 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:57:07,465 p=34330 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:07,692 p=34330 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:57:07,692 p=34330 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:07,707 p=34330 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:57:07,707 p=34330 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:57:07,989 p=34330 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:57:07,989 p=34330 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:57:08,015 p=34330 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:57:08,015 p=34330 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:57:08,037 p=34330 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:57:08,037 p=34330 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:57:08,048 p=34330 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:57:08,562 p=34330 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:57:08,562 p=34330 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:57:09,596 p=34330 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-04-16 22:57:09,597 p=34330 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:57:09,920 p=34330 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 22:57:09,921 p=34330 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:10,361 p=34330 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-04-16 22:57:10,361 p=34330 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:10,951 p=34330 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-04-16 22:57:10,951 p=34330 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:57:10,964 p=34330 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:57:10,964 p=34330 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 2024/04/16 22:57:11 {{ } { } [{{ } {mysql test-oadp-1077 d3dfb09c-b64f-4608-af77-e3def90531f4 57087 0 2024-04-16 22:56:44 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-89-31.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-04-16 22:56:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 22:56:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 22:56:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 22:56:49 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-d3dfb09c-b64f-4608-af77-e3def90531f4 0xc00172d9e0 0xc00172d9f0 nil nil} {Bound [ReadWriteOnce] map[storage:{{2147483648 0} {} 2Gi BinarySI}] [] map[] }} {{ } {mysql-1 test-oadp-1077 0f7bfc7b-e798-47ea-8a92-ccb6cd75949c 57084 0 2024-04-16 22:56:44 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-89-31.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-04-16 22:56:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 22:56:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 22:56:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 22:56:49 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-0f7bfc7b-e798-47ea-8a92-ccb6cd75949c 0xc00172db10 0xc00172db20 nil nil} {Bound [ReadWriteOnce] map[storage:{{2147483648 0} {} 2Gi BinarySI}] [] map[] }}]} STEP: Creating backup mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 @ 04/16/24 22:57:11.035 2024/04/16 22:57:11 Wait until backup mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 is completed backup phase: InProgress backup phase: InProgress backup phase: InProgress backup phase: Completed 2024/04/16 22:58:31 Verify the PodVolumeBackup is completed successfully and BackupRepository type is matching with DPA.nodeAgent.uploaderType 2024/04/16 22:58:31 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql-1 creationTimestamp: "2024-04-16T22:57:18Z" generateName: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5- generation: 4 labels: velero.io/backup-name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 velero.io/backup-uid: 6bdd20cf-d976-469b-9eea-a712c7c0a38c velero.io/pvc-uid: 0f7bfc7b-e798-47ea-8a92-ccb6cd75949c managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"6bdd20cf-d976-469b-9eea-a712c7c0a38c"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T22:57:18Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T22:58:22Z" name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5-dzpkx namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 uid: 6bdd20cf-d976-469b-9eea-a712c7c0a38c resourceVersion: "57592" uid: 32229c85-fdfc-4495-9a73-6cd6a09fb22b spec: backupStorageLocation: ts-dpa-1 node: ip-10-0-89-31.us-east-2.compute.internal pod: kind: Pod name: mysql-7f77d47fd7-q7q45 namespace: test-oadp-1077 uid: bba39d06-cc97-40b6-837d-8a18b24c060b repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-1077 tags: backup: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 backup-uid: 6bdd20cf-d976-469b-9eea-a712c7c0a38c ns: test-oadp-1077 pod: mysql-7f77d47fd7-q7q45 pod-uid: bba39d06-cc97-40b6-837d-8a18b24c060b pvc-uid: 0f7bfc7b-e798-47ea-8a92-ccb6cd75949c volume: mysql-data1 uploaderType: restic volume: mysql-data1 status: completionTimestamp: "2024-04-16T22:58:22Z" path: /host_pods/bba39d06-cc97-40b6-837d-8a18b24c060b/volumes/kubernetes.io~csi/pvc-0f7bfc7b-e798-47ea-8a92-ccb6cd75949c/mount phase: Completed progress: bytesDone: 104857640 totalBytes: 104857640 snapshotID: 7cfe72e8 startTimestamp: "2024-04-16T22:58:19Z" 2024/04/16 22:58:31 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql creationTimestamp: "2024-04-16T22:57:18Z" generateName: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5- generation: 4 labels: velero.io/backup-name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 velero.io/backup-uid: 6bdd20cf-d976-469b-9eea-a712c7c0a38c velero.io/pvc-uid: d3dfb09c-b64f-4608-af77-e3def90531f4 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"6bdd20cf-d976-469b-9eea-a712c7c0a38c"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T22:57:18Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T22:57:21Z" name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5-nsgjp namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 uid: 6bdd20cf-d976-469b-9eea-a712c7c0a38c resourceVersion: "57269" uid: 9c401370-cde2-4e45-a6b0-e25943e8c83d spec: backupStorageLocation: ts-dpa-1 node: ip-10-0-89-31.us-east-2.compute.internal pod: kind: Pod name: mysql-7f77d47fd7-q7q45 namespace: test-oadp-1077 uid: bba39d06-cc97-40b6-837d-8a18b24c060b repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-1077 tags: backup: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 backup-uid: 6bdd20cf-d976-469b-9eea-a712c7c0a38c ns: test-oadp-1077 pod: mysql-7f77d47fd7-q7q45 pod-uid: bba39d06-cc97-40b6-837d-8a18b24c060b pvc-uid: d3dfb09c-b64f-4608-af77-e3def90531f4 volume: mysql-data uploaderType: restic volume: mysql-data status: completionTimestamp: "2024-04-16T22:57:21Z" path: /host_pods/bba39d06-cc97-40b6-837d-8a18b24c060b/volumes/kubernetes.io~csi/pvc-d3dfb09c-b64f-4608-af77-e3def90531f4/mount phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 snapshotID: 5e3be896 startTimestamp: "2024-04-16T22:57:18Z" STEP: Verify backup mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 22:58:31.166 2024/04/16 22:58:31 Backup for case mysql succeeded STEP: Delete the appplication resources mysql @ 04/16/24 22:58:31.242 STEP: Cleanup Application for case mysql @ 04/16/24 22:58:31.242 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-1077] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 22:58:50 2024-04-16 22:58:32,646 p=34636 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 22:58:32,646 p=34636 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:58:32,875 p=34636 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 22:58:32,875 p=34636 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:58:33,115 p=34636 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 22:58:33,115 p=34636 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:58:33,130 p=34636 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:58:33,130 p=34636 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:58:33,408 p=34636 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 22:58:33,409 p=34636 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:58:33,435 p=34636 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 22:58:33,435 p=34636 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:58:33,459 p=34636 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 22:58:33,459 p=34636 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:58:33,469 p=34636 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 22:58:34,015 p=34636 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 22:58:34,016 p=34636 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 22:58:49,896 p=34636 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-1077] *** 2024-04-16 22:58:49,897 p=34636 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 22:58:50,158 p=34636 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 22:58:50,158 p=34636 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 22:58:50 Creating restore mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 for case mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 STEP: Create restore mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 from backup mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 @ 04/16/24 22:58:50.201 2024/04/16 22:58:50 Wait until restore mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 is complete restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: Completed 2024/04/16 23:00:30 Verify the PodVolumeBackup and PodVolumeRestore count is equal 2024/04/16 23:00:30 Verify the PodVolumeRestore is completed sucessfully and uploaderType is matching 2024/04/16 23:00:30 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-04-16T22:59:07Z" generateName: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5- generation: 5 labels: velero.io/pod-uid: c6cfd89a-da88-4efe-bb7d-90daef01b07b velero.io/pvc-uid: 1f39ac7d-a024-431b-9a25-4e35f7c8c8ef velero.io/restore-name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 velero.io/restore-uid: 91155a3e-e64e-4fbc-86c5-c9e9c16e753f managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"91155a3e-e64e-4fbc-86c5-c9e9c16e753f"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T22:59:07Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T23:00:22Z" name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5-92s94 namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 uid: 91155a3e-e64e-4fbc-86c5-c9e9c16e753f resourceVersion: "58452" uid: 04ff8b75-9723-470c-89b3-30bd3801ae38 spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-7f77d47fd7-q7q45 namespace: test-oadp-1077 uid: c6cfd89a-da88-4efe-bb7d-90daef01b07b repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-1077 snapshotID: 7cfe72e8 sourceNamespace: test-oadp-1077 uploaderType: restic volume: mysql-data1 status: completionTimestamp: "2024-04-16T23:00:22Z" phase: Completed progress: bytesDone: 104857640 totalBytes: 104857640 startTimestamp: "2024-04-16T23:00:19Z" 2024/04/16 23:00:30 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-04-16T22:59:07Z" generateName: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5- generation: 5 labels: velero.io/pod-uid: c6cfd89a-da88-4efe-bb7d-90daef01b07b velero.io/pvc-uid: a74f6b75-804f-415a-a194-04e61c62f48c velero.io/restore-name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 velero.io/restore-uid: 91155a3e-e64e-4fbc-86c5-c9e9c16e753f managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"91155a3e-e64e-4fbc-86c5-c9e9c16e753f"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T22:59:07Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T22:59:21Z" name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5-s2mhw namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5 uid: 91155a3e-e64e-4fbc-86c5-c9e9c16e753f resourceVersion: "58085" uid: 24e6cc91-85c0-47ef-bf79-3b9c9f7edac3 spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-7f77d47fd7-q7q45 namespace: test-oadp-1077 uid: c6cfd89a-da88-4efe-bb7d-90daef01b07b repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-1077 snapshotID: 5e3be896 sourceNamespace: test-oadp-1077 uploaderType: restic volume: mysql-data status: completionTimestamp: "2024-04-16T22:59:21Z" phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 startTimestamp: "2024-04-16T22:59:19Z" STEP: Verify restore mysql-8eff35dd-fc44-11ee-99f6-0a580a8231b5has completed successfully @ 04/16/24 23:00:30.436 STEP: Verify Application restore @ 04/16/24 23:00:30.45 STEP: Verify Application deployment for case mysql @ 04/16/24 23:00:30.45 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2024/04/16 23:00:35 2024-04-16 23:00:31,853 p=34845 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:00:31,854 p=34845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:32,075 p=34845 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:00:32,075 p=34845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:32,298 p=34845 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:00:32,299 p=34845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:32,314 p=34845 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:00:32,314 p=34845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:32,589 p=34845 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:00:32,590 p=34845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:32,616 p=34845 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:00:32,616 p=34845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:32,638 p=34845 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:00:32,639 p=34845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:32,649 p=34845 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:00:33,177 p=34845 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:00:33,177 p=34845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:34,247 p=34845 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-04-16 23:00:34,248 p=34845 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:34,574 p=34845 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 23:00:34,574 p=34845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:35,019 p=34845 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-04-16 23:00:35,019 p=34845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:35,644 p=34845 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-04-16 23:00:35,644 p=34845 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:35,657 p=34845 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:00:35,657 p=34845 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 2024/04/16 23:00:35 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 2024/04/16 23:00:35 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-1077] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 23:00:54 2024-04-16 23:00:37,105 p=35149 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:00:37,105 p=35149 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:37,328 p=35149 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:00:37,328 p=35149 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:37,546 p=35149 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:00:37,546 p=35149 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:37,560 p=35149 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:00:37,560 p=35149 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:37,839 p=35149 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:00:37,839 p=35149 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:37,865 p=35149 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:00:37,866 p=35149 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:37,888 p=35149 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:00:37,888 p=35149 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:37,899 p=35149 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:00:38,427 p=35149 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:00:38,428 p=35149 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:00:54,284 p=35149 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-1077] *** 2024-04-16 23:00:54,284 p=35149 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:00:54,542 p=35149 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:00:54,542 p=35149 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 23:00:54 Cleaning setup resources for the backup • [266.769 seconds] ------------------------------ Backup restore tests Application backup [tc-id:OADP-437][interop] MySQL application with filesystem, Kopia [mr-check] /alabama/cspi/e2e/app_backup/backup_restore.go:62 2024/04/16 23:00:54 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 04/16/24 23:00:54.681 2024/04/16 23:00:54 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "aa467b02-0ea5-4079-a366-87964dbe9963", "resourceVersion": "58733", "generation": 1, "creationTimestamp": "2024-04-16T23:00:54Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T23:00:54Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "kopia" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 23:00:54.842 2024/04/16 23:00:54 Waiting for velero pod to be running 2024/04/16 23:00:54 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 23:00:54.885 2024/04/16 23:00:54 Checking for correct number of running NodeAgent pods... 2024/04/16 23:00:54 pod: node-agent-h8lrw is not yet running with status: {Pending [] [] [] [] Burstable []} STEP: Installing application for case mysql @ 04/16/24 23:00:59.989 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-437-kopia] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Check pod status (30 retries left). FAILED - RETRYING: [localhost]: Check pod status (29 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** changed: [localhost] FAILED - RETRYING: [localhost]: Wait until service ready for connections (30 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=19  changed=10  unreachable=0 failed=0 skipped=5  rescued=0 ignored=0 2024/04/16 23:01:25 2024-04-16 23:01:01,374 p=35357 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:01:01,374 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:01,599 p=35357 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:01:01,599 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:01,820 p=35357 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:01:01,820 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:01,834 p=35357 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:01:01,834 p=35357 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:02,109 p=35357 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:01:02,109 p=35357 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:02,135 p=35357 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:01:02,135 p=35357 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:02,158 p=35357 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:01:02,158 p=35357 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:02,168 p=35357 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:01:02,681 p=35357 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:01:02,682 p=35357 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:03,466 p=35357 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check namespace test-oadp-437-kopia] *** 2024-04-16 23:01:03,467 p=35357 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:03,835 p=35357 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create namespace] *** 2024-04-16 23:01:03,836 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:04,786 p=35357 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Deploy a mysql pod] *** 2024-04-16 23:01:04,786 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:16,605 p=35357 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check pod status] *** 2024-04-16 23:01:16,605 p=35357 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:17,234 p=35357 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Copy mysql provision script to pod] *** 2024-04-16 23:01:17,234 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:22,851 p=35357 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 23:01:22,852 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:23,494 p=35357 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Provision the mysql database] *** 2024-04-16 23:01:23,494 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:24,367 p=35357 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Add dummy data into mysql-data1 pvc] *** 2024-04-16 23:01:24,367 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:24,971 p=35357 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Create md5 hashes for the files] *** 2024-04-16 23:01:24,971 p=35357 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:25,065 p=35357 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:01:25,065 p=35357 u=1009100000 n=ansible | localhost : ok=19 changed=10 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 23:01:25.108 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2024/04/16 23:01:30 2024-04-16 23:01:26,485 p=35841 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:01:26,485 p=35841 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:26,705 p=35841 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:01:26,705 p=35841 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:26,933 p=35841 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:01:26,933 p=35841 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:26,948 p=35841 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:01:26,948 p=35841 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:27,218 p=35841 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:01:27,218 p=35841 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:27,244 p=35841 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:01:27,245 p=35841 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:27,267 p=35841 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:01:27,267 p=35841 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:27,277 p=35841 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:01:27,803 p=35841 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:01:27,803 p=35841 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:28,800 p=35841 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-04-16 23:01:28,800 p=35841 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:01:29,119 p=35841 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 23:01:29,119 p=35841 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:29,549 p=35841 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-04-16 23:01:29,550 p=35841 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:30,148 p=35841 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-04-16 23:01:30,149 p=35841 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:01:30,161 p=35841 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:01:30,161 p=35841 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 2024/04/16 23:01:30 {{ } { } [{{ } {mysql test-oadp-437-kopia e6fcbd22-fa26-418f-a1fa-4b5ae15f07cf 59026 0 2024-04-16 23:01:04 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-20-65.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-04-16 23:01:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 23:01:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 23:01:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 23:01:09 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-e6fcbd22-fa26-418f-a1fa-4b5ae15f07cf 0xc00172ced0 0xc00172cee0 nil nil} {Bound [ReadWriteOnce] map[storage:{{2147483648 0} {} 2Gi BinarySI}] [] map[] }} {{ } {mysql-1 test-oadp-437-kopia 615d7476-f48c-462b-b75d-c72b37f05640 59029 0 2024-04-16 23:01:04 +0000 UTC map[app:mysql testlabel:selectors testlabel2:foo] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-20-65.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{OpenAPI-Generator Update v1 2024-04-16 23:01:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{},"f:testlabel":{},"f:testlabel2":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 23:01:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 23:01:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 23:01:09 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-615d7476-f48c-462b-b75d-c72b37f05640 0xc00172d130 0xc00172d140 nil nil} {Bound [ReadWriteOnce] map[storage:{{2147483648 0} {} 2Gi BinarySI}] [] map[] }}]} STEP: Creating backup mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 @ 04/16/24 23:01:30.216 2024/04/16 23:01:30 Wait until backup mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 is completed backup phase: InProgress backup phase: InProgress backup phase: InProgress backup phase: Completed 2024/04/16 23:02:50 Verify the PodVolumeBackup is completed successfully and BackupRepository type is matching with DPA.nodeAgent.uploaderType 2024/04/16 23:02:50 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql-1 creationTimestamp: "2024-04-16T23:01:35Z" generateName: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5- generation: 5 labels: velero.io/backup-name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 velero.io/backup-uid: 96d333dd-2865-4b05-8e61-b12153e970b7 velero.io/pvc-uid: 615d7476-f48c-462b-b75d-c72b37f05640 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"96d333dd-2865-4b05-8e61-b12153e970b7"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T23:01:35Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T23:02:40Z" name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5-xbt9s namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 uid: 96d333dd-2865-4b05-8e61-b12153e970b7 resourceVersion: "59568" uid: 4d4f1021-31a6-42a3-be59-ba265e285ef9 spec: backupStorageLocation: ts-dpa-1 node: ip-10-0-20-65.us-east-2.compute.internal pod: kind: Pod name: mysql-7f77d47fd7-4ctgk namespace: test-oadp-437-kopia uid: 2a4ee696-6612-4104-bc4f-386d571c825c repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-437-kopia tags: backup: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 backup-uid: 96d333dd-2865-4b05-8e61-b12153e970b7 ns: test-oadp-437-kopia pod: mysql-7f77d47fd7-4ctgk pod-uid: 2a4ee696-6612-4104-bc4f-386d571c825c pvc-uid: 615d7476-f48c-462b-b75d-c72b37f05640 volume: mysql-data1 uploaderType: kopia volume: mysql-data1 status: completionTimestamp: "2024-04-16T23:02:40Z" path: /host_pods/2a4ee696-6612-4104-bc4f-386d571c825c/volumes/kubernetes.io~csi/pvc-615d7476-f48c-462b-b75d-c72b37f05640/mount phase: Completed progress: bytesDone: 104857640 totalBytes: 104857640 snapshotID: 64ac08f5d40f68fefa875f8ce4ff63aa startTimestamp: "2024-04-16T23:02:37Z" 2024/04/16 23:02:50 apiVersion: velero.io/v1 kind: PodVolumeBackup metadata: annotations: velero.io/pvc-name: mysql creationTimestamp: "2024-04-16T23:01:35Z" generateName: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5- generation: 5 labels: velero.io/backup-name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 velero.io/backup-uid: 96d333dd-2865-4b05-8e61-b12153e970b7 velero.io/pvc-uid: e6fcbd22-fa26-418f-a1fa-4b5ae15f07cf managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:velero.io/pvc-name: {} f:generateName: {} f:labels: .: {} f:velero.io/backup-name: {} f:velero.io/backup-uid: {} f:velero.io/pvc-uid: {} f:ownerReferences: .: {} k:{"uid":"96d333dd-2865-4b05-8e61-b12153e970b7"}: {} f:spec: .: {} f:backupStorageLocation: {} f:node: {} f:pod: {} f:repoIdentifier: {} f:tags: .: {} f:backup: {} f:backup-uid: {} f:ns: {} f:pod: {} f:pod-uid: {} f:pvc-uid: {} f:volume: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T23:01:35Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:path: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:snapshotID: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T23:01:38Z" name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5-xtvfz namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Backup name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 uid: 96d333dd-2865-4b05-8e61-b12153e970b7 resourceVersion: "59242" uid: 3680bc2d-f4aa-42bf-ad9c-8406599de4e4 spec: backupStorageLocation: ts-dpa-1 node: ip-10-0-20-65.us-east-2.compute.internal pod: kind: Pod name: mysql-7f77d47fd7-4ctgk namespace: test-oadp-437-kopia uid: 2a4ee696-6612-4104-bc4f-386d571c825c repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-437-kopia tags: backup: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 backup-uid: 96d333dd-2865-4b05-8e61-b12153e970b7 ns: test-oadp-437-kopia pod: mysql-7f77d47fd7-4ctgk pod-uid: 2a4ee696-6612-4104-bc4f-386d571c825c pvc-uid: e6fcbd22-fa26-418f-a1fa-4b5ae15f07cf volume: mysql-data uploaderType: kopia volume: mysql-data status: completionTimestamp: "2024-04-16T23:01:38Z" path: /host_pods/2a4ee696-6612-4104-bc4f-386d571c825c/volumes/kubernetes.io~csi/pvc-e6fcbd22-fa26-418f-a1fa-4b5ae15f07cf/mount phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 snapshotID: f7d3e85cda5f22c7631b38183b1b7c22 startTimestamp: "2024-04-16T23:01:35Z" STEP: Verify backup mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 23:02:50.338 2024/04/16 23:02:50 Backup for case mysql succeeded STEP: Delete the appplication resources mysql @ 04/16/24 23:02:50.415 STEP: Cleanup Application for case mysql @ 04/16/24 23:02:50.415 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-437-kopia] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 23:03:04 2024-04-16 23:02:51,806 p=36145 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:02:51,807 p=36145 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:02:52,029 p=36145 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:02:52,030 p=36145 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:02:52,249 p=36145 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:02:52,249 p=36145 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:02:52,264 p=36145 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:02:52,264 p=36145 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:02:52,536 p=36145 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:02:52,537 p=36145 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:02:52,566 p=36145 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:02:52,566 p=36145 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:02:52,590 p=36145 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:02:52,590 p=36145 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:02:52,601 p=36145 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:02:53,120 p=36145 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:02:53,120 p=36145 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:03:03,966 p=36145 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-437-kopia] *** 2024-04-16 23:03:03,966 p=36145 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:03:04,241 p=36145 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:03:04,241 p=36145 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 23:03:04 Creating restore mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 for case mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 STEP: Create restore mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 from backup mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 @ 04/16/24 23:03:04.282 2024/04/16 23:03:04 Wait until restore mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 is complete restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: InProgress restore phase: Completed 2024/04/16 23:04:44 Verify the PodVolumeBackup and PodVolumeRestore count is equal 2024/04/16 23:04:44 Verify the PodVolumeRestore is completed sucessfully and uploaderType is matching 2024/04/16 23:04:44 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-04-16T23:03:21Z" generateName: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5- generation: 4 labels: velero.io/pod-uid: 5b03abd9-0aa7-4aea-b82d-7524f46e6672 velero.io/pvc-uid: 832ffa4c-219c-430c-bb68-c3fcae203ef8 velero.io/restore-name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 velero.io/restore-uid: ad47161f-dec8-407f-9cb8-ad081aad2fa7 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"ad47161f-dec8-407f-9cb8-ad081aad2fa7"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T23:03:21Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T23:03:39Z" name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5-mk8zw namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 uid: ad47161f-dec8-407f-9cb8-ad081aad2fa7 resourceVersion: "60126" uid: abd691b0-a892-43b8-a608-0823aa02c125 spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-7f77d47fd7-4ctgk namespace: test-oadp-437-kopia uid: 5b03abd9-0aa7-4aea-b82d-7524f46e6672 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-437-kopia snapshotID: f7d3e85cda5f22c7631b38183b1b7c22 sourceNamespace: test-oadp-437-kopia uploaderType: kopia volume: mysql-data status: completionTimestamp: "2024-04-16T23:03:39Z" phase: Completed progress: bytesDone: 107854713 totalBytes: 107854713 startTimestamp: "2024-04-16T23:03:37Z" 2024/04/16 23:04:44 apiVersion: velero.io/v1 kind: PodVolumeRestore metadata: creationTimestamp: "2024-04-16T23:03:21Z" generateName: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5- generation: 4 labels: velero.io/pod-uid: 5b03abd9-0aa7-4aea-b82d-7524f46e6672 velero.io/pvc-uid: b07fa73d-a593-43b9-b607-e8361aeef479 velero.io/restore-name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 velero.io/restore-uid: ad47161f-dec8-407f-9cb8-ad081aad2fa7 managedFields: - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:velero.io/pod-uid: {} f:velero.io/pvc-uid: {} f:velero.io/restore-name: {} f:velero.io/restore-uid: {} f:ownerReferences: .: {} k:{"uid":"ad47161f-dec8-407f-9cb8-ad081aad2fa7"}: {} f:spec: .: {} f:backupStorageLocation: {} f:pod: {} f:repoIdentifier: {} f:snapshotID: {} f:sourceNamespace: {} f:uploaderType: {} f:volume: {} f:status: .: {} f:progress: {} manager: velero-server operation: Update time: "2024-04-16T23:03:21Z" - apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:completionTimestamp: {} f:phase: {} f:progress: f:bytesDone: {} f:totalBytes: {} f:startTimestamp: {} manager: node-agent-server operation: Update time: "2024-04-16T23:04:42Z" name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5-tkhnq namespace: openshift-adp ownerReferences: - apiVersion: velero.io/v1 controller: true kind: Restore name: mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5 uid: ad47161f-dec8-407f-9cb8-ad081aad2fa7 resourceVersion: "60530" uid: cb2b376b-086f-4247-9432-697413694cd5 spec: backupStorageLocation: ts-dpa-1 pod: kind: Pod name: mysql-7f77d47fd7-4ctgk namespace: test-oadp-437-kopia uid: 5b03abd9-0aa7-4aea-b82d-7524f46e6672 repoIdentifier: s3:s3-us-east-2.amazonaws.com/ci-op-zciz01md-interopoadp/velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5/restic/test-oadp-437-kopia snapshotID: 64ac08f5d40f68fefa875f8ce4ff63aa sourceNamespace: test-oadp-437-kopia uploaderType: kopia volume: mysql-data1 status: completionTimestamp: "2024-04-16T23:04:42Z" phase: Completed progress: bytesDone: 104857640 totalBytes: 104857640 startTimestamp: "2024-04-16T23:04:38Z" STEP: Verify restore mysql-2dcfb382-fc45-11ee-99f6-0a580a8231b5has completed successfully @ 04/16/24 23:04:44.569 STEP: Verify Application restore @ 04/16/24 23:04:44.583 STEP: Verify Application deployment for case mysql @ 04/16/24 23:04:44.583 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=10  rescued=0 ignored=0 2024/04/16 23:04:49 2024-04-16 23:04:45,954 p=36355 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:04:45,955 p=36355 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:04:46,175 p=36355 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:04:46,175 p=36355 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:04:46,393 p=36355 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:04:46,393 p=36355 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:04:46,408 p=36355 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:04:46,408 p=36355 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:46,678 p=36355 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:04:46,678 p=36355 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:46,705 p=36355 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:04:46,705 p=36355 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:46,727 p=36355 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:04:46,727 p=36355 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:46,737 p=36355 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:04:47,250 p=36355 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:04:47,250 p=36355 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:48,300 p=36355 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Check mysql pod status] *** 2024-04-16 23:04:48,300 p=36355 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:48,617 p=36355 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Wait until service ready for connections] *** 2024-04-16 23:04:48,618 p=36355 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:04:49,052 p=36355 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Query the inserted data] *** 2024-04-16 23:04:49,052 p=36355 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:04:49,644 p=36355 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Validate test1 file has correct md5 hash] *** 2024-04-16 23:04:49,644 p=36355 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:04:49,657 p=36355 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:04:49,657 p=36355 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 2024/04/16 23:04:49 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 2024/04/16 23:04:49 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-437-kopia] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=13  rescued=0 ignored=0 2024/04/16 23:05:03 2024-04-16 23:04:51,085 p=36661 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:04:51,085 p=36661 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:04:51,306 p=36661 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:04:51,306 p=36661 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:04:51,524 p=36661 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:04:51,525 p=36661 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:04:51,539 p=36661 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:04:51,539 p=36661 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:51,807 p=36661 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:04:51,808 p=36661 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:51,833 p=36661 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:04:51,834 p=36661 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:51,856 p=36661 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:04:51,856 p=36661 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:04:51,866 p=36661 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:04:52,380 p=36661 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:04:52,380 p=36661 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:03,203 p=36661 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-mysql : Remove namespace test-oadp-437-kopia] *** 2024-04-16 23:05:03,204 p=36661 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:03,469 p=36661 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:05:03,469 p=36661 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 2024/04/16 23:05:03 Cleaning setup resources for the backup • [248.911 seconds] ------------------------------ S ------------------------------ Backup restore tests Application backup [tc-id:OADP-122] [interop] [skip-disconnected] Django application with BSL&CSI [exclude_aro-4] /alabama/cspi/e2e/app_backup/backup_restore.go:93 2024/04/16 23:05:03 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 04/16/24 23:05:03.584 2024/04/16 23:05:03 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "faba8ff5-bcc1-4a54-9215-604ad66752a0", "resourceVersion": "60769", "generation": 1, "creationTimestamp": "2024-04-16T23:05:03Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T23:05:03Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt", "csi" ] } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 23:05:03.712 2024/04/16 23:05:03 Waiting for velero pod to be running 2024/04/16 23:05:03 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 23:05:03.76 Run the command: oc get ns openshift-storage &> /dev/null && echo true || echo false 2024/04/16 23:05:03 The 'openshift-storage' namespace does not exist 2024/04/16 23:05:03 Using default CSI driver based on infrastructure: ebs.csi.aws.com 2024/04/16 23:05:03 Snapclass 'example-snapclass' doesn't exist, creating 2024/04/16 23:05:03 Setting new default StorageClass 'gp2-csi' 2024/04/16 23:05:03 Checking default storage class count Run the command: oc get sc 2024/04/16 23:05:04 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 74m gp3-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 74m STEP: Installing application for case django-persistent @ 04/16/24 23:05:04.137 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check namespace test-oadp-122] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create namespace test-oadp-122] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create the mtc test django psql persistent template] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create openshift django psql persistent application from openshift templates] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/04/16 23:05:09 2024-04-16 23:05:05,509 p=36910 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:05:05,509 p=36910 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:05,729 p=36910 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:05:05,729 p=36910 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:05,952 p=36910 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:05:05,952 p=36910 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:05,966 p=36910 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:05:05,966 p=36910 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:06,234 p=36910 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:05:06,235 p=36910 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:06,260 p=36910 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:05:06,261 p=36910 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:06,283 p=36910 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:05:06,283 p=36910 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:06,293 p=36910 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:05:06,805 p=36910 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:05:06,805 p=36910 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:07,566 p=36910 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check namespace test-oadp-122] *** 2024-04-16 23:05:07,566 p=36910 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:07,953 p=36910 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create namespace test-oadp-122] *** 2024-04-16 23:05:07,953 p=36910 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:08,779 p=36910 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create the mtc test django psql persistent template] *** 2024-04-16 23:05:08,780 p=36910 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:09,377 p=36910 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create openshift django psql persistent application from openshift templates] *** 2024-04-16 23:05:09,378 p=36910 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:09,544 p=36910 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:05:09,544 p=36910 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 23:05:09.584 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] FAILED - RETRYING: [localhost]: Check postgresql pod status (30 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (29 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (28 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (27 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (26 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (25 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Check application pod status (30 retries left). FAILED - RETRYING: [localhost]: Check application pod status (29 retries left). FAILED - RETRYING: [localhost]: Check application pod status (28 retries left). FAILED - RETRYING: [localhost]: Check application pod status (27 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** ok: [localhost] => {  "msg": "PASS: # of visits should be 1; actual 1" } PLAY RECAP ********************************************************************* localhost : ok=17  changed=3  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/04/16 23:06:11 2024-04-16 23:05:10,951 p=37195 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:05:10,951 p=37195 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:11,172 p=37195 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:05:11,172 p=37195 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:11,389 p=37195 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:05:11,389 p=37195 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:05:11,403 p=37195 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:05:11,404 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:11,674 p=37195 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:05:11,674 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:11,701 p=37195 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:05:11,701 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:11,723 p=37195 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:05:11,723 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:11,733 p=37195 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:05:12,246 p=37195 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:05:12,246 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:05:46,549 p=37195 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** 2024-04-16 23:05:46,549 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:06:09,449 p=37195 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** 2024-04-16 23:06:09,449 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:06:10,900 p=37195 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** 2024-04-16 23:06:10,901 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:06:11,350 p=37195 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** 2024-04-16 23:06:11,350 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:06:11,378 p=37195 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** 2024-04-16 23:06:11,378 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:06:11,694 p=37195 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** 2024-04-16 23:06:11,694 p=37195 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:06:11,740 p=37195 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** 2024-04-16 23:06:11,741 p=37195 u=1009100000 n=ansible | ok: [localhost] => { "msg": "PASS: # of visits should be 1; actual 1" } 2024-04-16 23:06:11,752 p=37195 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:06:11,752 p=37195 u=1009100000 n=ansible | localhost : ok=17 changed=3 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 2024/04/16 23:06:11 {{ } { } [{{ } {postgresql test-oadp-122 6232d313-8352-4e4b-96b1-ac6645fb32e6 61029 0 2024-04-16 23:05:09 +0000 UTC map[app:django-psql-persistent template:django-psql-persistent] map[openshift.io/generated-by:OpenShiftNewApp pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-89-31.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{oc Update v1 2024-04-16 23:05:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/generated-by":{}},"f:labels":{".":{},"f:app":{},"f:template":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 23:05:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 23:05:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 23:05:14 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{1073741824 0} {} 1Gi BinarySI}]} pvc-6232d313-8352-4e4b-96b1-ac6645fb32e6 0xc0009f88c0 0xc0009f88d0 nil nil} {Bound [ReadWriteOnce] map[storage:{{1073741824 0} {} 1Gi BinarySI}] [] map[] }}]} STEP: Creating backup django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 @ 04/16/24 23:06:11.811 2024/04/16 23:06:11 Wait until backup django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 is completed backup phase: WaitingForPluginOperations backup phase: WaitingForPluginOperations backup phase: WaitingForPluginOperations backup phase: Completed 2024/04/16 23:07:31 Verify the PVC and VSC count is matching 2024/04/16 23:07:31 Verify the Backup has CSIVolumeSnapshotsAttempted and CSIVolumeSnapshotsCompleted field on status 2024/04/16 23:07:32 Run velero describe on the backup 2024/04/16 23:07:32 [./velero describe backup django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 -n openshift-adp --details] 2024/04/16 23:07:32 Exec stderr: "" 2024/04/16 23:07:32 Name: django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 Namespace: openshift-adp Labels: velero.io/storage-location=ts-dpa-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.3+5fa1806 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: Completed Namespaces: Included: test-oadp-122 Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: ts-dpa-1 Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-04-16 23:06:12 +0000 UTC Completed: 2024-04-16 23:07:24 +0000 UTC Expiration: 2024-05-16 23:06:11 +0000 UTC Total items to be backed up: 99 Items backed up: 99 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io test-oadp-122/velero-postgresql-9scp9: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: test-oadp-122/velero-postgresql-9scp9/2024-04-16T23:06:21Z Items to Update: volumesnapshots.snapshot.storage.k8s.io test-oadp-122/velero-postgresql-9scp9 Phase: Completed Created: 2024-04-16 23:06:21 +0000 UTC Started: 2024-04-16 23:06:21 +0000 UTC Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-401c46e7-5c19-496d-87ac-c25e55ae80e4: Backup Item Action Plugin: velero.io/csi-volumesnapshotcontent-backupper Operation ID: snapcontent-401c46e7-5c19-496d-87ac-c25e55ae80e4/2024-04-16T23:06:21Z Items to Update: volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-401c46e7-5c19-496d-87ac-c25e55ae80e4 Phase: Completed Created: 2024-04-16 23:06:21 +0000 UTC Started: 2024-04-16 23:06:21 +0000 UTC Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - clusterserviceversions.operators.coreos.com apps.openshift.io/v1/DeploymentConfig: - test-oadp-122/django-psql-persistent - test-oadp-122/postgresql authorization.openshift.io/v1/RoleBinding: - test-oadp-122/admin - test-oadp-122/system:deployers - test-oadp-122/system:image-builders - test-oadp-122/system:image-pullers build.openshift.io/v1/Build: - test-oadp-122/django-psql-persistent-1 build.openshift.io/v1/BuildConfig: - test-oadp-122/django-psql-persistent discovery.k8s.io/v1/EndpointSlice: - test-oadp-122/django-psql-persistent-4rft6 - test-oadp-122/postgresql-xzwps image.openshift.io/v1/ImageStream: - test-oadp-122/django-psql-persistent image.openshift.io/v1/ImageStreamTag: - test-oadp-122/django-psql-persistent:latest image.openshift.io/v1/ImageTag: - test-oadp-122/django-psql-persistent:latest operators.coreos.com/v1alpha1/ClusterServiceVersion: - test-oadp-122/volsync-product.v0.9.1 rbac.authorization.k8s.io/v1/RoleBinding: - test-oadp-122/admin - test-oadp-122/system:deployers - test-oadp-122/system:image-builders - test-oadp-122/system:image-pullers route.openshift.io/v1/Route: - test-oadp-122/django-psql-persistent snapshot.storage.k8s.io/v1/VolumeSnapshot: - test-oadp-122/velero-postgresql-9scp9 snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - example-snapclass snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-401c46e7-5c19-496d-87ac-c25e55ae80e4 template.openshift.io/v1/Template: - test-oadp-122/mtc-test-django-psql-persistent v1/ConfigMap: - test-oadp-122/django-psql-persistent-1-ca - test-oadp-122/django-psql-persistent-1-global-ca - test-oadp-122/django-psql-persistent-1-sys-config - test-oadp-122/kube-root-ca.crt - test-oadp-122/openshift-service-ca.crt v1/Endpoints: - test-oadp-122/django-psql-persistent - test-oadp-122/postgresql v1/Event: - test-oadp-122/django-psql-persistent-1-5qxqm.17c6e53c5217b08c - test-oadp-122/django-psql-persistent-1-5qxqm.17c6e53c76a85df4 - test-oadp-122/django-psql-persistent-1-5qxqm.17c6e53c781de5bc - test-oadp-122/django-psql-persistent-1-5qxqm.17c6e53eeb8c8f06 - test-oadp-122/django-psql-persistent-1-5qxqm.17c6e53ef1b715f3 - test-oadp-122/django-psql-persistent-1-5qxqm.17c6e53ef2c7d526 - test-oadp-122/django-psql-persistent-1-build.17c6e53195aa30d8 - test-oadp-122/django-psql-persistent-1-build.17c6e531baae023f - test-oadp-122/django-psql-persistent-1-build.17c6e531bc74143b - test-oadp-122/django-psql-persistent-1-build.17c6e532a057037f - test-oadp-122/django-psql-persistent-1-build.17c6e532aad1ad41 - test-oadp-122/django-psql-persistent-1-build.17c6e532ab9c8815 - test-oadp-122/django-psql-persistent-1-build.17c6e532dc258b05 - test-oadp-122/django-psql-persistent-1-build.17c6e532e6c90d3f - test-oadp-122/django-psql-persistent-1-build.17c6e532e80b29ee - test-oadp-122/django-psql-persistent-1-build.17c6e53317e20789 - test-oadp-122/django-psql-persistent-1-build.17c6e53322b31566 - test-oadp-122/django-psql-persistent-1-build.17c6e5332391ec92 - test-oadp-122/django-psql-persistent-1-deploy.17c6e53c1fd8aa48 - test-oadp-122/django-psql-persistent-1-deploy.17c6e53c43a200d8 - test-oadp-122/django-psql-persistent-1-deploy.17c6e53c4510b143 - test-oadp-122/django-psql-persistent-1-deploy.17c6e53c4aa872d1 - test-oadp-122/django-psql-persistent-1-deploy.17c6e53c4ba7157a - test-oadp-122/django-psql-persistent-1.17c6e532de4a894c - test-oadp-122/django-psql-persistent-1.17c6e53c51803150 - test-oadp-122/django-psql-persistent-1.17c6e53c7a2bbbb6 - test-oadp-122/django-psql-persistent.17c6e53c1db715b4 - test-oadp-122/postgresql-1-deploy.17c6e53197c26dfd - test-oadp-122/postgresql-1-deploy.17c6e531bcf4de8e - test-oadp-122/postgresql-1-deploy.17c6e531be2e66f0 - test-oadp-122/postgresql-1-deploy.17c6e531f10d5178 - test-oadp-122/postgresql-1-deploy.17c6e531f7758f54 - test-oadp-122/postgresql-1-deploy.17c6e531f8a87cf2 - test-oadp-122/postgresql-1-t64h7.17c6e532eef91970 - test-oadp-122/postgresql-1-t64h7.17c6e5337b9c28ef - test-oadp-122/postgresql-1-t64h7.17c6e5353610e6a5 - test-oadp-122/postgresql-1-t64h7.17c6e53537831dc5 - test-oadp-122/postgresql-1-t64h7.17c6e537642f826e - test-oadp-122/postgresql-1-t64h7.17c6e5376c89d349 - test-oadp-122/postgresql-1-t64h7.17c6e5376fa87d9b - test-oadp-122/postgresql-1.17c6e531ffa5484e - test-oadp-122/postgresql.17c6e5318e906376 - test-oadp-122/postgresql.17c6e53193b0aaa9 - test-oadp-122/postgresql.17c6e532008e3164 - test-oadp-122/postgresql.17c6e53200969143 - test-oadp-122/postgresql.17c6e532d53a89ad v1/Namespace: - test-oadp-122 v1/PersistentVolume: - pvc-6232d313-8352-4e4b-96b1-ac6645fb32e6 v1/PersistentVolumeClaim: - test-oadp-122/postgresql v1/Pod: - test-oadp-122/django-psql-persistent-1-5qxqm - test-oadp-122/django-psql-persistent-1-build - test-oadp-122/django-psql-persistent-1-deploy - test-oadp-122/postgresql-1-deploy - test-oadp-122/postgresql-1-t64h7 v1/ReplicationController: - test-oadp-122/django-psql-persistent-1 - test-oadp-122/postgresql-1 v1/Secret: - test-oadp-122/builder-dockercfg-jlfkf - test-oadp-122/builder-token-kdsph - test-oadp-122/default-dockercfg-lgtxz - test-oadp-122/default-token-2wrkd - test-oadp-122/deployer-dockercfg-j9bqd - test-oadp-122/deployer-token-xvxwd - test-oadp-122/django-psql-persistent v1/Service: - test-oadp-122/django-psql-persistent - test-oadp-122/postgresql v1/ServiceAccount: - test-oadp-122/builder - test-oadp-122/default - test-oadp-122/deployer Velero-Native Snapshots: STEP: Verify backup django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 23:07:32.47 2024/04/16 23:07:32 Backup for case django-persistent succeeded STEP: Delete the appplication resources django-persistent @ 04/16/24 23:07:32.538 STEP: Cleanup Application for case django-persistent @ 04/16/24 23:07:32.538 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-122] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/04/16 23:07:46 2024-04-16 23:07:33,947 p=37584 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:07:33,948 p=37584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:07:34,168 p=37584 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:07:34,168 p=37584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:07:34,385 p=37584 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:07:34,385 p=37584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:07:34,400 p=37584 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:07:34,400 p=37584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:07:34,677 p=37584 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:07:34,677 p=37584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:07:34,704 p=37584 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:07:34,704 p=37584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:07:34,726 p=37584 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:07:34,727 p=37584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:07:34,737 p=37584 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:07:35,269 p=37584 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:07:35,269 p=37584 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:07:46,089 p=37584 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-122] *** 2024-04-16 23:07:46,089 p=37584 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:07:46,334 p=37584 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:07:46,334 p=37584 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/04/16 23:07:46 Creating restore django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 for case django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 STEP: Create restore django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 from backup django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 @ 04/16/24 23:07:46.377 2024/04/16 23:07:46 Wait until restore django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5 is complete restore phase: InProgress restore phase: InProgress restore phase: Completed STEP: Verify restore django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5has completed successfully @ 04/16/24 23:08:06.453 STEP: Verify Application restore @ 04/16/24 23:08:06.469 STEP: Verify Application deployment for case django-persistent @ 04/16/24 23:08:06.469 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] FAILED - RETRYING: [localhost]: Check postgresql pod status (30 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (29 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (28 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (27 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Check application pod status (30 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** ok: [localhost] FAILED - RETRYING: [localhost]: Access the html file (20 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** ok: [localhost] => {  "msg": "PASS: # of visits should be 2; actual 2" } PLAY RECAP ********************************************************************* localhost : ok=17  changed=3  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/04/16 23:09:11 2024-04-16 23:08:07,871 p=37793 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:08:07,871 p=37793 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:08:08,090 p=37793 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:08:08,090 p=37793 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:08:08,309 p=37793 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:08:08,309 p=37793 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:08:08,324 p=37793 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:08:08,324 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:08:08,601 p=37793 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:08:08,601 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:08:08,628 p=37793 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:08:08,628 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:08:08,650 p=37793 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:08:08,651 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:08:08,662 p=37793 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:08:09,178 p=37793 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:08:09,178 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:08:32,475 p=37793 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** 2024-04-16 23:08:32,475 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:08:38,840 p=37793 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** 2024-04-16 23:08:38,840 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:08:40,410 p=37793 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** 2024-04-16 23:08:40,410 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:11,230 p=37793 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** 2024-04-16 23:09:11,230 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:11,259 p=37793 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** 2024-04-16 23:09:11,260 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:11,606 p=37793 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** 2024-04-16 23:09:11,606 p=37793 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:11,657 p=37793 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** 2024-04-16 23:09:11,657 p=37793 u=1009100000 n=ansible | ok: [localhost] => { "msg": "PASS: # of visits should be 2; actual 2" } 2024-04-16 23:09:11,672 p=37793 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:09:11,672 p=37793 u=1009100000 n=ansible | localhost : ok=17 changed=3 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 2024/04/16 23:09:11 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 2024/04/16 23:09:11 Reset number of visits to 0 2024/04/16 23:09:11 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-122] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/04/16 23:09:30 2024-04-16 23:09:13,304 p=38133 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:09:13,305 p=38133 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:13,535 p=38133 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:09:13,535 p=38133 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:13,759 p=38133 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:09:13,759 p=38133 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:13,775 p=38133 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:09:13,775 p=38133 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:14,056 p=38133 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:09:14,056 p=38133 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:14,083 p=38133 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:09:14,083 p=38133 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:14,106 p=38133 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:09:14,106 p=38133 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:14,118 p=38133 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:09:14,665 p=38133 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:09:14,665 p=38133 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:30,616 p=38133 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-122] *** 2024-04-16 23:09:30,616 p=38133 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:30,878 p=38133 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:09:30,878 p=38133 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/04/16 23:09:30 Cleaning setup resources for the backup 2024/04/16 23:09:30 Setting new default StorageClass 'gp3-csi' 2024/04/16 23:09:30 Checking default storage class count Run the command: oc get sc 2024/04/16 23:09:31 oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 78m gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 78m 2024/04/16 23:09:31 Deleting VolumeSnapshotClass 'example-snapclass' • [267.632 seconds] ------------------------------ SS ------------------------------ Backup restore tests Application backup [tc-id:OADP-352][interop][skip-disconnected] Django application with BSL&VSL [vsl] /alabama/cspi/e2e/app_backup/backup_restore.go:145 2024/04/16 23:09:31 Check if VSL custom credentials exist 2024/04/16 23:09:31 Check if the cloud provider is AWS 2024/04/16 23:09:31 Delete all downloadrequest django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5-9d4cd595-723a-4096-b937-57916962c441 django-persistent-c22afb85-fc45-11ee-99f6-0a580a8231b5-da5327f1-a595-4aca-9a6c-68340e7bde53 STEP: Create DPA CR @ 04/16/24 23:09:31.268 2024/04/16 23:09:31 Check if VSL custom credentials exist 2024/04/16 23:09:31 Check if the cloud provider is AWS 2024/04/16 23:09:31 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "231d51ad-e202-4a36-befb-628044287861", "resourceVersion": "64210", "generation": 1, "creationTimestamp": "2024-04-16T23:09:31Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T23:09:31Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [ { "velero": { "provider": "aws", "config": { "profile": "default", "region": "us-east-2" } } } ], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt" ] } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 23:09:31.384 2024/04/16 23:09:31 Waiting for velero pod to be running 2024/04/16 23:09:31 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 23:09:31.419 2024/04/16 23:09:31 Checking default storage class count STEP: Installing application for case django-persistent @ 04/16/24 23:09:31.462 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check namespace test-oadp-352] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create namespace test-oadp-352] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create the mtc test django psql persistent template] *** changed: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create openshift django psql persistent application from openshift templates] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14  changed=6  unreachable=0 failed=0 skipped=9  rescued=0 ignored=0 2024/04/16 23:09:37 2024-04-16 23:09:32,835 p=38359 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:09:32,835 p=38359 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:33,057 p=38359 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:09:33,058 p=38359 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:33,278 p=38359 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:09:33,279 p=38359 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:33,294 p=38359 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:09:33,294 p=38359 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:33,565 p=38359 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:09:33,565 p=38359 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:33,591 p=38359 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:09:33,591 p=38359 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:33,613 p=38359 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:09:33,613 p=38359 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:33,623 p=38359 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:09:34,140 p=38359 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:09:34,140 p=38359 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:34,906 p=38359 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check namespace test-oadp-352] *** 2024-04-16 23:09:34,906 p=38359 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:35,295 p=38359 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create namespace test-oadp-352] *** 2024-04-16 23:09:35,296 p=38359 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:36,151 p=38359 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create the mtc test django psql persistent template] *** 2024-04-16 23:09:36,151 p=38359 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:36,941 p=38359 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Create openshift django psql persistent application from openshift templates] *** 2024-04-16 23:09:36,941 p=38359 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:37,107 p=38359 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:09:37,107 p=38359 u=1009100000 n=ansible | localhost : ok=14 changed=6 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 23:09:37.149 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] FAILED - RETRYING: [localhost]: Check postgresql pod status (30 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (29 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (28 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (27 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (26 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Check application pod status (30 retries left). FAILED - RETRYING: [localhost]: Check application pod status (29 retries left). FAILED - RETRYING: [localhost]: Check application pod status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** ok: [localhost] => {  "msg": "PASS: # of visits should be 1; actual 1" } PLAY RECAP ********************************************************************* localhost : ok=17  changed=3  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/04/16 23:10:28 2024-04-16 23:09:38,534 p=38644 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:09:38,534 p=38644 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:38,755 p=38644 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:09:38,756 p=38644 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:38,973 p=38644 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:09:38,973 p=38644 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:09:38,988 p=38644 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:09:38,989 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:39,260 p=38644 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:09:39,260 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:39,287 p=38644 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:09:39,287 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:39,309 p=38644 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:09:39,309 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:09:39,320 p=38644 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:09:39,835 p=38644 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:09:39,835 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:08,628 p=38644 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** 2024-04-16 23:10:08,629 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:25,967 p=38644 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** 2024-04-16 23:10:25,967 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:27,419 p=38644 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** 2024-04-16 23:10:27,419 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:27,881 p=38644 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** 2024-04-16 23:10:27,882 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:27,909 p=38644 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** 2024-04-16 23:10:27,909 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:28,225 p=38644 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** 2024-04-16 23:10:28,226 p=38644 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:28,270 p=38644 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** 2024-04-16 23:10:28,271 p=38644 u=1009100000 n=ansible | ok: [localhost] => { "msg": "PASS: # of visits should be 1; actual 1" } 2024-04-16 23:10:28,282 p=38644 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:10:28,282 p=38644 u=1009100000 n=ansible | localhost : ok=17 changed=3 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 2024/04/16 23:10:28 {{ } { } [{{ } {postgresql test-oadp-352 8cd5ba52-2c70-45d9-8ed0-9d7161303ebe 64466 0 2024-04-16 23:09:36 +0000 UTC map[app:django-psql-persistent template:django-psql-persistent] map[openshift.io/generated-by:OpenShiftNewApp pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:ebs.csi.aws.com volume.kubernetes.io/selected-node:ip-10-0-20-65.us-east-2.compute.internal volume.kubernetes.io/storage-provisioner:ebs.csi.aws.com] [] [kubernetes.io/pvc-protection] [{oc Update v1 2024-04-16 23:09:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/generated-by":{}},"f:labels":{".":{},"f:app":{},"f:template":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}}} } {kube-scheduler Update v1 2024-04-16 23:09:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:volume.kubernetes.io/selected-node":{}}}} } {kube-controller-manager Update v1 2024-04-16 23:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-04-16 23:09:42 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{1073741824 0} {} 1Gi BinarySI}]} pvc-8cd5ba52-2c70-45d9-8ed0-9d7161303ebe 0xc0008b04f0 0xc0008b0500 nil nil} {Bound [ReadWriteOnce] map[storage:{{1073741824 0} {} 1Gi BinarySI}] [] map[] }}]} STEP: Creating backup django-persistent-61b0e7d8-fc46-11ee-99f6-0a580a8231b5 @ 04/16/24 23:10:28.338 2024/04/16 23:10:28 Wait until backup django-persistent-61b0e7d8-fc46-11ee-99f6-0a580a8231b5 is completed backup phase: Completed STEP: Verify backup django-persistent-61b0e7d8-fc46-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 23:10:48.395 2024/04/16 23:10:48 Backup for case django-persistent succeeded STEP: Delete the appplication resources django-persistent @ 04/16/24 23:10:48.494 STEP: Cleanup Application for case django-persistent @ 04/16/24 23:10:48.494 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-352] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/04/16 23:11:02 2024-04-16 23:10:49,868 p=39008 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:10:49,868 p=39008 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:10:50,091 p=39008 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:10:50,091 p=39008 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:10:50,310 p=39008 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:10:50,311 p=39008 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:10:50,325 p=39008 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:10:50,325 p=39008 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:50,594 p=39008 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:10:50,595 p=39008 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:50,620 p=39008 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:10:50,620 p=39008 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:50,642 p=39008 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:10:50,642 p=39008 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:10:50,652 p=39008 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:10:51,166 p=39008 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:10:51,166 p=39008 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:11:02,027 p=39008 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-352] *** 2024-04-16 23:11:02,027 p=39008 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:11:02,275 p=39008 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:11:02,275 p=39008 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/04/16 23:11:02 Creating restore django-persistent-61b0e7d8-fc46-11ee-99f6-0a580a8231b5 for case django-persistent-61b0e7d8-fc46-11ee-99f6-0a580a8231b5 STEP: Create restore django-persistent-61b0e7d8-fc46-11ee-99f6-0a580a8231b5 from backup django-persistent-61b0e7d8-fc46-11ee-99f6-0a580a8231b5 @ 04/16/24 23:11:02.318 2024/04/16 23:11:02 Wait until restore django-persistent-61b0e7d8-fc46-11ee-99f6-0a580a8231b5 is complete restore phase: InProgress restore phase: InProgress restore phase: Completed STEP: Verify restore django-persistent-61b0e7d8-fc46-11ee-99f6-0a580a8231b5has completed successfully @ 04/16/24 23:11:22.387 STEP: Verify Application restore @ 04/16/24 23:11:22.401 STEP: Verify Application deployment for case django-persistent @ 04/16/24 23:11:22.401 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] FAILED - RETRYING: [localhost]: Check postgresql pod status (30 retries left). FAILED - RETRYING: [localhost]: Check postgresql pod status (29 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** ok: [localhost] FAILED - RETRYING: [localhost]: Check application pod status (30 retries left). FAILED - RETRYING: [localhost]: Check application pod status (29 retries left). FAILED - RETRYING: [localhost]: Check application pod status (28 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** ok: [localhost] FAILED - RETRYING: [localhost]: Access the html file (20 retries left). TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** ok: [localhost] => {  "msg": "PASS: # of visits should be 2; actual 2" } PLAY RECAP ********************************************************************* localhost : ok=17  changed=3  unreachable=0 failed=0 skipped=6  rescued=0 ignored=0 2024/04/16 23:12:27 2024-04-16 23:11:23,764 p=39215 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:11:23,764 p=39215 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:11:23,981 p=39215 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:11:23,981 p=39215 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:11:24,201 p=39215 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:11:24,201 p=39215 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:11:24,215 p=39215 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:11:24,216 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:11:24,487 p=39215 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:11:24,487 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:11:24,513 p=39215 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:11:24,513 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:11:24,535 p=39215 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:11:24,536 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:11:24,546 p=39215 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:11:25,059 p=39215 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:11:25,059 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:11:37,164 p=39215 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check postgresql pod status] *** 2024-04-16 23:11:37,164 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:11:54,542 p=39215 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Check application pod status] *** 2024-04-16 23:11:54,543 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:11:56,064 p=39215 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get route] *** 2024-04-16 23:11:56,065 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:27,078 p=39215 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Access the html file] *** 2024-04-16 23:12:27,079 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:27,122 p=39215 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : set_fact] *** 2024-04-16 23:12:27,122 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:27,640 p=39215 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Get num visits up to now] *** 2024-04-16 23:12:27,640 p=39215 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:27,714 p=39215 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Print num of visits] *** 2024-04-16 23:12:27,714 p=39215 u=1009100000 n=ansible | ok: [localhost] => { "msg": "PASS: # of visits should be 2; actual 2" } 2024-04-16 23:12:27,734 p=39215 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:12:27,735 p=39215 u=1009100000 n=ansible | localhost : ok=17 changed=3 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 2024/04/16 23:12:27 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 2024/04/16 23:12:27 Reset number of visits to 0 2024/04/16 23:12:27 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-352] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=12  rescued=0 ignored=0 2024/04/16 23:12:48 2024-04-16 23:12:30,173 p=39554 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:12:30,173 p=39554 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:12:30,561 p=39554 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:12:30,561 p=39554 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:12:30,963 p=39554 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:12:30,963 p=39554 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:12:30,986 p=39554 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:12:30,986 p=39554 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:31,444 p=39554 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:12:31,445 p=39554 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:31,489 p=39554 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:12:31,489 p=39554 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:31,525 p=39554 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:12:31,525 p=39554 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:31,544 p=39554 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:12:32,281 p=39554 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:12:32,281 p=39554 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:48,447 p=39554 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-django : Remove namespace test-oadp-352] *** 2024-04-16 23:12:48,447 p=39554 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:12:48,690 p=39554 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:12:48,690 p=39554 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 2024/04/16 23:12:48 Cleaning setup resources for the backup • [197.587 seconds] ------------------------------ SS ------------------------------ Backup restore tests Application backup [tc-id:OADP-97][interop] Empty-project application with Restic /alabama/cspi/e2e/app_backup/backup_restore.go:191 2024/04/16 23:12:48 Delete all downloadrequest No download requests are found STEP: Create DPA CR @ 04/16/24 23:12:48.809 2024/04/16 23:12:48 { "metadata": { "name": "ts-dpa", "namespace": "openshift-adp", "uid": "597b2146-0fd0-44c4-b4ad-0b28063b262a", "resourceVersion": "66126", "generation": 1, "creationTimestamp": "2024-04-16T23:12:48Z", "managedFields": [ { "manager": "e2e.test", "operation": "Update", "apiVersion": "oadp.openshift.io/v1alpha1", "time": "2024-04-16T23:12:48Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:spec": { ".": {}, "f:backupLocations": {}, "f:configuration": { ".": {}, "f:nodeAgent": { ".": {}, "f:enable": {}, "f:podConfig": { ".": {}, "f:resourceAllocations": {} }, "f:uploaderType": {} }, "f:velero": { ".": {}, "f:defaultPlugins": {} } }, "f:podDnsConfig": {}, "f:snapshotLocations": {} } } } ] }, "spec": { "backupLocations": [ { "velero": { "provider": "aws", "config": { "region": "us-east-2" }, "credential": { "name": "cloud-credentials", "key": "cloud" }, "objectStorage": { "bucket": "ci-op-zciz01md-interopoadp", "prefix": "velero-e2e-0cc4d3da-fc3f-11ee-99f6-0a580a8231b5" }, "default": true } } ], "snapshotLocations": [], "podDnsConfig": {}, "configuration": { "velero": { "defaultPlugins": [ "openshift", "aws", "kubevirt" ] }, "nodeAgent": { "enable": true, "podConfig": { "resourceAllocations": {} }, "uploaderType": "restic" } }, "features": null }, "status": {} } Delete all the backups that remained in the phase InProgress Deleting backup CRs in progress Deletion of backup CRs in progress completed Delete all the restores that remained in the phase InProgress Deleting restore CRs in progress Deletion of restore CRs in progress completed STEP: Verify DPA CR setup @ 04/16/24 23:12:48.923 2024/04/16 23:12:48 Waiting for velero pod to be running 2024/04/16 23:12:48 Wait for DPA status.condition.reason to be 'Completed' and and message to be 'Reconcile complete' STEP: Prepare backup resources, depending on the volumes backup type @ 04/16/24 23:12:48.982 2024/04/16 23:12:48 Checking for correct number of running NodeAgent pods... 2024/04/16 23:12:49 pod: node-agent-2c24m is not yet running with status: {Pending [] [] [] [] Burstable []} STEP: Installing application for case empty-project-e2e @ 04/16/24 23:12:54.076 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Deploy project with labels and selectors] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/04/16 23:12:58 2024-04-16 23:12:55,448 p=39746 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:12:55,448 p=39746 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:12:55,668 p=39746 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:12:55,668 p=39746 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:12:55,892 p=39746 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:12:55,892 p=39746 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:12:55,906 p=39746 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:12:55,906 p=39746 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:56,183 p=39746 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:12:56,184 p=39746 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:56,210 p=39746 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:12:56,210 p=39746 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:56,233 p=39746 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:12:56,233 p=39746 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:56,247 p=39746 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:12:56,774 p=39746 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:12:56,774 p=39746 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:12:58,071 p=39746 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Deploy project with labels and selectors] *** 2024-04-16 23:12:58,072 p=39746 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:12:58,160 p=39746 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:12:58,160 p=39746 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 STEP: Verify Application deployment @ 04/16/24 23:12:58.27 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Check project status] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=3  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/04/16 23:13:03 2024-04-16 23:13:00,493 p=39944 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:13:00,493 p=39944 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:00,801 p=39944 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:13:00,801 p=39944 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:01,170 p=39944 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:13:01,171 p=39944 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:01,195 p=39944 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:13:01,196 p=39944 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:01,639 p=39944 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:13:01,639 p=39944 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:01,681 p=39944 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:13:01,681 p=39944 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:01,717 p=39944 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:13:01,717 p=39944 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:01,735 p=39944 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:13:02,460 p=39944 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:13:02,461 p=39944 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:03,582 p=39944 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Check project status] *** 2024-04-16 23:13:03,582 p=39944 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:03,635 p=39944 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:13:03,635 p=39944 u=1009100000 n=ansible | localhost : ok=11 changed=3 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/04/16 23:13:03 {{ } { } []} STEP: Creating backup empty-project-e2e-d7771e25-fc46-11ee-99f6-0a580a8231b5 @ 04/16/24 23:13:03.732 2024/04/16 23:13:03 Wait until backup empty-project-e2e-d7771e25-fc46-11ee-99f6-0a580a8231b5 is completed backup phase: Completed STEP: Verify backup empty-project-e2e-d7771e25-fc46-11ee-99f6-0a580a8231b5 has completed successfully @ 04/16/24 23:13:23.783 2024/04/16 23:13:23 Backup for case empty-project-e2e succeeded STEP: Delete the appplication resources empty-project-e2e @ 04/16/24 23:13:23.871 STEP: Cleanup Application for case empty-project-e2e @ 04/16/24 23:13:23.871 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Remove namespace test-oadp-97] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/04/16 23:13:37 2024-04-16 23:13:25,277 p=40128 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:13:25,278 p=40128 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:25,502 p=40128 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:13:25,503 p=40128 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:25,733 p=40128 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:13:25,733 p=40128 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:25,748 p=40128 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:13:25,748 p=40128 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:26,036 p=40128 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:13:26,036 p=40128 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:26,063 p=40128 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:13:26,063 p=40128 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:26,086 p=40128 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:13:26,086 p=40128 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:26,096 p=40128 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:13:26,606 p=40128 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:13:26,606 p=40128 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:37,452 p=40128 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Remove namespace test-oadp-97] *** 2024-04-16 23:13:37,453 p=40128 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:37,523 p=40128 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:13:37,523 p=40128 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/04/16 23:13:37 Creating restore empty-project-e2e-d7771e25-fc46-11ee-99f6-0a580a8231b5 for case empty-project-e2e-d7771e25-fc46-11ee-99f6-0a580a8231b5 STEP: Create restore empty-project-e2e-d7771e25-fc46-11ee-99f6-0a580a8231b5 from backup empty-project-e2e-d7771e25-fc46-11ee-99f6-0a580a8231b5 @ 04/16/24 23:13:37.568 2024/04/16 23:13:37 Wait until restore empty-project-e2e-d7771e25-fc46-11ee-99f6-0a580a8231b5 is complete restore phase: InProgress restore phase: Completed 2024/04/16 23:13:57 No PodVolumeBackup CR found for the Restore STEP: Verify restore empty-project-e2e-d7771e25-fc46-11ee-99f6-0a580a8231b5has completed successfully @ 04/16/24 23:13:57.691 STEP: Verify Application restore @ 04/16/24 23:13:57.707 STEP: Verify Application deployment for case empty-project-e2e @ 04/16/24 23:13:57.707 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Check project status] *** ok: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=3  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/04/16 23:14:01 2024-04-16 23:13:59,075 p=40326 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:13:59,075 p=40326 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:59,293 p=40326 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:13:59,294 p=40326 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:59,512 p=40326 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:13:59,512 p=40326 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:13:59,526 p=40326 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:13:59,526 p=40326 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:59,799 p=40326 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:13:59,800 p=40326 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:59,826 p=40326 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:13:59,826 p=40326 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:59,849 p=40326 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:13:59,849 p=40326 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:13:59,859 p=40326 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:14:00,370 p=40326 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:14:00,370 p=40326 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:14:01,157 p=40326 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Check project status] *** 2024-04-16 23:14:01,157 p=40326 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:14:01,193 p=40326 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:14:01,193 p=40326 u=1009100000 n=ansible | localhost : ok=11 changed=3 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/04/16 23:14:01 Using Must-gather image: registry.redhat.io/oadp/oadp-mustgather-rhel9:1.3.1 2024/04/16 23:14:01 Cleaning app [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Found variable using reserved name: namespace PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [include_vars] ************************************************************ ok: [localhost] TASK [Remove all the contents from the file] *********************************** changed: [localhost] TASK [Get cluster endpoint] **************************************************** changed: [localhost] TASK [Get current admin token] ************************************************* changed: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] TASK [Extract kubernetes minor version from cluster_info] ********************** ok: [localhost] TASK [Map kubernetes minor to ocp releases] ************************************ ok: [localhost] TASK [set_fact] **************************************************************** ok: [localhost] PLAY [Execute Task] ************************************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Remove namespace test-oadp-97] *** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=11  changed=4  unreachable=0 failed=0 skipped=3  rescued=0 ignored=0 2024/04/16 23:14:15 2024-04-16 23:14:02,710 p=40523 u=1009100000 n=ansible | TASK [Remove all the contents from the file] *********************************** 2024-04-16 23:14:02,710 p=40523 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:14:02,937 p=40523 u=1009100000 n=ansible | TASK [Get cluster endpoint] **************************************************** 2024-04-16 23:14:02,937 p=40523 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:14:03,165 p=40523 u=1009100000 n=ansible | TASK [Get current admin token] ************************************************* 2024-04-16 23:14:03,165 p=40523 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:14:03,180 p=40523 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:14:03,180 p=40523 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:14:03,459 p=40523 u=1009100000 n=ansible | TASK [Extract kubernetes minor version from cluster_info] ********************** 2024-04-16 23:14:03,459 p=40523 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:14:03,484 p=40523 u=1009100000 n=ansible | TASK [Map kubernetes minor to ocp releases] ************************************ 2024-04-16 23:14:03,484 p=40523 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:14:03,506 p=40523 u=1009100000 n=ansible | TASK [set_fact] **************************************************************** 2024-04-16 23:14:03,506 p=40523 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:14:03,516 p=40523 u=1009100000 n=ansible | PLAY [Execute Task] ************************************************************ 2024-04-16 23:14:04,069 p=40523 u=1009100000 n=ansible | TASK [Gathering Facts] ********************************************************* 2024-04-16 23:14:04,070 p=40523 u=1009100000 n=ansible | ok: [localhost] 2024-04-16 23:14:14,943 p=40523 u=1009100000 n=ansible | TASK [/alabama/cspi/sample-applications/ocpdeployer/ansible/roles/ocp-project : Remove namespace test-oadp-97] *** 2024-04-16 23:14:14,943 p=40523 u=1009100000 n=ansible | changed: [localhost] 2024-04-16 23:14:15,013 p=40523 u=1009100000 n=ansible | PLAY RECAP ********************************************************************* 2024-04-16 23:14:15,013 p=40523 u=1009100000 n=ansible | localhost : ok=11 changed=4 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 2024/04/16 23:14:15 Cleaning setup resources for the backup • [86.319 seconds] ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ------------------------------ [SynchronizedAfterSuite]  /alabama/cspi/e2e/e2e_suite_test.go:170 2024/04/16 23:14:15 Deleting Velero CR [SynchronizedAfterSuite] PASSED [0.032 seconds] ------------------------------ [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report autogenerated by Ginkgo [ReportAfterSuite] PASSED [0.014 seconds] ------------------------------ Summarizing 2 Failures: [FAIL] [datamover] DataMover: Backup/Restore stateful application with CSI  [It] [tc-id:OADP-439][interop] MySQL application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 [FAIL] [datamover] DataMover: Backup/Restore stateful application with CSI  [It] [tc-id:OADP-440][interop] [bug-id:OADP-2742] Cassandra application /alabama/cspi/e2e/app_backup/backup_restore_datamover.go:72 Ran 10 of 184 Specs in 3432.878 seconds FAIL! -- 8 Passed | 2 Failed | 0 Pending | 174 Skipped --- FAIL: TestOADPE2E (3432.93s) FAIL Ginkgo ran 1 suite in 58m31.656466401s Test Suite Failed Copying /alabama/cspi/e2e/junit_report.xml to /logs/artifacts/junit_oadp_interop_results.xml... Copying /alabama/cspi/e2e/logs to /logs/artifacts...