Cloning into '/tmp/tempo-tests'... Switched to a new branch 'downstream-release' test -s /tmp/tempo-tests/bin/controller-gen-v0.9.2 || @[ -f /tmp/tempo-tests/bin/controller-gen-v0.9.2 ] || { set -e ; TMP_DIR=$(mktemp -d) ; cd $TMP_DIR ; go mod init tmp ; echo "Downloading sigs.k8s.io/controller-tools/cmd/controller-gen" ; go get -d sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.2 ; GOBIN=/tmp/tempo-tests/bin go install -mod=mod sigs.k8s.io/controller-tools/cmd/controller-gen ; APP=$(echo "/tmp/tempo-tests/bin/controller-gen") ; APP_NAME=$(echo "$APP-v0.9.2") ; mv "$APP" "$APP_NAME" ; rm -rf $TMP_DIR ; } bash: line 1: @[: command not found go: creating new go.mod: module tmp Downloading sigs.k8s.io/controller-tools/cmd/controller-gen go: downloading sigs.k8s.io/controller-tools v0.9.2 go: downloading github.com/spf13/cobra v1.4.0 go: downloading golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/fatih/color v1.12.0 go: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b go: downloading k8s.io/apimachinery v0.24.0 go: downloading k8s.io/apiextensions-apiserver v0.24.0 go: downloading sigs.k8s.io/yaml v1.3.0 go: downloading k8s.io/api v0.24.0 go: downloading github.com/gobuffalo/flect v0.2.5 go: downloading github.com/mattn/go-colorable v0.1.8 go: downloading github.com/mattn/go-isatty v0.0.12 go: downloading github.com/inconshreveable/mousetrap v1.0.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading golang.org/x/sys v0.0.0-20220209214540-3681064d5158 go: downloading k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading k8s.io/klog/v2 v2.60.1 go: downloading github.com/google/gofuzz v1.1.0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.1 go: downloading golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd go: downloading sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 go: downloading github.com/go-logr/logr v1.2.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading golang.org/x/text v0.3.7 go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 go: added github.com/fatih/color v1.12.0 go: added github.com/go-logr/logr v1.2.0 go: added github.com/gobuffalo/flect v0.2.5 go: added github.com/gogo/protobuf v1.3.2 go: added github.com/google/gofuzz v1.1.0 go: added github.com/inconshreveable/mousetrap v1.0.0 go: added github.com/json-iterator/go v1.1.12 go: added github.com/mattn/go-colorable v0.1.8 go: added github.com/mattn/go-isatty v0.0.12 go: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: added github.com/modern-go/reflect2 v1.0.2 go: added github.com/spf13/cobra v1.4.0 go: added github.com/spf13/pflag v1.0.5 go: added golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 go: added golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd go: added golang.org/x/sys v0.0.0-20220209214540-3681064d5158 go: added golang.org/x/text v0.3.7 go: added golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717 go: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 go: added gopkg.in/inf.v0 v0.9.1 go: added gopkg.in/yaml.v2 v2.4.0 go: added gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b go: added k8s.io/api v0.24.0 go: added k8s.io/apiextensions-apiserver v0.24.0 go: added k8s.io/apimachinery v0.24.0 go: added k8s.io/klog/v2 v2.60.1 go: added k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 go: added sigs.k8s.io/controller-tools v0.9.2 go: added sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 go: added sigs.k8s.io/structured-merge-diff/v4 v4.2.1 go: added sigs.k8s.io/yaml v1.3.0 /tmp/tempo-tests/bin/controller-gen-v0.9.2 object:headerFile="hack/boilerplate.go.txt" paths="./..." go fmt ./... CGO_ENABLED=0 go build -o bin/manager -ldflags "-X github.com/grafana/tempo-operator/internal/version.buildDate=2023-12-11T06:48:20Z -X github.com/grafana/tempo-operator/internal/version.revision=599381998f7e4543f62545adac9e1822f22a3ff7 -X github.com/grafana/tempo-operator/internal/version.operatorVersion=0.6.0 -X github.com/grafana/tempo-operator/internal/version.tempoVersion=2.3.0 -X github.com/grafana/tempo-operator/internal/version.tempoQueryVersion=2.3.0" main.go 2023/12/11 06:48:37 kutt-test config testdirs is overridden with args: [ tests/e2e, tests/e2e-openshift ] === RUN kuttl harness.go:462: starting setup harness.go:252: running tests using configured kubeconfig. harness.go:275: Successful connection to cluster at: https://api.ci-op-411hthrh-1d552.cspilp.interop.ccitredhat.com:6443 2023/12/11 06:48:37 Namespace:/minio created 2023/12/11 06:48:37 PersistentVolumeClaim:minio/minio created 2023/12/11 06:48:37 Deployment:minio/minio created 2023/12/11 06:48:37 Service:minio/minio created 2023/12/11 06:48:37 Secret:minio/minio-test created harness.go:360: running tests harness.go:73: going to run test suite with timeout of 180 seconds for each step harness.go:372: testsuite: tests/e2e has 7 tests harness.go:73: going to run test suite with timeout of 180 seconds for each step harness.go:372: testsuite: tests/e2e-openshift has 4 tests === RUN kuttl/harness === RUN kuttl/harness/compatibility === PAUSE kuttl/harness/compatibility === RUN kuttl/harness/custom-ca === PAUSE kuttl/harness/custom-ca === RUN kuttl/harness/gateway === PAUSE kuttl/harness/gateway === RUN kuttl/harness/generate === PAUSE kuttl/harness/generate === RUN kuttl/harness/receivers-mtls === PAUSE kuttl/harness/receivers-mtls === RUN kuttl/harness/receivers-tls === PAUSE kuttl/harness/receivers-tls === RUN kuttl/harness/reconcile === PAUSE kuttl/harness/reconcile === RUN kuttl/harness/monitoring === PAUSE kuttl/harness/monitoring === RUN kuttl/harness/multitenancy === PAUSE kuttl/harness/multitenancy === RUN kuttl/harness/red-metrics === PAUSE kuttl/harness/red-metrics === RUN kuttl/harness/route === PAUSE kuttl/harness/route === CONT kuttl/harness/compatibility === CONT kuttl/harness/reconcile === CONT kuttl/harness/generate === CONT kuttl/harness/gateway === CONT kuttl/harness/generate logger.go:42: 06:48:37 | generate | Ignoring .gitignore as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$ logger.go:42: 06:48:37 | generate | Ignoring config.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$ logger.go:42: 06:48:37 | generate | Ignoring cr.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$ === CONT kuttl/harness/receivers-tls === CONT kuttl/harness/generate logger.go:42: 06:48:37 | generate | Creating namespace: kuttl-test-equal-fawn === CONT kuttl/harness/reconcile logger.go:42: 06:48:37 | reconcile | Creating namespace: kuttl-test-central-mackerel === CONT kuttl/harness/gateway logger.go:42: 06:48:37 | gateway | Creating namespace: kuttl-test-renewed-tadpole === CONT kuttl/harness/receivers-tls logger.go:42: 06:48:37 | receivers-tls | Creating namespace: kuttl-test-brave-oryx === CONT kuttl/harness/compatibility logger.go:42: 06:48:37 | compatibility | Creating namespace: kuttl-test-picked-rooster === CONT kuttl/harness/generate logger.go:42: 06:48:37 | generate/0-storage-secret | starting test step 0-storage-secret === CONT kuttl/harness/reconcile logger.go:42: 06:48:37 | reconcile/1-install | starting test step 1-install === CONT kuttl/harness/receivers-tls logger.go:42: 06:48:37 | receivers-tls/0-install-storage | starting test step 0-install-storage === CONT kuttl/harness/compatibility logger.go:42: 06:48:37 | compatibility/0-install-storage | starting test step 0-install-storage === CONT kuttl/harness/gateway logger.go:42: 06:48:37 | gateway/0-install | starting test step 0-install === CONT kuttl/harness/generate logger.go:42: 06:48:37 | generate/0-storage-secret | Secret:kuttl-test-equal-fawn/minio-test created logger.go:42: 06:48:37 | generate/0-storage-secret | test step completed 0-storage-secret logger.go:42: 06:48:37 | generate/1-generate | starting test step 1-generate logger.go:42: 06:48:37 | generate/1-generate | running command: [/bin/sh -c RELATED_IMAGE_TEMPO=docker.io/grafana/tempo:2.2.1 RELATED_IMAGE_TEMPO_QUERY=docker.io/grafana/tempo-query:2.2.1 RELATED_IMAGE_TEMPO_GATEWAY=quay.io/observatorium/api:main-2023-09-13-14e06c6 RELATED_IMAGE_TEMPO_GATEWAY_OPA=quay.io/observatorium/opa-openshift:main-2023-05-24-8e91537 ../../../bin/manager generate --config config.yaml --cr cr.yaml --output generated.yaml] logger.go:42: 06:48:37 | generate/1-generate | running command: [kubectl apply -f generated.yaml --namespace kuttl-test-equal-fawn] === CONT kuttl/harness/gateway logger.go:42: 06:48:37 | gateway/0-install | ConfigMap:kuttl-test-renewed-tadpole/dex-config created === CONT kuttl/harness/reconcile logger.go:42: 06:48:37 | reconcile/1-install | Secret:kuttl-test-central-mackerel/minio-test created === CONT kuttl/harness/receivers-tls logger.go:42: 06:48:37 | receivers-tls/0-install-storage | PersistentVolumeClaim:kuttl-test-brave-oryx/minio created === CONT kuttl/harness/compatibility logger.go:42: 06:48:37 | compatibility/0-install-storage | PersistentVolumeClaim:kuttl-test-picked-rooster/minio created === CONT kuttl/harness/receivers-tls logger.go:42: 06:48:37 | receivers-tls/0-install-storage | Deployment:kuttl-test-brave-oryx/minio created === CONT kuttl/harness/reconcile logger.go:42: 06:48:37 | reconcile/1-install | TempoStack:kuttl-test-central-mackerel/simplest created === CONT kuttl/harness/compatibility logger.go:42: 06:48:37 | compatibility/0-install-storage | Deployment:kuttl-test-picked-rooster/minio created === CONT kuttl/harness/receivers-tls logger.go:42: 06:48:37 | receivers-tls/0-install-storage | Service:kuttl-test-brave-oryx/minio created logger.go:42: 06:48:37 | receivers-tls/0-install-storage | Secret:kuttl-test-brave-oryx/minio created === CONT kuttl/harness/compatibility logger.go:42: 06:48:37 | compatibility/0-install-storage | Service:kuttl-test-picked-rooster/minio created logger.go:42: 06:48:38 | compatibility/0-install-storage | Secret:kuttl-test-picked-rooster/minio created === CONT kuttl/harness/gateway logger.go:42: 06:48:38 | gateway/0-install | Deployment:kuttl-test-renewed-tadpole/dex created logger.go:42: 06:48:38 | gateway/0-install | Service:kuttl-test-renewed-tadpole/dex created === CONT kuttl/harness/reconcile logger.go:42: 06:48:39 | reconcile/1-install | test step completed 1-install logger.go:42: 06:48:39 | reconcile/2-delete-svc | starting test step 2-delete-svc logger.go:42: 06:48:39 | reconcile/2-delete-svc | running command: [kubectl delete service tempo-simplest-querier --namespace kuttl-test-central-mackerel] logger.go:42: 06:48:40 | reconcile/2-delete-svc | service "tempo-simplest-querier" deleted === CONT kuttl/harness/generate logger.go:42: 06:48:40 | generate/1-generate | configmap/tempo-generated-tempo created === CONT kuttl/harness/reconcile logger.go:42: 06:48:40 | reconcile/2-delete-svc | test step completed 2-delete-svc logger.go:42: 06:48:40 | reconcile/3-update-storage-secret | starting test step 3-update-storage-secret === CONT kuttl/harness/generate logger.go:42: 06:48:40 | generate/1-generate | serviceaccount/tempo-generated-tempo created logger.go:42: 06:48:40 | generate/1-generate | deployment.apps/tempo-generated-tempo-distributor created === CONT kuttl/harness/reconcile logger.go:42: 06:48:40 | reconcile/3-update-storage-secret | Secret:kuttl-test-central-mackerel/minio-test updated === CONT kuttl/harness/generate logger.go:42: 06:48:40 | generate/1-generate | service/tempo-generated-tempo-distributor created logger.go:42: 06:48:40 | generate/1-generate | statefulset.apps/tempo-generated-tempo-ingester created logger.go:42: 06:48:40 | generate/1-generate | service/tempo-generated-tempo-ingester created logger.go:42: 06:48:40 | generate/1-generate | service/tempo-generated-tempo-gossip-ring created logger.go:42: 06:48:40 | generate/1-generate | deployment.apps/tempo-generated-tempo-query-frontend created logger.go:42: 06:48:40 | generate/1-generate | service/tempo-generated-tempo-query-frontend created logger.go:42: 06:48:40 | generate/1-generate | service/tempo-generated-tempo-query-frontend-discovery created logger.go:42: 06:48:40 | generate/1-generate | deployment.apps/tempo-generated-tempo-querier created logger.go:42: 06:48:40 | generate/1-generate | service/tempo-generated-tempo-querier created logger.go:42: 06:48:40 | generate/1-generate | deployment.apps/tempo-generated-tempo-compactor created logger.go:42: 06:48:40 | generate/1-generate | service/tempo-generated-tempo-compactor created === CONT kuttl/harness/gateway logger.go:42: 06:48:43 | gateway/0-install | test step completed 0-install logger.go:42: 06:48:43 | gateway/1-install | starting test step 1-install logger.go:42: 06:48:43 | gateway/1-install | Secret:kuttl-test-renewed-tadpole/minio-test created logger.go:42: 06:48:43 | gateway/1-install | Secret:kuttl-test-renewed-tadpole/oidc-test created logger.go:42: 06:48:43 | gateway/1-install | test step completed 1-install logger.go:42: 06:48:43 | gateway/2-install | starting test step 2-install logger.go:42: 06:48:43 | gateway/2-install | TempoStack:kuttl-test-renewed-tadpole/foo created === CONT kuttl/harness/compatibility logger.go:42: 06:48:48 | compatibility/0-install-storage | test step completed 0-install-storage logger.go:42: 06:48:48 | compatibility/1-install | starting test step 1-install logger.go:42: 06:48:48 | compatibility/1-install | TempoStack:kuttl-test-picked-rooster/simplest created === CONT kuttl/harness/receivers-tls logger.go:42: 06:48:49 | receivers-tls/0-install-storage | test step completed 0-install-storage logger.go:42: 06:48:49 | receivers-tls/1-install-tempo | starting test step 1-install-tempo logger.go:42: 06:48:49 | receivers-tls/1-install-tempo | TempoStack:kuttl-test-brave-oryx/simplest created logger.go:42: 06:48:49 | receivers-tls/1-install-tempo | ConfigMap:kuttl-test-brave-oryx/custom-ca created logger.go:42: 06:48:49 | receivers-tls/1-install-tempo | Secret:kuttl-test-brave-oryx/custom-cert created === CONT kuttl/harness/gateway logger.go:42: 06:48:52 | gateway/2-install | test step completed 2-install logger.go:42: 06:48:52 | gateway | gateway events from ns kuttl-test-renewed-tadpole: logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:38 +0000 UTC Normal Pod dex-7bbf55b5cd-c4rmh Binding Scheduled Successfully assigned kuttl-test-renewed-tadpole/dex-7bbf55b5cd-c4rmh to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:38 +0000 UTC Normal ReplicaSet.apps dex-7bbf55b5cd SuccessfulCreate Created pod: dex-7bbf55b5cd-c4rmh replicaset-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:38 +0000 UTC Normal Deployment.apps dex ScalingReplicaSet Scaled up replica set dex-7bbf55b5cd to 1 deployment-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:39 +0000 UTC Normal Pod dex-7bbf55b5cd-c4rmh AddedInterface Add eth0 [10.129.2.15/23] from ovn-kubernetes logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:39 +0000 UTC Normal Pod dex-7bbf55b5cd-c4rmh.spec.containers{hydra} Pulling Pulling image "ghcr.io/iblancasa/dex:v2.37.0" kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:41 +0000 UTC Normal Pod dex-7bbf55b5cd-c4rmh.spec.containers{hydra} Pulled Successfully pulled image "ghcr.io/iblancasa/dex:v2.37.0" in 2.122s (2.122s including waiting) kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:41 +0000 UTC Normal Pod dex-7bbf55b5cd-c4rmh.spec.containers{hydra} Created Created container hydra kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:41 +0000 UTC Normal Pod dex-7bbf55b5cd-c4rmh.spec.containers{hydra} Started Started container hydra kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal PersistentVolumeClaim data-tempo-foo-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal PersistentVolumeClaim data-tempo-foo-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-renewed-tadpole/data-tempo-foo-ingester-0" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal PersistentVolumeClaim data-tempo-foo-ingester-0 ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal Pod tempo-foo-distributor-8fcd7c584-mlcck Binding Scheduled Successfully assigned kuttl-test-renewed-tadpole/tempo-foo-distributor-8fcd7c584-mlcck to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal ReplicaSet.apps tempo-foo-distributor-8fcd7c584 SuccessfulCreate Created pod: tempo-foo-distributor-8fcd7c584-mlcck replicaset-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal Deployment.apps tempo-foo-distributor ScalingReplicaSet Scaled up replica set tempo-foo-distributor-8fcd7c584 to 1 deployment-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal StatefulSet.apps tempo-foo-ingester SuccessfulCreate create Claim data-tempo-foo-ingester-0 Pod tempo-foo-ingester-0 in StatefulSet tempo-foo-ingester success statefulset-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal StatefulSet.apps tempo-foo-ingester SuccessfulCreate create Pod tempo-foo-ingester-0 in StatefulSet tempo-foo-ingester successful statefulset-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal Pod tempo-foo-querier-559f84f87-vgdks Binding Scheduled Successfully assigned kuttl-test-renewed-tadpole/tempo-foo-querier-559f84f87-vgdks to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal ReplicaSet.apps tempo-foo-querier-559f84f87 SuccessfulCreate Created pod: tempo-foo-querier-559f84f87-vgdks replicaset-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal Deployment.apps tempo-foo-querier ScalingReplicaSet Scaled up replica set tempo-foo-querier-559f84f87 to 1 deployment-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz Binding Scheduled Successfully assigned kuttl-test-renewed-tadpole/tempo-foo-query-frontend-66647d97d8-k6trz to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal ReplicaSet.apps tempo-foo-query-frontend-66647d97d8 SuccessfulCreate Created pod: tempo-foo-query-frontend-66647d97d8-k6trz replicaset-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:44 +0000 UTC Normal Deployment.apps tempo-foo-query-frontend ScalingReplicaSet Scaled up replica set tempo-foo-query-frontend-66647d97d8 to 1 deployment-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-compactor-54db8cf464-jxwf4 Binding Scheduled Successfully assigned kuttl-test-renewed-tadpole/tempo-foo-compactor-54db8cf464-jxwf4 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-compactor-54db8cf464-jxwf4 AddedInterface Add eth0 [10.129.2.24/23] from ovn-kubernetes logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-compactor-54db8cf464-jxwf4.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal ReplicaSet.apps tempo-foo-compactor-54db8cf464 SuccessfulCreate Created pod: tempo-foo-compactor-54db8cf464-jxwf4 replicaset-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Deployment.apps tempo-foo-compactor ScalingReplicaSet Scaled up replica set tempo-foo-compactor-54db8cf464 to 1 deployment-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-distributor-8fcd7c584-mlcck AddedInterface Add eth0 [10.129.2.23/23] from ovn-kubernetes logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-distributor-8fcd7c584-mlcck.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-gateway-85c7d95f76-fwrqp Binding Scheduled Successfully assigned kuttl-test-renewed-tadpole/tempo-foo-gateway-85c7d95f76-fwrqp to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-gateway-85c7d95f76-fwrqp AddedInterface Add eth0 [10.131.0.25/23] from ovn-kubernetes logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-gateway-85c7d95f76-fwrqp.spec.containers{tempo-gateway} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:131f5ec8d1b290ee2b485eb57d5170fb46d66d5dfd7c8e52c5e346648ab2ae9e" kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal ReplicaSet.apps tempo-foo-gateway-85c7d95f76 SuccessfulCreate Created pod: tempo-foo-gateway-85c7d95f76-fwrqp replicaset-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Deployment.apps tempo-foo-gateway ScalingReplicaSet Scaled up replica set tempo-foo-gateway-85c7d95f76 to 1 deployment-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-querier-559f84f87-vgdks AddedInterface Add eth0 [10.131.0.24/23] from ovn-kubernetes logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-querier-559f84f87-vgdks.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz AddedInterface Add eth0 [10.128.2.24/23] from ovn-kubernetes logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-foo-compactor-54db8cf464-jxwf4.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 2.158s (2.158s including waiting) kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 2.144s (2.144s including waiting) kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz.spec.containers{tempo-query} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:48 +0000 UTC Normal PersistentVolumeClaim data-tempo-foo-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-c2288baf-5512-43cf-b972-aa71648ebae1 ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-foo-compactor-54db8cf464-jxwf4.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-foo-compactor-54db8cf464-jxwf4.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-foo-distributor-8fcd7c584-mlcck.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 2.881s (2.881s including waiting) kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-foo-distributor-8fcd7c584-mlcck.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-foo-distributor-8fcd7c584-mlcck.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-foo-ingester-0 Binding Scheduled Successfully assigned kuttl-test-renewed-tadpole/tempo-foo-ingester-0 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-foo-querier-559f84f87-vgdks.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 4.89s (4.89s including waiting) kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-foo-querier-559f84f87-vgdks.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-foo-querier-559f84f87-vgdks.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-foo-gateway-85c7d95f76-fwrqp.spec.containers{tempo-gateway} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:131f5ec8d1b290ee2b485eb57d5170fb46d66d5dfd7c8e52c5e346648ab2ae9e" in 5.749s (5.749s including waiting) kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-foo-gateway-85c7d95f76-fwrqp.spec.containers{tempo-gateway} Created Created container tempo-gateway kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-foo-gateway-85c7d95f76-fwrqp.spec.containers{tempo-gateway} Started Started container tempo-gateway kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-foo-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-c2288baf-5512-43cf-b972-aa71648ebae1" attachdetach-controller logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz.spec.containers{tempo-query} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" in 4.936s (4.936s including waiting) kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz.spec.containers{tempo-query} Created Created container tempo-query kubelet logger.go:42: 06:48:52 | gateway | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-foo-query-frontend-66647d97d8-k6trz.spec.containers{tempo-query} Started Started container tempo-query kubelet logger.go:42: 06:48:53 | gateway | Deleting namespace: kuttl-test-renewed-tadpole === CONT kuttl/harness/generate logger.go:42: 06:49:00 | generate/1-generate | test step completed 1-generate logger.go:42: 06:49:00 | generate | generate events from ns kuttl-test-equal-fawn: logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal PersistentVolumeClaim data-tempo-generated-tempo-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal PersistentVolumeClaim data-tempo-generated-tempo-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-equal-fawn/data-tempo-generated-tempo-ingester-0" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal PersistentVolumeClaim data-tempo-generated-tempo-ingester-0 ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal ReplicaSet.apps tempo-generated-tempo-compactor-5958cc945f SuccessfulCreate Created pod: tempo-generated-tempo-compactor-5958cc945f-cjw65 replicaset-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal Deployment.apps tempo-generated-tempo-compactor ScalingReplicaSet Scaled up replica set tempo-generated-tempo-compactor-5958cc945f to 1 deployment-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-generated-tempo-distributor-867984587b-2gpw6 Binding Scheduled Successfully assigned kuttl-test-equal-fawn/tempo-generated-tempo-distributor-867984587b-2gpw6 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal ReplicaSet.apps tempo-generated-tempo-distributor-867984587b SuccessfulCreate Created pod: tempo-generated-tempo-distributor-867984587b-2gpw6 replicaset-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal Deployment.apps tempo-generated-tempo-distributor ScalingReplicaSet Scaled up replica set tempo-generated-tempo-distributor-867984587b to 1 deployment-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal StatefulSet.apps tempo-generated-tempo-ingester SuccessfulCreate create Claim data-tempo-generated-tempo-ingester-0 Pod tempo-generated-tempo-ingester-0 in StatefulSet tempo-generated-tempo-ingester success statefulset-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal StatefulSet.apps tempo-generated-tempo-ingester SuccessfulCreate create Pod tempo-generated-tempo-ingester-0 in StatefulSet tempo-generated-tempo-ingester successful statefulset-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-generated-tempo-querier-7bf4755975-xfqmx Binding Scheduled Successfully assigned kuttl-test-equal-fawn/tempo-generated-tempo-querier-7bf4755975-xfqmx to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal ReplicaSet.apps tempo-generated-tempo-querier-7bf4755975 SuccessfulCreate Created pod: tempo-generated-tempo-querier-7bf4755975-xfqmx replicaset-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal Deployment.apps tempo-generated-tempo-querier ScalingReplicaSet Scaled up replica set tempo-generated-tempo-querier-7bf4755975 to 1 deployment-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w Binding Scheduled Successfully assigned kuttl-test-equal-fawn/tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal ReplicaSet.apps tempo-generated-tempo-query-frontend-f5dffd85b SuccessfulCreate Created pod: tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w replicaset-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:40 +0000 UTC Normal Deployment.apps tempo-generated-tempo-query-frontend ScalingReplicaSet Scaled up replica set tempo-generated-tempo-query-frontend-f5dffd85b to 1 deployment-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-generated-tempo-compactor-5958cc945f-cjw65 Binding Scheduled Successfully assigned kuttl-test-equal-fawn/tempo-generated-tempo-compactor-5958cc945f-cjw65 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-generated-tempo-compactor-5958cc945f-cjw65 AddedInterface Add eth0 [10.129.2.18/23] from ovn-kubernetes logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-generated-tempo-compactor-5958cc945f-cjw65.spec.containers{tempo} Pulling Pulling image "docker.io/grafana/tempo:2.2.1" kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-generated-tempo-distributor-867984587b-2gpw6 AddedInterface Add eth0 [10.129.2.16/23] from ovn-kubernetes logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-generated-tempo-distributor-867984587b-2gpw6.spec.containers{tempo} Pulling Pulling image "docker.io/grafana/tempo:2.2.1" kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-generated-tempo-querier-7bf4755975-xfqmx AddedInterface Add eth0 [10.128.2.23/23] from ovn-kubernetes logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-generated-tempo-querier-7bf4755975-xfqmx.spec.containers{tempo} Pulling Pulling image "docker.io/grafana/tempo:2.2.1" kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w AddedInterface Add eth0 [10.129.2.17/23] from ovn-kubernetes logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w.spec.containers{tempo} Pulling Pulling image "docker.io/grafana/tempo:2.2.1" kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-compactor-5958cc945f-cjw65.spec.containers{tempo} Pulled Successfully pulled image "docker.io/grafana/tempo:2.2.1" in 1.811s (1.811s including waiting) kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-compactor-5958cc945f-cjw65.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-compactor-5958cc945f-cjw65.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-distributor-867984587b-2gpw6.spec.containers{tempo} Pulled Successfully pulled image "docker.io/grafana/tempo:2.2.1" in 2.217s (2.217s including waiting) kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-distributor-867984587b-2gpw6.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-distributor-867984587b-2gpw6.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-querier-7bf4755975-xfqmx.spec.containers{tempo} Pulled Successfully pulled image "docker.io/grafana/tempo:2.2.1" in 1.764s (1.764s including waiting) kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-querier-7bf4755975-xfqmx.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-querier-7bf4755975-xfqmx.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-querier-7bf4755975-xfqmx.spec.containers{tempo} Pulled Container image "docker.io/grafana/tempo:2.2.1" already present on machine kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w.spec.containers{tempo} Pulled Successfully pulled image "docker.io/grafana/tempo:2.2.1" in 2.184s (2.184s including waiting) kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:44 +0000 UTC Normal PersistentVolumeClaim data-tempo-generated-tempo-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-38ea3e41-f5ff-4189-98f1-6e8d2a5993fe ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:44 +0000 UTC Normal Pod tempo-generated-tempo-compactor-5958cc945f-cjw65.spec.containers{tempo} Pulled Container image "docker.io/grafana/tempo:2.2.1" already present on machine kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:44 +0000 UTC Warning Endpoints tempo-generated-tempo-gossip-ring FailedToUpdateEndpoint Failed to update endpoint kuttl-test-equal-fawn/tempo-generated-tempo-gossip-ring: Operation cannot be fulfilled on endpoints "tempo-generated-tempo-gossip-ring": the object has been modified; please apply your changes to the latest version and try again endpoint-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:44 +0000 UTC Warning Pod tempo-generated-tempo-querier-7bf4755975-xfqmx.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-generated-tempo-querier-7bf4755975-xfqmx_kuttl-test-equal-fawn(b86475c8-85a0-487d-baf0-80fadd116448) kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:44 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w.spec.containers{tempo} Pulled Container image "docker.io/grafana/tempo:2.2.1" already present on machine kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:45 +0000 UTC Warning Pod tempo-generated-tempo-compactor-5958cc945f-cjw65.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-generated-tempo-compactor-5958cc945f-cjw65_kuttl-test-equal-fawn(352e7876-d76f-4aba-b199-5caeac4e4bba) kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:45 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0 Binding Scheduled Successfully assigned kuttl-test-equal-fawn/tempo-generated-tempo-ingester-0 to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:45 +0000 UTC Warning Pod tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-generated-tempo-query-frontend-f5dffd85b-s6m8w_kuttl-test-equal-fawn(f976b71b-3c65-455f-9fb2-b40ef45b8117) kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-38ea3e41-f5ff-4189-98f1-6e8d2a5993fe" attachdetach-controller logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0 AddedInterface Add eth0 [10.128.2.25/23] from ovn-kubernetes logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0.spec.containers{tempo} Pulled Container image "docker.io/grafana/tempo:2.2.1" already present on machine kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:49:00 | generate | 2023-12-11 06:48:50 +0000 UTC Warning Pod tempo-generated-tempo-ingester-0.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-generated-tempo-ingester-0_kuttl-test-equal-fawn(ecc1ef98-078c-4ba3-a4fc-996a60f55ca7) kubelet logger.go:42: 06:49:00 | generate | Deleting namespace: kuttl-test-equal-fawn === CONT kuttl/harness/receivers-mtls logger.go:42: 06:49:28 | receivers-mtls | Creating namespace: kuttl-test-renewing-penguin logger.go:42: 06:49:28 | receivers-mtls/0-install-storage | starting test step 0-install-storage logger.go:42: 06:49:28 | receivers-mtls/0-install-storage | PersistentVolumeClaim:kuttl-test-renewing-penguin/minio created logger.go:42: 06:49:28 | receivers-mtls/0-install-storage | Deployment:kuttl-test-renewing-penguin/minio created logger.go:42: 06:49:28 | receivers-mtls/0-install-storage | Service:kuttl-test-renewing-penguin/minio created logger.go:42: 06:49:28 | receivers-mtls/0-install-storage | Secret:kuttl-test-renewing-penguin/minio created === CONT kuttl/harness/custom-ca logger.go:42: 06:49:37 | custom-ca | Ignoring README.md as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$ logger.go:42: 06:49:37 | custom-ca | Creating namespace: kuttl-test-sensible-shrew logger.go:42: 06:49:37 | custom-ca/0-install-storage | starting test step 0-install-storage logger.go:42: 06:49:37 | custom-ca/0-install-storage | Secret:kuttl-test-sensible-shrew/minio-cert created logger.go:42: 06:49:37 | custom-ca/0-install-storage | Deployment:kuttl-test-sensible-shrew/minio created logger.go:42: 06:49:37 | custom-ca/0-install-storage | Service:kuttl-test-sensible-shrew/minio created logger.go:42: 06:49:37 | custom-ca/0-install-storage | Secret:kuttl-test-sensible-shrew/minio created logger.go:42: 06:49:39 | custom-ca/0-install-storage | test step completed 0-install-storage logger.go:42: 06:49:39 | custom-ca/1-install-tempo | starting test step 1-install-tempo logger.go:42: 06:49:39 | custom-ca/1-install-tempo | ConfigMap:kuttl-test-sensible-shrew/custom-ca created logger.go:42: 06:49:39 | custom-ca/1-install-tempo | TempoStack:kuttl-test-sensible-shrew/simplest created === CONT kuttl/harness/receivers-mtls logger.go:42: 06:49:42 | receivers-mtls/0-install-storage | test step completed 0-install-storage logger.go:42: 06:49:42 | receivers-mtls/1-install-tempo | starting test step 1-install-tempo logger.go:42: 06:49:42 | receivers-mtls/1-install-tempo | TempoStack:kuttl-test-renewing-penguin/simplest created logger.go:42: 06:49:42 | receivers-mtls/1-install-tempo | ConfigMap:kuttl-test-renewing-penguin/custom-ca created logger.go:42: 06:49:42 | receivers-mtls/1-install-tempo | Secret:kuttl-test-renewing-penguin/custom-cert created === CONT kuttl/harness/compatibility logger.go:42: 06:50:00 | compatibility/1-install | test step completed 1-install logger.go:42: 06:50:00 | compatibility/2- | starting test step 2- logger.go:42: 06:50:00 | compatibility/2- | running command: [/bin/sh -c kubectl get --namespace kuttl-test-picked-rooster tempo simplest -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' | grep True] logger.go:42: 06:50:00 | compatibility/2- | True logger.go:42: 06:50:00 | compatibility/2- | test step completed 2- logger.go:42: 06:50:00 | compatibility/3-generate-traces | starting test step 3-generate-traces === CONT kuttl/harness/reconcile logger.go:42: 06:50:00 | reconcile/3-update-storage-secret | test step completed 3-update-storage-secret logger.go:42: 06:50:00 | reconcile/4-check-bucket-name | starting test step 4-check-bucket-name logger.go:42: 06:50:00 | reconcile/4-check-bucket-name | running command: [/bin/sh -c kubectl get --namespace kuttl-test-central-mackerel configmap tempo-simplest -o jsonpath="{.data.tempo\\.yaml}" | grep "endpoint: minio.minio.svc.cluster.local"] === CONT kuttl/harness/compatibility logger.go:42: 06:50:00 | compatibility/3-generate-traces | Job:kuttl-test-picked-rooster/generate-traces created === CONT kuttl/harness/reconcile logger.go:42: 06:50:00 | reconcile/4-check-bucket-name | endpoint: minio.minio.svc.cluster.local:9000 logger.go:42: 06:50:00 | reconcile/4-check-bucket-name | test step completed 4-check-bucket-name logger.go:42: 06:50:00 | reconcile/5-disable-reconciliation | starting test step 5-disable-reconciliation logger.go:42: 06:50:00 | reconcile/5-disable-reconciliation | ServiceAccount:kuttl-test-central-mackerel/fake-sa created logger.go:42: 06:50:01 | reconcile/5-disable-reconciliation | TempoStack:kuttl-test-central-mackerel/simplest updated logger.go:42: 06:50:01 | reconcile/5-disable-reconciliation | test step completed 5-disable-reconciliation logger.go:42: 06:50:01 | reconcile/6-check-config | starting test step 6-check-config logger.go:42: 06:50:01 | reconcile/6-check-config | running command: [/bin/sh -c ! kubectl get --namespace kuttl-test-central-mackerel configmap tempo-simplest -o jsonpath='{.data.tempo\.yaml}' | grep '[[:blank:]]$'] logger.go:42: 06:50:01 | reconcile/6-check-config | test step completed 6-check-config logger.go:42: 06:50:01 | reconcile | reconcile events from ns kuttl-test-central-mackerel: logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:38 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:38 +0000 UTC Normal Pod tempo-simplest-distributor-65777488c5-ghbc9 Binding Scheduled Successfully assigned kuttl-test-central-mackerel/tempo-simplest-distributor-65777488c5-ghbc9 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:38 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-65777488c5 SuccessfulCreate Created pod: tempo-simplest-distributor-65777488c5-ghbc9 replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:38 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-65777488c5 to 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:38 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Claim data-tempo-simplest-ingester-0 Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester success statefulset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:38 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful statefulset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:38 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-77bb4b6d4b to 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-central-mackerel/data-tempo-simplest-ingester-0" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-compactor-6bffc9b57f-6hv2k Binding Scheduled Successfully assigned kuttl-test-central-mackerel/tempo-simplest-compactor-6bffc9b57f-6hv2k to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-compactor-6bffc9b57f-6hv2k AddedInterface Add eth0 [10.131.0.20/23] from ovn-kubernetes logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-compactor-6bffc9b57f-6hv2k.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-6bffc9b57f SuccessfulCreate Created pod: tempo-simplest-compactor-6bffc9b57f-6hv2k replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-6bffc9b57f to 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-distributor-65777488c5-ghbc9 AddedInterface Add eth0 [10.129.2.14/23] from ovn-kubernetes logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-distributor-65777488c5-ghbc9.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-querier-74f575d6dd-gtgwg Binding Scheduled Successfully assigned kuttl-test-central-mackerel/tempo-simplest-querier-74f575d6dd-gtgwg to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-querier-74f575d6dd-gtgwg AddedInterface Add eth0 [10.128.2.21/23] from ovn-kubernetes logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-querier-74f575d6dd-gtgwg.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-74f575d6dd SuccessfulCreate Created pod: tempo-simplest-querier-74f575d6dd-gtgwg replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-74f575d6dd to 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-query-frontend-77bb4b6d4b-vdnjj Binding Scheduled Successfully assigned kuttl-test-central-mackerel/tempo-simplest-query-frontend-77bb4b6d4b-vdnjj to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-query-frontend-77bb4b6d4b-vdnjj AddedInterface Add eth0 [10.128.2.20/23] from ovn-kubernetes logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal Pod tempo-simplest-query-frontend-77bb4b6d4b-vdnjj.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:39 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-77bb4b6d4b SuccessfulCreate Created pod: tempo-simplest-query-frontend-77bb4b6d4b-vdnjj replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-simplest-compactor-5469bbf5c7-scqvs Binding Scheduled Successfully assigned kuttl-test-central-mackerel/tempo-simplest-compactor-5469bbf5c7-scqvs to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-simplest-compactor-5469bbf5c7-scqvs AddedInterface Add eth0 [10.128.2.22/23] from ovn-kubernetes logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-5469bbf5c7 SuccessfulCreate Created pod: tempo-simplest-compactor-5469bbf5c7-scqvs replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-5469bbf5c7 to 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-simplest-distributor-65f558bbd5-qhd5r Binding Scheduled Successfully assigned kuttl-test-central-mackerel/tempo-simplest-distributor-65f558bbd5-qhd5r to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-65f558bbd5 SuccessfulCreate Created pod: tempo-simplest-distributor-65f558bbd5-qhd5r replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-65f558bbd5 to 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulDelete delete Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful statefulset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-simplest-querier-565d76d4f4-8zsc2 Binding Scheduled Successfully assigned kuttl-test-central-mackerel/tempo-simplest-querier-565d76d4f4-8zsc2 to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-simplest-querier-565d76d4f4-8zsc2 AddedInterface Add eth0 [10.131.0.23/23] from ovn-kubernetes logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-simplest-querier-565d76d4f4-8zsc2.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-565d76d4f4 SuccessfulCreate Created pod: tempo-simplest-querier-565d76d4f4-8zsc2 replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-565d76d4f4 to 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Pod tempo-simplest-query-frontend-66489857cb-w5sfh Binding Scheduled Successfully assigned kuttl-test-central-mackerel/tempo-simplest-query-frontend-66489857cb-w5sfh to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-66489857cb SuccessfulCreate Created pod: tempo-simplest-query-frontend-66489857cb-w5sfh replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:40 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-66489857cb to 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-simplest-compactor-5469bbf5c7-scqvs.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-simplest-distributor-65f558bbd5-qhd5r AddedInterface Add eth0 [10.131.0.21/23] from ovn-kubernetes logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-simplest-distributor-65f558bbd5-qhd5r.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-simplest-query-frontend-66489857cb-w5sfh AddedInterface Add eth0 [10.131.0.22/23] from ovn-kubernetes logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:41 +0000 UTC Normal Pod tempo-simplest-query-frontend-66489857cb-w5sfh.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:43 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-0fdbc4fd-6fe3-4574-a154-dd59bad4c3dd ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:43 +0000 UTC Normal Pod tempo-simplest-ingester-0 Binding Scheduled Successfully assigned kuttl-test-central-mackerel/tempo-simplest-ingester-0 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:44 +0000 UTC Warning Pod tempo-simplest-ingester-0 Scheduling FailedScheduling running Bind plugin "DefaultBinder": Operation cannot be fulfilled on pods "tempo-simplest-ingester-0": StorageError: invalid object, Code: 4, Key: /kubernetes.io/pods/kuttl-test-central-mackerel/tempo-simplest-ingester-0, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9a25fe07-9389-448f-9551-0393040e0afc, UID in object meta: 627c7256-2b2a-4184-8b8c-e71d4e02caed default-scheduler logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-compactor-5469bbf5c7-scqvs.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 6.448s (6.448s including waiting) kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-compactor-5469bbf5c7-scqvs.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-compactor-5469bbf5c7-scqvs.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-distributor-65777488c5-ghbc9.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 8.288s (8.288s including waiting) kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-0fdbc4fd-6fe3-4574-a154-dd59bad4c3dd" attachdetach-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-ingester-0 AddedInterface Add eth0 [10.129.2.22/23] from ovn-kubernetes logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-querier-74f575d6dd-gtgwg.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 7.914s (7.914s including waiting) kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-querier-74f575d6dd-gtgwg.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:47 +0000 UTC Normal Pod tempo-simplest-querier-74f575d6dd-gtgwg.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-simplest-compactor-5469bbf5c7-scqvs.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-simplest-distributor-65777488c5-ghbc9.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-simplest-distributor-65777488c5-ghbc9.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-simplest-querier-74f575d6dd-gtgwg.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-simplest-query-frontend-77bb4b6d4b-vdnjj.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 9.391s (9.391s including waiting) kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-simplest-query-frontend-77bb4b6d4b-vdnjj.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:48 +0000 UTC Normal Pod tempo-simplest-query-frontend-77bb4b6d4b-vdnjj.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-compactor-6bffc9b57f-6hv2k.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 10.938s (10.938s including waiting) kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-compactor-6bffc9b57f-6hv2k.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-compactor-6bffc9b57f-6hv2k.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-distributor-65f558bbd5-qhd5r.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 9.382s (9.382s including waiting) kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-distributor-65f558bbd5-qhd5r.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-distributor-65f558bbd5-qhd5r.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-querier-565d76d4f4-8zsc2.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 9.621s (9.621s including waiting) kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-querier-565d76d4f4-8zsc2.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-querier-565d76d4f4-8zsc2.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-query-frontend-66489857cb-w5sfh.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" in 9.377s (9.377s including waiting) kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-query-frontend-66489857cb-w5sfh.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-query-frontend-66489857cb-w5sfh.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:07 +0000 UTC Warning Pod tempo-simplest-ingester-0.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:09 +0000 UTC Warning Pod tempo-simplest-compactor-6bffc9b57f-6hv2k.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:10 +0000 UTC Warning Pod tempo-simplest-compactor-5469bbf5c7-scqvs.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:10 +0000 UTC Normal Pod tempo-simplest-querier-74f575d6dd-gtgwg.spec.containers{tempo} Killing Stopping container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:10 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-74f575d6dd SuccessfulDelete Deleted pod: tempo-simplest-querier-74f575d6dd-gtgwg replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:10 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled down replica set tempo-simplest-querier-74f575d6dd to 0 from 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:11 +0000 UTC Normal Pod tempo-simplest-distributor-65777488c5-ghbc9.spec.containers{tempo} Killing Stopping container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:11 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-65777488c5 SuccessfulDelete Deleted pod: tempo-simplest-distributor-65777488c5-ghbc9 replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:11 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled down replica set tempo-simplest-distributor-65777488c5 to 0 from 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:11 +0000 UTC Normal Pod tempo-simplest-query-frontend-77bb4b6d4b-vdnjj.spec.containers{tempo} Killing Stopping container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:11 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-77bb4b6d4b SuccessfulDelete Deleted pod: tempo-simplest-query-frontend-77bb4b6d4b-vdnjj replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:11 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled down replica set tempo-simplest-query-frontend-77bb4b6d4b to 0 from 1 deployment-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:49:19 +0000 UTC Warning Pod tempo-simplest-distributor-65777488c5-ghbc9.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:50:00 +0000 UTC Normal Pod tempo-simplest-compactor-6bffc9b57f-6hv2k.spec.containers{tempo} Killing Stopping container tempo kubelet logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:50:00 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-6bffc9b57f SuccessfulDelete Deleted pod: tempo-simplest-compactor-6bffc9b57f-6hv2k replicaset-controller logger.go:42: 06:50:01 | reconcile | 2023-12-11 06:50:00 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled down replica set tempo-simplest-compactor-6bffc9b57f to 0 from 1 deployment-controller logger.go:42: 06:50:01 | reconcile | Deleting namespace: kuttl-test-central-mackerel === CONT kuttl/harness/receivers-tls logger.go:42: 06:50:03 | receivers-tls/1-install-tempo | test step completed 1-install-tempo logger.go:42: 06:50:03 | receivers-tls/2-install-otel | starting test step 2-install-otel logger.go:42: 06:50:03 | receivers-tls/2-install-otel | Secret:kuttl-test-brave-oryx/opentelemetry-collector-cert created logger.go:42: 06:50:03 | receivers-tls/2-install-otel | ConfigMap:kuttl-test-brave-oryx/opentelemetry-collector-configmap created logger.go:42: 06:50:03 | receivers-tls/2-install-otel | Deployment:kuttl-test-brave-oryx/opentelemetry-collector created logger.go:42: 06:50:03 | receivers-tls/2-install-otel | Service:kuttl-test-brave-oryx/opentelemetry-collector created === CONT kuttl/harness/compatibility logger.go:42: 06:50:04 | compatibility/3-generate-traces | test step completed 3-generate-traces logger.go:42: 06:50:04 | compatibility/4-verify-traces-jaeger | starting test step 4-verify-traces-jaeger logger.go:42: 06:50:04 | compatibility/4-verify-traces-jaeger | Job:kuttl-test-picked-rooster/verify-traces-jaeger created === CONT kuttl/harness/receivers-tls logger.go:42: 06:50:09 | receivers-tls/2-install-otel | test step completed 2-install-otel logger.go:42: 06:50:09 | receivers-tls/3-generate-traces | starting test step 3-generate-traces logger.go:42: 06:50:09 | receivers-tls/3-generate-traces | Job:kuttl-test-brave-oryx/generate-traces created === CONT kuttl/harness/compatibility logger.go:42: 06:50:10 | compatibility/4-verify-traces-jaeger | test step completed 4-verify-traces-jaeger logger.go:42: 06:50:10 | compatibility/5-verify-traces-grafana | starting test step 5-verify-traces-grafana logger.go:42: 06:50:10 | compatibility/5-verify-traces-grafana | Job:kuttl-test-picked-rooster/verify-traces-grafana created === CONT kuttl/harness/receivers-tls logger.go:42: 06:50:13 | receivers-tls/3-generate-traces | test step completed 3-generate-traces logger.go:42: 06:50:13 | receivers-tls/4-verify-traces | starting test step 4-verify-traces logger.go:42: 06:50:13 | receivers-tls/4-verify-traces | Job:kuttl-test-brave-oryx/verify-traces created === CONT kuttl/harness/compatibility logger.go:42: 06:50:15 | compatibility/5-verify-traces-grafana | test step completed 5-verify-traces-grafana logger.go:42: 06:50:15 | compatibility | compatibility events from ns kuttl-test-picked-rooster: logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:37 +0000 UTC Normal ReplicaSet.apps minio-744f658d8d SuccessfulCreate Created pod: minio-744f658d8d-4c2mp replicaset-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:37 +0000 UTC Normal PersistentVolumeClaim minio WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:37 +0000 UTC Normal Deployment.apps minio ScalingReplicaSet Scaled up replica set minio-744f658d8d to 1 deployment-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:38 +0000 UTC Normal PersistentVolumeClaim minio ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:38 +0000 UTC Normal PersistentVolumeClaim minio Provisioning External provisioner is provisioning volume for claim "kuttl-test-picked-rooster/minio" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:41 +0000 UTC Normal Pod minio-744f658d8d-4c2mp Binding Scheduled Successfully assigned kuttl-test-picked-rooster/minio-744f658d8d-4c2mp to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:41 +0000 UTC Normal PersistentVolumeClaim minio ProvisioningSucceeded Successfully provisioned volume pvc-8d460c3e-8954-459a-a5dd-c5acdb16e9e7 ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:43 +0000 UTC Normal Pod minio-744f658d8d-4c2mp SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-8d460c3e-8954-459a-a5dd-c5acdb16e9e7" attachdetach-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:44 +0000 UTC Normal Pod minio-744f658d8d-4c2mp AddedInterface Add eth0 [10.129.2.19/23] from ovn-kubernetes logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:44 +0000 UTC Normal Pod minio-744f658d8d-4c2mp.spec.containers{minio} Pulling Pulling image "minio/minio" kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:47 +0000 UTC Normal Pod minio-744f658d8d-4c2mp.spec.containers{minio} Pulled Successfully pulled image "minio/minio" in 2.531s (2.531s including waiting) kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:47 +0000 UTC Normal Pod minio-744f658d8d-4c2mp.spec.containers{minio} Created Created container minio kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:47 +0000 UTC Normal Pod minio-744f658d8d-4c2mp.spec.containers{minio} Started Started container minio kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-picked-rooster/data-tempo-simplest-ingester-0" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-simplest-compactor-7489885868-dqts2 Binding Scheduled Successfully assigned kuttl-test-picked-rooster/tempo-simplest-compactor-7489885868-dqts2 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-7489885868 SuccessfulCreate Created pod: tempo-simplest-compactor-7489885868-dqts2 replicaset-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-7489885868 to 1 deployment-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-simplest-distributor-f5d9545d6-rkg7b Binding Scheduled Successfully assigned kuttl-test-picked-rooster/tempo-simplest-distributor-f5d9545d6-rkg7b to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-simplest-distributor-f5d9545d6-rkg7b AddedInterface Add eth0 [10.128.2.26/23] from ovn-kubernetes logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-simplest-distributor-f5d9545d6-rkg7b.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-simplest-distributor-f5d9545d6-rkg7b.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-f5d9545d6 SuccessfulCreate Created pod: tempo-simplest-distributor-f5d9545d6-rkg7b replicaset-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-f5d9545d6 to 1 deployment-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Claim data-tempo-simplest-ingester-0 Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester success statefulset-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful statefulset-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-simplest-querier-644b484cbf-sxnrk Binding Scheduled Successfully assigned kuttl-test-picked-rooster/tempo-simplest-querier-644b484cbf-sxnrk to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-644b484cbf SuccessfulCreate Created pod: tempo-simplest-querier-644b484cbf-sxnrk replicaset-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-644b484cbf to 1 deployment-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-simplest-query-frontend-c76f68bdd-vmch7 Binding Scheduled Successfully assigned kuttl-test-picked-rooster/tempo-simplest-query-frontend-c76f68bdd-vmch7 to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-simplest-query-frontend-c76f68bdd-vmch7 AddedInterface Add eth0 [10.128.2.27/23] from ovn-kubernetes logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Pod tempo-simplest-query-frontend-c76f68bdd-vmch7.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-c76f68bdd SuccessfulCreate Created pod: tempo-simplest-query-frontend-c76f68bdd-vmch7 replicaset-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:49 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-c76f68bdd to 1 deployment-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-compactor-7489885868-dqts2 AddedInterface Add eth0 [10.129.2.25/23] from ovn-kubernetes logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-compactor-7489885868-dqts2.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-compactor-7489885868-dqts2.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-compactor-7489885868-dqts2.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-distributor-f5d9545d6-rkg7b.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-querier-644b484cbf-sxnrk AddedInterface Add eth0 [10.128.2.28/23] from ovn-kubernetes logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-querier-644b484cbf-sxnrk.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-querier-644b484cbf-sxnrk.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-querier-644b484cbf-sxnrk.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-query-frontend-c76f68bdd-vmch7.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-query-frontend-c76f68bdd-vmch7.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:50 +0000 UTC Normal Pod tempo-simplest-query-frontend-c76f68bdd-vmch7.spec.containers{tempo-query} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:53 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-76fe3ff7-676a-4c25-b0b1-4fa54ac93f7b ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:53 +0000 UTC Normal Pod tempo-simplest-query-frontend-c76f68bdd-vmch7.spec.containers{tempo-query} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" in 3.441s (3.441s including waiting) kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:53 +0000 UTC Normal Pod tempo-simplest-query-frontend-c76f68bdd-vmch7.spec.containers{tempo-query} Created Created container tempo-query kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:53 +0000 UTC Normal Pod tempo-simplest-query-frontend-c76f68bdd-vmch7.spec.containers{tempo-query} Started Started container tempo-query kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:54 +0000 UTC Normal Pod tempo-simplest-ingester-0 Binding Scheduled Successfully assigned kuttl-test-picked-rooster/tempo-simplest-ingester-0 to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:56 +0000 UTC Normal Pod tempo-simplest-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-76fe3ff7-676a-4c25-b0b1-4fa54ac93f7b" attachdetach-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:56 +0000 UTC Normal Pod tempo-simplest-ingester-0 AddedInterface Add eth0 [10.131.0.27/23] from ovn-kubernetes logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:56 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:56 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:48:56 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:49:10 +0000 UTC Warning Pod tempo-simplest-compactor-7489885868-dqts2.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:49:16 +0000 UTC Warning Pod tempo-simplest-ingester-0.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:00 +0000 UTC Normal Pod generate-traces-4jplc Binding Scheduled Successfully assigned kuttl-test-picked-rooster/generate-traces-4jplc to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:00 +0000 UTC Normal Job.batch generate-traces SuccessfulCreate Created pod: generate-traces-4jplc job-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:01 +0000 UTC Normal Pod generate-traces-4jplc AddedInterface Add eth0 [10.128.2.36/23] from ovn-kubernetes logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:01 +0000 UTC Normal Pod generate-traces-4jplc.spec.containers{telemetrygen} Pulling Pulling image "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.75.0" kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:02 +0000 UTC Normal Pod generate-traces-4jplc.spec.containers{telemetrygen} Pulled Successfully pulled image "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.75.0" in 1.113s (1.113s including waiting) kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:02 +0000 UTC Normal Pod generate-traces-4jplc.spec.containers{telemetrygen} Created Created container telemetrygen kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:02 +0000 UTC Normal Pod generate-traces-4jplc.spec.containers{telemetrygen} Started Started container telemetrygen kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:04 +0000 UTC Normal Job.batch generate-traces Completed Job completed job-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:04 +0000 UTC Normal Pod verify-traces-jaeger-blnb5 Binding Scheduled Successfully assigned kuttl-test-picked-rooster/verify-traces-jaeger-blnb5 to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:04 +0000 UTC Normal Job.batch verify-traces-jaeger SuccessfulCreate Created pod: verify-traces-jaeger-blnb5 job-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:05 +0000 UTC Normal Pod verify-traces-jaeger-blnb5 AddedInterface Add eth0 [10.131.0.31/23] from ovn-kubernetes logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:05 +0000 UTC Normal Pod verify-traces-jaeger-blnb5.spec.containers{verify-traces-jaeger} Pulling Pulling image "ghcr.io/grafana/tempo-operator/test-utils:main" kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:07 +0000 UTC Normal Pod verify-traces-jaeger-blnb5.spec.containers{verify-traces-jaeger} Pulled Successfully pulled image "ghcr.io/grafana/tempo-operator/test-utils:main" in 1.699s (1.699s including waiting) kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:07 +0000 UTC Normal Pod verify-traces-jaeger-blnb5.spec.containers{verify-traces-jaeger} Created Created container verify-traces-jaeger kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:07 +0000 UTC Normal Pod verify-traces-jaeger-blnb5.spec.containers{verify-traces-jaeger} Started Started container verify-traces-jaeger kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:09 +0000 UTC Normal Job.batch verify-traces-jaeger Completed Job completed job-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:10 +0000 UTC Normal Pod verify-traces-grafana-2hlfm Binding Scheduled Successfully assigned kuttl-test-picked-rooster/verify-traces-grafana-2hlfm to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:10 +0000 UTC Normal Pod verify-traces-grafana-2hlfm AddedInterface Add eth0 [10.131.0.33/23] from ovn-kubernetes logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:10 +0000 UTC Normal Pod verify-traces-grafana-2hlfm.spec.containers{verify-traces-grafana} Pulling Pulling image "registry.gitlab.com/gitlab-ci-utils/curl-jq:1.1.0" kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:10 +0000 UTC Normal Job.batch verify-traces-grafana SuccessfulCreate Created pod: verify-traces-grafana-2hlfm job-controller logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:12 +0000 UTC Normal Pod verify-traces-grafana-2hlfm.spec.containers{verify-traces-grafana} Pulled Successfully pulled image "registry.gitlab.com/gitlab-ci-utils/curl-jq:1.1.0" in 1.604s (1.604s including waiting) kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:12 +0000 UTC Normal Pod verify-traces-grafana-2hlfm.spec.containers{verify-traces-grafana} Created Created container verify-traces-grafana kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:12 +0000 UTC Normal Pod verify-traces-grafana-2hlfm.spec.containers{verify-traces-grafana} Started Started container verify-traces-grafana kubelet logger.go:42: 06:50:15 | compatibility | 2023-12-11 06:50:14 +0000 UTC Normal Job.batch verify-traces-grafana Completed Job completed job-controller logger.go:42: 06:50:15 | compatibility | Deleting namespace: kuttl-test-picked-rooster === CONT kuttl/harness/receivers-tls logger.go:42: 06:50:27 | receivers-tls/4-verify-traces | test step completed 4-verify-traces logger.go:42: 06:50:27 | receivers-tls | receivers-tls events from ns kuttl-test-brave-oryx: logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:37 +0000 UTC Normal ReplicaSet.apps minio-744f658d8d SuccessfulCreate Created pod: minio-744f658d8d-lptnm replicaset-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:37 +0000 UTC Normal PersistentVolumeClaim minio WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:37 +0000 UTC Normal Deployment.apps minio ScalingReplicaSet Scaled up replica set minio-744f658d8d to 1 deployment-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:37 +0000 UTC Normal PersistentVolumeClaim minio Provisioning External provisioner is provisioning volume for claim "kuttl-test-brave-oryx/minio" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:37 +0000 UTC Normal PersistentVolumeClaim minio ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:41 +0000 UTC Normal PersistentVolumeClaim minio ProvisioningSucceeded Successfully provisioned volume pvc-c056728d-5f21-4bc2-a703-401991708b0c ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:42 +0000 UTC Normal Pod minio-744f658d8d-lptnm Binding Scheduled Successfully assigned kuttl-test-brave-oryx/minio-744f658d8d-lptnm to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:46 +0000 UTC Normal Pod minio-744f658d8d-lptnm SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-c056728d-5f21-4bc2-a703-401991708b0c" attachdetach-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:47 +0000 UTC Normal Pod minio-744f658d8d-lptnm AddedInterface Add eth0 [10.129.2.21/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:48 +0000 UTC Normal Pod minio-744f658d8d-lptnm.spec.containers{minio} Pulling Pulling image "minio/minio" kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:48 +0000 UTC Normal Pod minio-744f658d8d-lptnm.spec.containers{minio} Pulled Successfully pulled image "minio/minio" in 352ms (352ms including waiting) kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:48 +0000 UTC Normal Pod minio-744f658d8d-lptnm.spec.containers{minio} Created Created container minio kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:48 +0000 UTC Normal Pod minio-744f658d8d-lptnm.spec.containers{minio} Started Started container minio kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-brave-oryx/data-tempo-simplest-ingester-0" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-simplest-compactor-6f455f6649-wh64t Binding Scheduled Successfully assigned kuttl-test-brave-oryx/tempo-simplest-compactor-6f455f6649-wh64t to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-6f455f6649 SuccessfulCreate Created pod: tempo-simplest-compactor-6f455f6649-wh64t replicaset-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-6f455f6649 to 1 deployment-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-simplest-distributor-56c9747c44-55g6b Binding Scheduled Successfully assigned kuttl-test-brave-oryx/tempo-simplest-distributor-56c9747c44-55g6b to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-simplest-distributor-56c9747c44-55g6b AddedInterface Add eth0 [10.129.2.27/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-simplest-distributor-56c9747c44-55g6b.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-simplest-distributor-56c9747c44-55g6b.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-56c9747c44 SuccessfulCreate Created pod: tempo-simplest-distributor-56c9747c44-55g6b replicaset-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-56c9747c44 to 1 deployment-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Claim data-tempo-simplest-ingester-0 Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester success statefulset-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful statefulset-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-simplest-querier-688958646-crr2l Binding Scheduled Successfully assigned kuttl-test-brave-oryx/tempo-simplest-querier-688958646-crr2l to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-688958646 SuccessfulCreate Created pod: tempo-simplest-querier-688958646-crr2l replicaset-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-688958646 to 1 deployment-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-simplest-query-frontend-68b797464f-bvrnq Binding Scheduled Successfully assigned kuttl-test-brave-oryx/tempo-simplest-query-frontend-68b797464f-bvrnq to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-simplest-query-frontend-68b797464f-bvrnq AddedInterface Add eth0 [10.129.2.28/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Pod tempo-simplest-query-frontend-68b797464f-bvrnq.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-68b797464f SuccessfulCreate Created pod: tempo-simplest-query-frontend-68b797464f-bvrnq replicaset-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:51 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-68b797464f to 1 deployment-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-compactor-6f455f6649-wh64t AddedInterface Add eth0 [10.128.2.29/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-compactor-6f455f6649-wh64t.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-compactor-6f455f6649-wh64t.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-compactor-6f455f6649-wh64t.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-distributor-56c9747c44-55g6b.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-querier-688958646-crr2l AddedInterface Add eth0 [10.131.0.26/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-querier-688958646-crr2l.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-querier-688958646-crr2l.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-querier-688958646-crr2l.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-68b797464f-bvrnq.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-68b797464f-bvrnq.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-68b797464f-bvrnq.spec.containers{tempo-query} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:55 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-417149d7-ec05-4850-aff9-184e64aa2167 ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:56 +0000 UTC Normal Pod tempo-simplest-ingester-0 Binding Scheduled Successfully assigned kuttl-test-brave-oryx/tempo-simplest-ingester-0 to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:58 +0000 UTC Normal Pod tempo-simplest-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-417149d7-ec05-4850-aff9-184e64aa2167" attachdetach-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:58 +0000 UTC Normal Pod tempo-simplest-ingester-0 AddedInterface Add eth0 [10.128.2.30/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:58 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:58 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:58 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:58 +0000 UTC Normal Pod tempo-simplest-query-frontend-68b797464f-bvrnq.spec.containers{tempo-query} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" in 6.883s (6.883s including waiting) kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:58 +0000 UTC Normal Pod tempo-simplest-query-frontend-68b797464f-bvrnq.spec.containers{tempo-query} Created Created container tempo-query kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:48:58 +0000 UTC Normal Pod tempo-simplest-query-frontend-68b797464f-bvrnq.spec.containers{tempo-query} Started Started container tempo-query kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:49:12 +0000 UTC Warning Pod tempo-simplest-compactor-6f455f6649-wh64t.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:49:18 +0000 UTC Warning Pod tempo-simplest-ingester-0.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:03 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-7rggz Binding Scheduled Successfully assigned kuttl-test-brave-oryx/opentelemetry-collector-9dcc5fdb6-7rggz to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:03 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-7rggz AddedInterface Add eth0 [10.131.0.30/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:03 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-7rggz.spec.containers{opentelemetry-collector} Pulling Pulling image "otel/opentelemetry-collector-contrib:0.82.0" kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:03 +0000 UTC Normal ReplicaSet.apps opentelemetry-collector-9dcc5fdb6 SuccessfulCreate Created pod: opentelemetry-collector-9dcc5fdb6-7rggz replicaset-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:03 +0000 UTC Normal Deployment.apps opentelemetry-collector ScalingReplicaSet Scaled up replica set opentelemetry-collector-9dcc5fdb6 to 1 deployment-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:06 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-7rggz.spec.containers{opentelemetry-collector} Pulled Successfully pulled image "otel/opentelemetry-collector-contrib:0.82.0" in 2.894s (2.894s including waiting) kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:06 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-7rggz.spec.containers{opentelemetry-collector} Created Created container opentelemetry-collector kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:06 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-7rggz.spec.containers{opentelemetry-collector} Started Started container opentelemetry-collector kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:09 +0000 UTC Normal Pod generate-traces-ndw47 Binding Scheduled Successfully assigned kuttl-test-brave-oryx/generate-traces-ndw47 to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:09 +0000 UTC Normal Job.batch generate-traces SuccessfulCreate Created pod: generate-traces-ndw47 job-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:10 +0000 UTC Normal Pod generate-traces-ndw47 AddedInterface Add eth0 [10.131.0.32/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:10 +0000 UTC Normal Pod generate-traces-ndw47.spec.containers{telemetrygen} Pulling Pulling image "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.75.0" kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:10 +0000 UTC Normal Pod generate-traces-ndw47.spec.containers{telemetrygen} Pulled Successfully pulled image "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.75.0" in 724ms (724ms including waiting) kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:11 +0000 UTC Normal Pod generate-traces-ndw47.spec.containers{telemetrygen} Created Created container telemetrygen kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:11 +0000 UTC Normal Pod generate-traces-ndw47.spec.containers{telemetrygen} Started Started container telemetrygen kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:13 +0000 UTC Normal Job.batch generate-traces Completed Job completed job-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:13 +0000 UTC Normal Pod verify-traces-bb7md Binding Scheduled Successfully assigned kuttl-test-brave-oryx/verify-traces-bb7md to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:13 +0000 UTC Normal Job.batch verify-traces SuccessfulCreate Created pod: verify-traces-bb7md job-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:14 +0000 UTC Normal Pod verify-traces-bb7md AddedInterface Add eth0 [10.131.0.34/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:14 +0000 UTC Normal Pod verify-traces-bb7md.spec.containers{verify-traces} Pulled Container image "ghcr.io/grafana/tempo-operator/test-utils:main" already present on machine kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:14 +0000 UTC Normal Pod verify-traces-bb7md.spec.containers{verify-traces} Created Created container verify-traces kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:14 +0000 UTC Normal Pod verify-traces-bb7md.spec.containers{verify-traces} Started Started container verify-traces kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:24 +0000 UTC Normal Pod verify-traces-594m7 Binding Scheduled Successfully assigned kuttl-test-brave-oryx/verify-traces-594m7 to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:24 +0000 UTC Normal Pod verify-traces-594m7 AddedInterface Add eth0 [10.131.0.35/23] from ovn-kubernetes logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:24 +0000 UTC Normal Pod verify-traces-594m7.spec.containers{verify-traces} Pulled Container image "ghcr.io/grafana/tempo-operator/test-utils:main" already present on machine kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:24 +0000 UTC Normal Pod verify-traces-594m7.spec.containers{verify-traces} Created Created container verify-traces kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:24 +0000 UTC Normal Pod verify-traces-594m7.spec.containers{verify-traces} Started Started container verify-traces kubelet logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:24 +0000 UTC Normal Job.batch verify-traces SuccessfulCreate Created pod: verify-traces-594m7 job-controller logger.go:42: 06:50:27 | receivers-tls | 2023-12-11 06:50:26 +0000 UTC Normal Job.batch verify-traces Completed Job completed job-controller logger.go:42: 06:50:27 | receivers-tls | Deleting namespace: kuttl-test-brave-oryx === CONT kuttl/harness/red-metrics logger.go:42: 06:50:36 | red-metrics | Ignoring check_user_workload_monitoring.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$ logger.go:42: 06:50:36 | red-metrics | Creating namespace: kuttl-test-ample-rattler logger.go:42: 06:50:36 | red-metrics/0-install-storage | starting test step 0-install-storage logger.go:42: 06:50:36 | red-metrics/0-install-storage | PersistentVolumeClaim:kuttl-test-ample-rattler/minio created logger.go:42: 06:50:36 | red-metrics/0-install-storage | Deployment:kuttl-test-ample-rattler/minio created logger.go:42: 06:50:36 | red-metrics/0-install-storage | Service:kuttl-test-ample-rattler/minio created === CONT kuttl/harness/route logger.go:42: 06:50:50 | route | Creating namespace: kuttl-test-relevant-cowbird logger.go:42: 06:50:50 | route/0-install | starting test step 0-install logger.go:42: 06:50:51 | route/0-install | Secret:kuttl-test-relevant-cowbird/minio-test created logger.go:42: 06:50:51 | route/0-install | TempoStack:kuttl-test-relevant-cowbird/simplest created === CONT kuttl/harness/custom-ca logger.go:42: 06:50:51 | custom-ca/1-install-tempo | test step completed 1-install-tempo logger.go:42: 06:50:51 | custom-ca/2-generate-traces | starting test step 2-generate-traces logger.go:42: 06:50:51 | custom-ca/2-generate-traces | Job:kuttl-test-sensible-shrew/generate-traces created === CONT kuttl/harness/red-metrics logger.go:42: 06:50:51 | red-metrics/0-install-storage | test step completed 0-install-storage logger.go:42: 06:50:51 | red-metrics/1-install-workload-monitoring | starting test step 1-install-workload-monitoring logger.go:42: 06:50:52 | red-metrics/1-install-workload-monitoring | ConfigMap:openshift-monitoring/cluster-monitoring-config updated logger.go:42: 06:50:52 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:50:52 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found logger.go:42: 06:50:53 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] === CONT kuttl/harness/route logger.go:42: 06:50:53 | route/0-install | test step completed 0-install logger.go:42: 06:50:53 | route | route events from ns kuttl-test-relevant-cowbird: logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-relevant-cowbird/data-tempo-simplest-ingester-0" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-compactor-54d4576fb9-nzr28 Binding Scheduled Successfully assigned kuttl-test-relevant-cowbird/tempo-simplest-compactor-54d4576fb9-nzr28 to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-compactor-54d4576fb9-nzr28 AddedInterface Add eth0 [10.131.0.39/23] from ovn-kubernetes logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-compactor-54d4576fb9-nzr28.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-compactor-54d4576fb9-nzr28.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-compactor-54d4576fb9-nzr28.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-54d4576fb9 SuccessfulCreate Created pod: tempo-simplest-compactor-54d4576fb9-nzr28 replicaset-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-54d4576fb9 to 1 deployment-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-distributor-b6f8d55c7-t4h6l Binding Scheduled Successfully assigned kuttl-test-relevant-cowbird/tempo-simplest-distributor-b6f8d55c7-t4h6l to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-distributor-b6f8d55c7-t4h6l AddedInterface Add eth0 [10.131.0.37/23] from ovn-kubernetes logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-distributor-b6f8d55c7-t4h6l.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-distributor-b6f8d55c7-t4h6l.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-distributor-b6f8d55c7-t4h6l.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-b6f8d55c7 SuccessfulCreate Created pod: tempo-simplest-distributor-b6f8d55c7-t4h6l replicaset-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-b6f8d55c7 to 1 deployment-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Claim data-tempo-simplest-ingester-0 Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester success statefulset-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful statefulset-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-querier-55957695d-98vw6 Binding Scheduled Successfully assigned kuttl-test-relevant-cowbird/tempo-simplest-querier-55957695d-98vw6 to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-querier-55957695d-98vw6 AddedInterface Add eth0 [10.131.0.38/23] from ovn-kubernetes logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-querier-55957695d-98vw6.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-querier-55957695d-98vw6.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-querier-55957695d-98vw6.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-55957695d SuccessfulCreate Created pod: tempo-simplest-querier-55957695d-98vw6 replicaset-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-55957695d to 1 deployment-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-8456bf4f45-pcpgq Binding Scheduled Successfully assigned kuttl-test-relevant-cowbird/tempo-simplest-query-frontend-8456bf4f45-pcpgq to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-8456bf4f45-pcpgq AddedInterface Add eth0 [10.129.2.35/23] from ovn-kubernetes logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-8456bf4f45-pcpgq.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-8456bf4f45-pcpgq.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-8456bf4f45-pcpgq.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-8456bf4f45-pcpgq.spec.containers{tempo-query} Pulled Container image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" already present on machine kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-8456bf4f45-pcpgq.spec.containers{tempo-query} Created Created container tempo-query kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Pod tempo-simplest-query-frontend-8456bf4f45-pcpgq.spec.containers{tempo-query} Started Started container tempo-query kubelet logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-8456bf4f45 SuccessfulCreate Created pod: tempo-simplest-query-frontend-8456bf4f45-pcpgq replicaset-controller logger.go:42: 06:50:53 | route | 2023-12-11 06:50:52 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-8456bf4f45 to 1 deployment-controller logger.go:42: 06:50:53 | route | Deleting namespace: kuttl-test-relevant-cowbird === CONT kuttl/harness/red-metrics logger.go:42: 06:50:53 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found logger.go:42: 06:50:54 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:50:54 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found logger.go:42: 06:50:55 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:50:55 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found === CONT kuttl/harness/receivers-mtls logger.go:42: 06:50:55 | receivers-mtls/1-install-tempo | test step completed 1-install-tempo logger.go:42: 06:50:55 | receivers-mtls/2-install-otel | starting test step 2-install-otel logger.go:42: 06:50:55 | receivers-mtls/2-install-otel | Secret:kuttl-test-renewing-penguin/opentelemetry-collector-cert created logger.go:42: 06:50:55 | receivers-mtls/2-install-otel | ConfigMap:kuttl-test-renewing-penguin/opentelemetry-collector-configmap created logger.go:42: 06:50:55 | receivers-mtls/2-install-otel | Deployment:kuttl-test-renewing-penguin/opentelemetry-collector created logger.go:42: 06:50:55 | receivers-mtls/2-install-otel | Service:kuttl-test-renewing-penguin/opentelemetry-collector created === CONT kuttl/harness/red-metrics logger.go:42: 06:50:56 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:50:56 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found === CONT kuttl/harness/custom-ca logger.go:42: 06:50:56 | custom-ca/2-generate-traces | test step completed 2-generate-traces logger.go:42: 06:50:56 | custom-ca/3-verify-traces | starting test step 3-verify-traces logger.go:42: 06:50:56 | custom-ca/3-verify-traces | Job:kuttl-test-sensible-shrew/verify-traces created === CONT kuttl/harness/red-metrics logger.go:42: 06:50:57 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:50:57 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found logger.go:42: 06:50:58 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:50:58 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found logger.go:42: 06:50:59 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] === CONT kuttl/harness/custom-ca logger.go:42: 06:51:00 | custom-ca/3-verify-traces | test step completed 3-verify-traces logger.go:42: 06:51:00 | custom-ca | custom-ca events from ns kuttl-test-sensible-shrew: logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:37 +0000 UTC Normal Pod minio-b55669bc8-lz7ld Binding Scheduled Successfully assigned kuttl-test-sensible-shrew/minio-b55669bc8-lz7ld to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:37 +0000 UTC Normal ReplicaSet.apps minio-b55669bc8 SuccessfulCreate Created pod: minio-b55669bc8-lz7ld replicaset-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:37 +0000 UTC Normal Deployment.apps minio ScalingReplicaSet Scaled up replica set minio-b55669bc8 to 1 deployment-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:38 +0000 UTC Normal Pod minio-b55669bc8-lz7ld AddedInterface Add eth0 [10.129.2.29/23] from ovn-kubernetes logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:38 +0000 UTC Normal Pod minio-b55669bc8-lz7ld.spec.containers{minio} Pulling Pulling image "minio/minio" kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:38 +0000 UTC Normal Pod minio-b55669bc8-lz7ld.spec.containers{minio} Pulled Successfully pulled image "minio/minio" in 513ms (513ms including waiting) kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:38 +0000 UTC Normal Pod minio-b55669bc8-lz7ld.spec.containers{minio} Created Created container minio kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:38 +0000 UTC Normal Pod minio-b55669bc8-lz7ld.spec.containers{minio} Started Started container minio kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-sensible-shrew/data-tempo-simplest-ingester-0" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal Pod tempo-simplest-compactor-575f8696bc-zx2vn Binding Scheduled Successfully assigned kuttl-test-sensible-shrew/tempo-simplest-compactor-575f8696bc-zx2vn to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-575f8696bc SuccessfulCreate Created pod: tempo-simplest-compactor-575f8696bc-zx2vn replicaset-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-575f8696bc to 1 deployment-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal Pod tempo-simplest-distributor-6db98dc4f8-dz56q Binding Scheduled Successfully assigned kuttl-test-sensible-shrew/tempo-simplest-distributor-6db98dc4f8-dz56q to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-6db98dc4f8 SuccessfulCreate Created pod: tempo-simplest-distributor-6db98dc4f8-dz56q replicaset-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-6db98dc4f8 to 1 deployment-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Claim data-tempo-simplest-ingester-0 Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester success statefulset-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful statefulset-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal Pod tempo-simplest-querier-66d7749f54-q27gq Binding Scheduled Successfully assigned kuttl-test-sensible-shrew/tempo-simplest-querier-66d7749f54-q27gq to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-66d7749f54 SuccessfulCreate Created pod: tempo-simplest-querier-66d7749f54-q27gq replicaset-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-66d7749f54 to 1 deployment-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal Pod tempo-simplest-query-frontend-677687b9c9-cq9dg Binding Scheduled Successfully assigned kuttl-test-sensible-shrew/tempo-simplest-query-frontend-677687b9c9-cq9dg to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-677687b9c9 SuccessfulCreate Created pod: tempo-simplest-query-frontend-677687b9c9-cq9dg replicaset-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:40 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-677687b9c9 to 1 deployment-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-compactor-575f8696bc-zx2vn AddedInterface Add eth0 [10.128.2.34/23] from ovn-kubernetes logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-compactor-575f8696bc-zx2vn.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-compactor-575f8696bc-zx2vn.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-compactor-575f8696bc-zx2vn.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-distributor-6db98dc4f8-dz56q AddedInterface Add eth0 [10.128.2.32/23] from ovn-kubernetes logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-distributor-6db98dc4f8-dz56q.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-distributor-6db98dc4f8-dz56q.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-distributor-6db98dc4f8-dz56q.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-querier-66d7749f54-q27gq AddedInterface Add eth0 [10.128.2.33/23] from ovn-kubernetes logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-querier-66d7749f54-q27gq.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-querier-66d7749f54-q27gq.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-querier-66d7749f54-q27gq.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-query-frontend-677687b9c9-cq9dg AddedInterface Add eth0 [10.129.2.30/23] from ovn-kubernetes logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-query-frontend-677687b9c9-cq9dg.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-query-frontend-677687b9c9-cq9dg.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-query-frontend-677687b9c9-cq9dg.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-query-frontend-677687b9c9-cq9dg.spec.containers{tempo-query} Pulled Container image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" already present on machine kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-query-frontend-677687b9c9-cq9dg.spec.containers{tempo-query} Created Created container tempo-query kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:41 +0000 UTC Normal Pod tempo-simplest-query-frontend-677687b9c9-cq9dg.spec.containers{tempo-query} Started Started container tempo-query kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:45 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-b6ebfdc0-0c41-491f-85b8-9a397a59fe88 ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:45 +0000 UTC Normal Pod tempo-simplest-ingester-0 Binding Scheduled Successfully assigned kuttl-test-sensible-shrew/tempo-simplest-ingester-0 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:47 +0000 UTC Normal Pod tempo-simplest-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-b6ebfdc0-0c41-491f-85b8-9a397a59fe88" attachdetach-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:56 +0000 UTC Normal Pod tempo-simplest-ingester-0 AddedInterface Add eth0 [10.129.2.33/23] from ovn-kubernetes logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:56 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:56 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:49:56 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:01 +0000 UTC Warning Pod tempo-simplest-compactor-575f8696bc-zx2vn.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:16 +0000 UTC Warning Pod tempo-simplest-ingester-0.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:51 +0000 UTC Normal Pod generate-traces-9l8dl Binding Scheduled Successfully assigned kuttl-test-sensible-shrew/generate-traces-9l8dl to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:51 +0000 UTC Normal Job.batch generate-traces SuccessfulCreate Created pod: generate-traces-9l8dl job-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:52 +0000 UTC Normal Pod generate-traces-9l8dl AddedInterface Add eth0 [10.129.2.34/23] from ovn-kubernetes logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:52 +0000 UTC Normal Pod generate-traces-9l8dl.spec.containers{telemetrygen} Pulling Pulling image "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.75.0" kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:53 +0000 UTC Normal Pod generate-traces-9l8dl.spec.containers{telemetrygen} Pulled Successfully pulled image "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.75.0" in 922ms (922ms including waiting) kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:53 +0000 UTC Normal Pod generate-traces-9l8dl.spec.containers{telemetrygen} Created Created container telemetrygen kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:53 +0000 UTC Normal Pod generate-traces-9l8dl.spec.containers{telemetrygen} Started Started container telemetrygen kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:56 +0000 UTC Normal Job.batch generate-traces Completed Job completed job-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:56 +0000 UTC Normal Pod verify-traces-hpshv Binding Scheduled Successfully assigned kuttl-test-sensible-shrew/verify-traces-hpshv to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:56 +0000 UTC Normal Job.batch verify-traces SuccessfulCreate Created pod: verify-traces-hpshv job-controller logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:57 +0000 UTC Normal Pod verify-traces-hpshv AddedInterface Add eth0 [10.131.0.40/23] from ovn-kubernetes logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:57 +0000 UTC Normal Pod verify-traces-hpshv.spec.containers{verify-traces} Pulled Container image "ghcr.io/grafana/tempo-operator/test-utils:main" already present on machine kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:57 +0000 UTC Normal Pod verify-traces-hpshv.spec.containers{verify-traces} Created Created container verify-traces kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:57 +0000 UTC Normal Pod verify-traces-hpshv.spec.containers{verify-traces} Started Started container verify-traces kubelet logger.go:42: 06:51:00 | custom-ca | 2023-12-11 06:50:59 +0000 UTC Normal Job.batch verify-traces Completed Job completed job-controller === CONT kuttl/harness/red-metrics logger.go:42: 06:51:00 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found === CONT kuttl/harness/custom-ca logger.go:42: 06:51:00 | custom-ca | Deleting namespace: kuttl-test-sensible-shrew === CONT kuttl/harness/red-metrics logger.go:42: 06:51:01 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] === CONT kuttl/harness/receivers-mtls logger.go:42: 06:51:01 | receivers-mtls/2-install-otel | test step completed 2-install-otel logger.go:42: 06:51:01 | receivers-mtls/3-generate-traces | starting test step 3-generate-traces logger.go:42: 06:51:01 | receivers-mtls/3-generate-traces | Job:kuttl-test-renewing-penguin/generate-traces created === CONT kuttl/harness/red-metrics logger.go:42: 06:51:01 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found logger.go:42: 06:51:02 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:02 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found === CONT kuttl/harness/multitenancy 2023/12/11 06:51:02 object detected with no GVK Kind for path /tmp/tempo-tests/tests/e2e-openshift/multitenancy/02-install-otelcol.yaml logger.go:42: 06:51:02 | multitenancy | Creating namespace: kuttl-test-crisp-buzzard logger.go:42: 06:51:02 | multitenancy/0-install-storage | starting test step 0-install-storage logger.go:42: 06:51:02 | multitenancy/0-install-storage | Namespace:/kuttl-multitenancy created logger.go:42: 06:51:02 | multitenancy/0-install-storage | PersistentVolumeClaim:kuttl-multitenancy/minio created logger.go:42: 06:51:02 | multitenancy/0-install-storage | Deployment:kuttl-multitenancy/minio created logger.go:42: 06:51:02 | multitenancy/0-install-storage | Service:kuttl-multitenancy/minio created logger.go:42: 06:51:02 | multitenancy/0-install-storage | Secret:kuttl-multitenancy/minio created === CONT kuttl/harness/red-metrics logger.go:42: 06:51:03 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:03 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): deployments.apps "prometheus-operator" not found logger.go:42: 06:51:04 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:04 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): statefulsets.apps "prometheus-user-workload" not found === CONT kuttl/harness/receivers-mtls logger.go:42: 06:51:05 | receivers-mtls/3-generate-traces | test step completed 3-generate-traces logger.go:42: 06:51:05 | receivers-mtls/4-verify-traces | starting test step 4-verify-traces logger.go:42: 06:51:05 | receivers-mtls/4-verify-traces | Job:kuttl-test-renewing-penguin/verify-traces created === CONT kuttl/harness/red-metrics logger.go:42: 06:51:05 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:06 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): statefulsets.apps "prometheus-user-workload" not found logger.go:42: 06:51:07 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:07 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): statefulsets.apps "prometheus-user-workload" not found logger.go:42: 06:51:08 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:08 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): statefulsets.apps "prometheus-user-workload" not found logger.go:42: 06:51:09 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:09 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): statefulsets.apps "prometheus-user-workload" not found logger.go:42: 06:51:10 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:11 | red-metrics/1-install-workload-monitoring | Error from server (NotFound): statefulsets.apps "prometheus-user-workload" not found logger.go:42: 06:51:12 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:12 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:13 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:13 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:14 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:14 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:15 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:16 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:17 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:17 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:18 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:18 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:19 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] === CONT kuttl/harness/receivers-mtls logger.go:42: 06:51:19 | receivers-mtls/4-verify-traces | test step completed 4-verify-traces logger.go:42: 06:51:19 | receivers-mtls | receivers-mtls events from ns kuttl-test-renewing-penguin: logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:28 +0000 UTC Normal ReplicaSet.apps minio-744f658d8d SuccessfulCreate Created pod: minio-744f658d8d-dglvk replicaset-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:28 +0000 UTC Normal PersistentVolumeClaim minio WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:28 +0000 UTC Normal Deployment.apps minio ScalingReplicaSet Scaled up replica set minio-744f658d8d to 1 deployment-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:28 +0000 UTC Normal PersistentVolumeClaim minio ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:28 +0000 UTC Normal PersistentVolumeClaim minio Provisioning External provisioner is provisioning volume for claim "kuttl-test-renewing-penguin/minio" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:32 +0000 UTC Normal PersistentVolumeClaim minio ProvisioningSucceeded Successfully provisioned volume pvc-306f12b5-b57c-4fb1-905e-d9e2f55c2424 ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:33 +0000 UTC Normal Pod minio-744f658d8d-dglvk Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/minio-744f658d8d-dglvk to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:35 +0000 UTC Normal Pod minio-744f658d8d-dglvk SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-306f12b5-b57c-4fb1-905e-d9e2f55c2424" attachdetach-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:38 +0000 UTC Normal Pod minio-744f658d8d-dglvk AddedInterface Add eth0 [10.128.2.31/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:38 +0000 UTC Normal Pod minio-744f658d8d-dglvk.spec.containers{minio} Pulling Pulling image "minio/minio" kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:41 +0000 UTC Normal Pod minio-744f658d8d-dglvk.spec.containers{minio} Pulled Successfully pulled image "minio/minio" in 2.469s (2.469s including waiting) kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:41 +0000 UTC Normal Pod minio-744f658d8d-dglvk.spec.containers{minio} Created Created container minio kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:41 +0000 UTC Normal Pod minio-744f658d8d-dglvk.spec.containers{minio} Started Started container minio kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-renewing-penguin/data-tempo-simplest-ingester-0" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal Pod tempo-simplest-distributor-858c555dcc-9qlwb Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/tempo-simplest-distributor-858c555dcc-9qlwb to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-858c555dcc SuccessfulCreate Created pod: tempo-simplest-distributor-858c555dcc-9qlwb replicaset-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-858c555dcc to 1 deployment-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Claim data-tempo-simplest-ingester-0 Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester success statefulset-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful statefulset-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal Pod tempo-simplest-querier-8649984579-fvfhl Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/tempo-simplest-querier-8649984579-fvfhl to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-8649984579 SuccessfulCreate Created pod: tempo-simplest-querier-8649984579-fvfhl replicaset-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-8649984579 to 1 deployment-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal Pod tempo-simplest-query-frontend-66dd7c978d-nwntq Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/tempo-simplest-query-frontend-66dd7c978d-nwntq to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-66dd7c978d SuccessfulCreate Created pod: tempo-simplest-query-frontend-66dd7c978d-nwntq replicaset-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:43 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-66dd7c978d to 1 deployment-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-compactor-585656d878-7sknk Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/tempo-simplest-compactor-585656d878-7sknk to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-compactor-585656d878-7sknk AddedInterface Add eth0 [10.131.0.28/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-compactor-585656d878-7sknk.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-compactor-585656d878-7sknk.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-compactor-585656d878-7sknk.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-585656d878 SuccessfulCreate Created pod: tempo-simplest-compactor-585656d878-7sknk replicaset-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-585656d878 to 1 deployment-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-distributor-858c555dcc-9qlwb AddedInterface Add eth0 [10.129.2.31/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-distributor-858c555dcc-9qlwb.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-distributor-858c555dcc-9qlwb.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-distributor-858c555dcc-9qlwb.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-querier-8649984579-fvfhl AddedInterface Add eth0 [10.128.2.35/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-querier-8649984579-fvfhl.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-querier-8649984579-fvfhl.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-querier-8649984579-fvfhl.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-query-frontend-66dd7c978d-nwntq AddedInterface Add eth0 [10.129.2.32/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-query-frontend-66dd7c978d-nwntq.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-query-frontend-66dd7c978d-nwntq.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-query-frontend-66dd7c978d-nwntq.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-query-frontend-66dd7c978d-nwntq.spec.containers{tempo-query} Pulled Container image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" already present on machine kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-query-frontend-66dd7c978d-nwntq.spec.containers{tempo-query} Created Created container tempo-query kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:44 +0000 UTC Normal Pod tempo-simplest-query-frontend-66dd7c978d-nwntq.spec.containers{tempo-query} Started Started container tempo-query kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:47 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-909b0fdf-5c53-48ad-a051-c4ae064bb4a9 ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:47 +0000 UTC Normal Pod tempo-simplest-ingester-0 Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/tempo-simplest-ingester-0 to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:49 +0000 UTC Normal Pod tempo-simplest-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-909b0fdf-5c53-48ad-a051-c4ae064bb4a9" attachdetach-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:57 +0000 UTC Normal Pod tempo-simplest-ingester-0 AddedInterface Add eth0 [10.131.0.29/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:57 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:58 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:49:58 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:04 +0000 UTC Warning Pod tempo-simplest-compactor-585656d878-7sknk.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:17 +0000 UTC Warning Pod tempo-simplest-ingester-0.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:55 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-8nds6 Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/opentelemetry-collector-9dcc5fdb6-8nds6 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:55 +0000 UTC Normal ReplicaSet.apps opentelemetry-collector-9dcc5fdb6 SuccessfulCreate Created pod: opentelemetry-collector-9dcc5fdb6-8nds6 replicaset-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:55 +0000 UTC Normal Deployment.apps opentelemetry-collector ScalingReplicaSet Scaled up replica set opentelemetry-collector-9dcc5fdb6 to 1 deployment-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:56 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-8nds6 AddedInterface Add eth0 [10.129.2.36/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:56 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-8nds6.spec.containers{opentelemetry-collector} Pulling Pulling image "otel/opentelemetry-collector-contrib:0.82.0" kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:59 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-8nds6.spec.containers{opentelemetry-collector} Pulled Successfully pulled image "otel/opentelemetry-collector-contrib:0.82.0" in 2.88s (2.88s including waiting) kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:59 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-8nds6.spec.containers{opentelemetry-collector} Created Created container opentelemetry-collector kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:50:59 +0000 UTC Normal Pod opentelemetry-collector-9dcc5fdb6-8nds6.spec.containers{opentelemetry-collector} Started Started container opentelemetry-collector kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:01 +0000 UTC Normal Pod generate-traces-9qpgm Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/generate-traces-9qpgm to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:01 +0000 UTC Normal Pod generate-traces-9qpgm AddedInterface Add eth0 [10.129.2.37/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:01 +0000 UTC Normal Pod generate-traces-9qpgm.spec.containers{telemetrygen} Pulled Container image "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.75.0" already present on machine kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:01 +0000 UTC Normal Pod generate-traces-9qpgm.spec.containers{telemetrygen} Created Created container telemetrygen kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:01 +0000 UTC Normal Pod generate-traces-9qpgm.spec.containers{telemetrygen} Started Started container telemetrygen kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:01 +0000 UTC Normal Job.batch generate-traces SuccessfulCreate Created pod: generate-traces-9qpgm job-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:04 +0000 UTC Normal Job.batch generate-traces Completed Job completed job-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:05 +0000 UTC Normal Pod verify-traces-ztbl4 Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/verify-traces-ztbl4 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:05 +0000 UTC Normal Pod verify-traces-ztbl4 AddedInterface Add eth0 [10.129.2.38/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:05 +0000 UTC Normal Pod verify-traces-ztbl4.spec.containers{verify-traces} Pulling Pulling image "ghcr.io/grafana/tempo-operator/test-utils:main" kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:05 +0000 UTC Normal Job.batch verify-traces SuccessfulCreate Created pod: verify-traces-ztbl4 job-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:06 +0000 UTC Normal Pod verify-traces-ztbl4.spec.containers{verify-traces} Pulled Successfully pulled image "ghcr.io/grafana/tempo-operator/test-utils:main" in 880ms (880ms including waiting) kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:06 +0000 UTC Normal Pod verify-traces-ztbl4.spec.containers{verify-traces} Created Created container verify-traces kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:06 +0000 UTC Normal Pod verify-traces-ztbl4.spec.containers{verify-traces} Started Started container verify-traces kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:16 +0000 UTC Normal Pod verify-traces-7xcgs Binding Scheduled Successfully assigned kuttl-test-renewing-penguin/verify-traces-7xcgs to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:16 +0000 UTC Normal Pod verify-traces-7xcgs AddedInterface Add eth0 [10.129.2.42/23] from ovn-kubernetes logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:16 +0000 UTC Normal Pod verify-traces-7xcgs.spec.containers{verify-traces} Pulled Container image "ghcr.io/grafana/tempo-operator/test-utils:main" already present on machine kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:16 +0000 UTC Normal Pod verify-traces-7xcgs.spec.containers{verify-traces} Created Created container verify-traces kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:16 +0000 UTC Normal Pod verify-traces-7xcgs.spec.containers{verify-traces} Started Started container verify-traces kubelet logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:16 +0000 UTC Normal Job.batch verify-traces SuccessfulCreate Created pod: verify-traces-7xcgs job-controller logger.go:42: 06:51:19 | receivers-mtls | 2023-12-11 06:51:19 +0000 UTC Normal Job.batch verify-traces Completed Job completed job-controller === CONT kuttl/harness/red-metrics logger.go:42: 06:51:19 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload === CONT kuttl/harness/receivers-mtls logger.go:42: 06:51:20 | receivers-mtls | Deleting namespace: kuttl-test-renewing-penguin === CONT kuttl/harness/multitenancy logger.go:42: 06:51:20 | multitenancy/0-install-storage | test step completed 0-install-storage logger.go:42: 06:51:20 | multitenancy/1-install-tempo | starting test step 1-install-tempo logger.go:42: 06:51:20 | multitenancy/1-install-tempo | TempoStack:kuttl-multitenancy/simplest created logger.go:42: 06:51:20 | multitenancy/1-install-tempo | ClusterRole:/tempostack-traces-reader created logger.go:42: 06:51:20 | multitenancy/1-install-tempo | ClusterRoleBinding:/tempostack-traces-reader created logger.go:42: 06:51:20 | multitenancy/1-install-tempo | RoleBinding:kuttl-multitenancy/view created === CONT kuttl/harness/red-metrics logger.go:42: 06:51:20 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:21 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:22 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:22 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:23 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:23 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:24 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:24 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:25 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:26 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:27 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:27 | red-metrics/1-install-workload-monitoring | The number of replicas is 0 for statefulset prometheus-user-workload logger.go:42: 06:51:28 | red-metrics/1-install-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/red-metrics/check_user_workload_monitoring.sh] logger.go:42: 06:51:28 | red-metrics/1-install-workload-monitoring | test step completed 1-install-workload-monitoring logger.go:42: 06:51:28 | red-metrics/2-install-otel-collector | starting test step 2-install-otel-collector logger.go:42: 06:51:28 | red-metrics/2-install-otel-collector | OpenTelemetryCollector:kuttl-test-ample-rattler/otel created logger.go:42: 06:51:28 | red-metrics/2-install-otel-collector | PodMonitor:kuttl-test-ample-rattler/otel-collector created === CONT kuttl/harness/monitoring logger.go:42: 06:51:29 | monitoring | Ignoring check_metrics.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$ logger.go:42: 06:51:29 | monitoring | Ignoring check_user_workload_monitoring.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$ logger.go:42: 06:51:29 | monitoring | Creating namespace: kuttl-test-amazing-serval logger.go:42: 06:51:29 | monitoring/0-install-storage | starting test step 0-install-storage logger.go:42: 06:51:30 | monitoring/0-install-storage | Namespace:/kuttl-monitoring created logger.go:42: 06:51:30 | monitoring/0-install-storage | PersistentVolumeClaim:kuttl-monitoring/minio created logger.go:42: 06:51:30 | monitoring/0-install-storage | Deployment:kuttl-monitoring/minio created logger.go:42: 06:51:30 | monitoring/0-install-storage | Service:kuttl-monitoring/minio created logger.go:42: 06:51:30 | monitoring/0-install-storage | Secret:kuttl-monitoring/minio-secret created === CONT kuttl/harness/red-metrics logger.go:42: 06:51:33 | red-metrics/2-install-otel-collector | test step completed 2-install-otel-collector logger.go:42: 06:51:33 | red-metrics/3-install-tempo | starting test step 3-install-tempo logger.go:42: 06:51:33 | red-metrics/3-install-tempo | Secret:kuttl-test-ample-rattler/minio-test created logger.go:42: 06:51:33 | red-metrics/3-install-tempo | TempoStack:kuttl-test-ample-rattler/redmetrics created === CONT kuttl/harness/monitoring logger.go:42: 06:51:41 | monitoring/0-install-storage | test step completed 0-install-storage logger.go:42: 06:51:41 | monitoring/1-workload-monitoring | starting test step 1-workload-monitoring logger.go:42: 06:51:41 | monitoring/1-workload-monitoring | ConfigMap:openshift-monitoring/cluster-monitoring-config updated logger.go:42: 06:51:41 | monitoring/1-workload-monitoring | running command: [sh -c ./tests/e2e-openshift/monitoring/check_user_workload_monitoring.sh] logger.go:42: 06:51:42 | monitoring/1-workload-monitoring | test step completed 1-workload-monitoring logger.go:42: 06:51:42 | monitoring/2-install-tempostack | starting test step 2-install-tempostack logger.go:42: 06:51:42 | monitoring/2-install-tempostack | TempoStack:kuttl-monitoring/tempostack created === CONT kuttl/harness/multitenancy logger.go:42: 06:52:33 | multitenancy/1-install-tempo | test step completed 1-install-tempo logger.go:42: 06:52:33 | multitenancy/2-install-otelcol | starting test step 2-install-otelcol logger.go:42: 06:52:33 | multitenancy/2-install-otelcol | OpenTelemetryCollector:kuttl-multitenancy/dev created logger.go:42: 06:52:33 | multitenancy/2-install-otelcol | ClusterRole:/tempostack-traces-write created logger.go:42: 06:52:33 | multitenancy/2-install-otelcol | ClusterRoleBinding:/tempostack-traces created logger.go:42: 06:52:40 | multitenancy/2-install-otelcol | test step completed 2-install-otelcol logger.go:42: 06:52:40 | multitenancy/3-generate-traces | starting test step 3-generate-traces logger.go:42: 06:52:40 | multitenancy/3-generate-traces | Job:kuttl-multitenancy/generate-traces created logger.go:42: 06:52:44 | multitenancy/3-generate-traces | test step completed 3-generate-traces logger.go:42: 06:52:44 | multitenancy/4-verify-traces | starting test step 4-verify-traces logger.go:42: 06:52:44 | multitenancy/4-verify-traces | Job:kuttl-multitenancy/verify-traces created logger.go:42: 06:52:44 | multitenancy/4-verify-traces | Job:kuttl-multitenancy/verify-traces-traceql created === CONT kuttl/harness/monitoring logger.go:42: 06:52:55 | monitoring/2-install-tempostack | test step completed 2-install-tempostack logger.go:42: 06:52:55 | monitoring/3-generate-traces | starting test step 3-generate-traces logger.go:42: 06:52:55 | monitoring/3-generate-traces | Job:kuttl-test-amazing-serval/generate-traces created logger.go:42: 06:52:55 | monitoring/3-generate-traces | test step completed 3-generate-traces logger.go:42: 06:52:55 | monitoring/4-verify-traces | starting test step 4-verify-traces logger.go:42: 06:52:55 | monitoring/4-verify-traces | Job:kuttl-test-amazing-serval/verify-traces created === CONT kuttl/harness/multitenancy logger.go:42: 06:53:00 | multitenancy/4-verify-traces | test step completed 4-verify-traces logger.go:42: 06:53:00 | multitenancy | multitenancy events from ns kuttl-test-crisp-buzzard: logger.go:42: 06:53:00 | multitenancy | Deleting namespace: kuttl-test-crisp-buzzard === CONT kuttl/harness/monitoring logger.go:42: 06:53:05 | monitoring/4-verify-traces | test step completed 4-verify-traces logger.go:42: 06:53:05 | monitoring/5- | starting test step 5- logger.go:42: 06:53:05 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:05 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:05 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:06 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 2039 0 --:--:-- --:--:-- --:--:-- 2069 logger.go:42: 06:53:06 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:06 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:07 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:07 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:07 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:07 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3440 0 --:--:-- --:--:-- --:--:-- 3465 logger.go:42: 06:53:07 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:07 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:08 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:08 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:08 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:08 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3582 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:08 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:08 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:09 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:10 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:10 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:10 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3438 0 --:--:-- --:--:-- --:--:-- 3465 logger.go:42: 06:53:10 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:10 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:11 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:11 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:11 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:11 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 2599 0 --:--:-- --:--:-- --:--:-- 2614 logger.go:42: 06:53:11 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:11 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:12 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:13 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:13 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:13 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3680 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:53:13 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:13 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:14 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:14 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:14 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:14 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3717 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:53:14 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:14 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:15 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:16 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:16 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:16 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3645 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:53:16 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:16 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:17 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:17 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:17 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:17 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3432 0 --:--:-- --:--:-- --:--:-- 3386 logger.go:42: 06:53:17 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:17 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:18 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:18 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:18 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:19 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3452 0 --:--:-- --:--:-- --:--:-- 3465 logger.go:42: 06:53:19 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:19 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:20 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:20 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:20 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:20 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3708 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:53:20 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:20 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:21 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:21 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:21 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:21 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3619 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:21 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:21 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:22 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:23 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:23 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:23 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3862 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:53:23 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:23 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:24 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:24 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:24 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:24 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3621 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:24 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:24 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:25 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:26 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:26 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:26 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3586 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:26 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:26 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:27 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:27 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:27 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:27 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 2750 0 --:--:-- --:--:-- --:--:-- 2759 logger.go:42: 06:53:27 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:27 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:28 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:29 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:29 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:29 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3570 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:29 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:29 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:30 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:30 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:30 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:30 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3548 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:30 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:30 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:31 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:31 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:31 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:32 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3583 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:32 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:32 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:33 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:33 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:33 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:33 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3833 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:53:33 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:33 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:34 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:34 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:34 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:34 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3973 0 --:--:-- --:--:-- --:--:-- 4027 logger.go:42: 06:53:34 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:34 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:35 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:36 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:36 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:36 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3369 0 --:--:-- --:--:-- --:--:-- 3386 logger.go:42: 06:53:36 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:36 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:37 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:37 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:37 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:37 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3818 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:53:37 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:37 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:38 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:39 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:39 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:39 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3257 0 --:--:-- --:--:-- --:--:-- 3311 logger.go:42: 06:53:39 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:39 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:40 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:40 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:40 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:40 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3650 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:40 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:40 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:41 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:41 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:41 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:42 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3596 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:42 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:42 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:43 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:43 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:43 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:43 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3341 0 --:--:-- --:--:-- --:--:-- 3386 logger.go:42: 06:53:43 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:43 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:44 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:44 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:44 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:44 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3714 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:53:44 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:44 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:45 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:46 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:46 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:46 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3936 0 --:--:-- --:--:-- --:--:-- 4027 logger.go:42: 06:53:46 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:46 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:47 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:47 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:47 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:47 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3692 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:53:47 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:47 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:48 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:49 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:49 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:49 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3692 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:53:49 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:49 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:50 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:50 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:50 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:50 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3551 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:53:50 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:50 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:51 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:52 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:52 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:52 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3664 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:53:52 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:52 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:53 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:53 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:53 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:53 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3920 0 --:--:-- --:--:-- --:--:-- 4027 logger.go:42: 06:53:53 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:53 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:54 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:54 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:54 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:54 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3640 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:53:54 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:54 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:55 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:56 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:56 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:56 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3833 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:53:56 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:56 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:57 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:57 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:57 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:57 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3771 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:53:57 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:57 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:53:58 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:53:59 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:53:59 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:53:59 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3832 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:53:59 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:53:59 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:00 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:00 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:00 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:00 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3700 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:54:00 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:00 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:01 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:02 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:02 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:02 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3564 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:54:02 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:02 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:03 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:03 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:03 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:03 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3779 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:54:03 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:03 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:04 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:04 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:04 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:05 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3760 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:54:05 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:05 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:06 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:06 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:06 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:06 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3413 0 --:--:-- --:--:-- --:--:-- 3465 logger.go:42: 06:54:06 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:06 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:07 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:07 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:07 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:07 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3722 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:54:07 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:07 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:08 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:09 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:09 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:09 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3802 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:54:09 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:09 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:10 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:10 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:10 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:10 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3824 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:54:10 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:10 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:11 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:12 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:12 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:12 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3494 0 --:--:-- --:--:-- --:--:-- 3547 logger.go:42: 06:54:12 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:12 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:13 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:13 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:13 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:13 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3712 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:54:13 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:13 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:14 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:15 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:15 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:15 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3621 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:54:15 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:15 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:16 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:16 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:16 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:16 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3253 0 --:--:-- --:--:-- --:--:-- 3311 logger.go:42: 06:54:16 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:16 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:17 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:17 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:17 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:17 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3835 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:54:17 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:17 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:18 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:19 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:19 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:19 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3625 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:54:19 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:19 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:20 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:20 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:20 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:20 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3854 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:54:20 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:20 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:21 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:22 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:22 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:22 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3624 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:54:22 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:22 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:23 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:23 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:23 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:23 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3716 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:54:23 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:23 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:24 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:25 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:25 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:25 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3538 0 --:--:-- --:--:-- --:--:-- 3547 logger.go:42: 06:54:25 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:25 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:26 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:26 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:26 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:26 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3893 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:54:26 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:26 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:27 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:27 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:27 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:27 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3331 0 --:--:-- --:--:-- --:--:-- 3386 logger.go:42: 06:54:28 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:28 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:29 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:29 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:29 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:29 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3884 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:54:29 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:29 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:30 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:30 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:30 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:30 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3686 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:54:30 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:30 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:31 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:32 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:32 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:32 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3872 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:54:32 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:32 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:33 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] === CONT kuttl/harness/red-metrics logger.go:42: 06:54:33 | red-metrics/3-install-tempo | test step failed 3-install-tempo case.go:364: failed in step 3-install-tempo case.go:366: clusterrolebindings.rbac.authorization.k8s.io "tempo-redmetrics-cluster-monitoring-view" not found logger.go:42: 06:54:33 | red-metrics | red-metrics events from ns kuttl-test-ample-rattler: logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:36 +0000 UTC Normal ReplicaSet.apps minio-744f658d8d SuccessfulCreate Created pod: minio-744f658d8d-tmpfm replicaset-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:36 +0000 UTC Normal PersistentVolumeClaim minio WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:36 +0000 UTC Normal Deployment.apps minio ScalingReplicaSet Scaled up replica set minio-744f658d8d to 1 deployment-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:36 +0000 UTC Normal PersistentVolumeClaim minio Provisioning External provisioner is provisioning volume for claim "kuttl-test-ample-rattler/minio" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:36 +0000 UTC Normal PersistentVolumeClaim minio ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:40 +0000 UTC Normal Pod minio-744f658d8d-tmpfm Binding Scheduled Successfully assigned kuttl-test-ample-rattler/minio-744f658d8d-tmpfm to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:40 +0000 UTC Normal PersistentVolumeClaim minio ProvisioningSucceeded Successfully provisioned volume pvc-fb74794c-fdd1-4989-9978-ce6e4d06bdb6 ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:42 +0000 UTC Normal Pod minio-744f658d8d-tmpfm SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-fb74794c-fdd1-4989-9978-ce6e4d06bdb6" attachdetach-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:49 +0000 UTC Normal Pod minio-744f658d8d-tmpfm AddedInterface Add eth0 [10.131.0.36/23] from ovn-kubernetes logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:49 +0000 UTC Normal Pod minio-744f658d8d-tmpfm.spec.containers{minio} Pulling Pulling image "minio/minio" kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:51 +0000 UTC Normal Pod minio-744f658d8d-tmpfm.spec.containers{minio} Pulled Successfully pulled image "minio/minio" in 2.266s (2.266s including waiting) kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:51 +0000 UTC Normal Pod minio-744f658d8d-tmpfm.spec.containers{minio} Created Created container minio kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:50:51 +0000 UTC Normal Pod minio-744f658d8d-tmpfm.spec.containers{minio} Started Started container minio kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:28 +0000 UTC Normal Pod otel-collector-659ffb6d8b-fzzkz Binding Scheduled Successfully assigned kuttl-test-ample-rattler/otel-collector-659ffb6d8b-fzzkz to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:28 +0000 UTC Warning ReplicaSet.apps otel-collector-659ffb6d8b FailedCreate Error creating: pods "otel-collector-659ffb6d8b-" is forbidden: error looking up service account kuttl-test-ample-rattler/otel-collector: serviceaccount "otel-collector" not found replicaset-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:28 +0000 UTC Normal ReplicaSet.apps otel-collector-659ffb6d8b SuccessfulCreate Created pod: otel-collector-659ffb6d8b-fzzkz replicaset-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:28 +0000 UTC Normal PodDisruptionBudget.policy otel-collector NoPods No matching pods found controllermanager logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:28 +0000 UTC Normal Deployment.apps otel-collector ScalingReplicaSet Scaled up replica set otel-collector-659ffb6d8b to 1 deployment-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:29 +0000 UTC Normal Pod otel-collector-659ffb6d8b-fzzkz AddedInterface Add eth0 [10.129.2.45/23] from ovn-kubernetes logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:29 +0000 UTC Normal Pod otel-collector-659ffb6d8b-fzzkz.spec.containers{otc-container} Pulling Pulling image "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.83.0" kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:29 +0000 UTC Normal OpenTelemetryCollector.opentelemetry.io otel Info applied status changes opentelemetry-operator logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:29 +0000 UTC Warning OpenTelemetryCollector.opentelemetry.io otel Error failed to create objects for otel: Operation cannot be fulfilled on deployments.apps "otel-collector": the object has been modified; please apply your changes to the latest version and try again opentelemetry-operator logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:32 +0000 UTC Normal Pod otel-collector-659ffb6d8b-fzzkz.spec.containers{otc-container} Pulled Successfully pulled image "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.83.0" in 2.783s (2.783s including waiting) kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:32 +0000 UTC Normal Pod otel-collector-659ffb6d8b-fzzkz.spec.containers{otc-container} Created Created container otc-container kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:32 +0000 UTC Normal Pod otel-collector-659ffb6d8b-fzzkz.spec.containers{otc-container} Started Started container otc-container kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal PersistentVolumeClaim data-tempo-redmetrics-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding persistentvolume-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal PersistentVolumeClaim data-tempo-redmetrics-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-ample-rattler/data-tempo-redmetrics-ingester-0" ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal PersistentVolumeClaim data-tempo-redmetrics-ingester-0 ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. persistentvolume-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-compactor-7c559d55d-qpfsc Binding Scheduled Successfully assigned kuttl-test-ample-rattler/tempo-redmetrics-compactor-7c559d55d-qpfsc to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-compactor-7c559d55d-qpfsc AddedInterface Add eth0 [10.131.0.45/23] from ovn-kubernetes logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-compactor-7c559d55d-qpfsc.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-compactor-7c559d55d-qpfsc.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-compactor-7c559d55d-qpfsc.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal ReplicaSet.apps tempo-redmetrics-compactor-7c559d55d SuccessfulCreate Created pod: tempo-redmetrics-compactor-7c559d55d-qpfsc replicaset-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Deployment.apps tempo-redmetrics-compactor ScalingReplicaSet Scaled up replica set tempo-redmetrics-compactor-7c559d55d to 1 deployment-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-distributor-76878c6cc6-78pz6 Binding Scheduled Successfully assigned kuttl-test-ample-rattler/tempo-redmetrics-distributor-76878c6cc6-78pz6 to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-distributor-76878c6cc6-78pz6 AddedInterface Add eth0 [10.128.2.44/23] from ovn-kubernetes logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-distributor-76878c6cc6-78pz6.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-distributor-76878c6cc6-78pz6.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-distributor-76878c6cc6-78pz6.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal ReplicaSet.apps tempo-redmetrics-distributor-76878c6cc6 SuccessfulCreate Created pod: tempo-redmetrics-distributor-76878c6cc6-78pz6 replicaset-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Deployment.apps tempo-redmetrics-distributor ScalingReplicaSet Scaled up replica set tempo-redmetrics-distributor-76878c6cc6 to 1 deployment-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal StatefulSet.apps tempo-redmetrics-ingester SuccessfulCreate create Claim data-tempo-redmetrics-ingester-0 Pod tempo-redmetrics-ingester-0 in StatefulSet tempo-redmetrics-ingester success statefulset-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal StatefulSet.apps tempo-redmetrics-ingester SuccessfulCreate create Pod tempo-redmetrics-ingester-0 in StatefulSet tempo-redmetrics-ingester successful statefulset-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-querier-54868859d7-kcxvz Binding Scheduled Successfully assigned kuttl-test-ample-rattler/tempo-redmetrics-querier-54868859d7-kcxvz to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-querier-54868859d7-kcxvz AddedInterface Add eth0 [10.129.2.46/23] from ovn-kubernetes logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-querier-54868859d7-kcxvz.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-querier-54868859d7-kcxvz.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-querier-54868859d7-kcxvz.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal ReplicaSet.apps tempo-redmetrics-querier-54868859d7 SuccessfulCreate Created pod: tempo-redmetrics-querier-54868859d7-kcxvz replicaset-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Deployment.apps tempo-redmetrics-querier ScalingReplicaSet Scaled up replica set tempo-redmetrics-querier-54868859d7 to 1 deployment-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-query-frontend-7b574685d6-wdsjk Binding Scheduled Successfully assigned kuttl-test-ample-rattler/tempo-redmetrics-query-frontend-7b574685d6-wdsjk to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-query-frontend-7b574685d6-wdsjk AddedInterface Add eth0 [10.128.2.45/23] from ovn-kubernetes logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-query-frontend-7b574685d6-wdsjk.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-query-frontend-7b574685d6-wdsjk.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-query-frontend-7b574685d6-wdsjk.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-query-frontend-7b574685d6-wdsjk.spec.containers{tempo-query} Pulled Container image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:5402f3af7bdee45bb3d514ed1054cdc2fb9475f6ca33ab513d226a3bd1bc46b9" already present on machine kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-query-frontend-7b574685d6-wdsjk.spec.containers{tempo-query} Created Created container tempo-query kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Pod tempo-redmetrics-query-frontend-7b574685d6-wdsjk.spec.containers{tempo-query} Started Started container tempo-query kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal ReplicaSet.apps tempo-redmetrics-query-frontend-7b574685d6 SuccessfulCreate Created pod: tempo-redmetrics-query-frontend-7b574685d6-wdsjk replicaset-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:34 +0000 UTC Normal Deployment.apps tempo-redmetrics-query-frontend ScalingReplicaSet Scaled up replica set tempo-redmetrics-query-frontend-7b574685d6 to 1 deployment-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:38 +0000 UTC Normal PersistentVolumeClaim data-tempo-redmetrics-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-493bf86d-afc0-4955-88ad-6ab169435eb1 ebs.csi.aws.com_aws-ebs-csi-driver-controller-5c6944759d-h8v5q_e35e5270-e944-44ba-b3bb-b51692ab67c1 logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:39 +0000 UTC Normal Pod tempo-redmetrics-ingester-0 Binding Scheduled Successfully assigned kuttl-test-ample-rattler/tempo-redmetrics-ingester-0 to ip-10-0-42-224.us-east-2.compute.internal default-scheduler logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:40 +0000 UTC Normal Pod tempo-redmetrics-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-493bf86d-afc0-4955-88ad-6ab169435eb1" attachdetach-controller logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:49 +0000 UTC Normal Pod tempo-redmetrics-ingester-0 AddedInterface Add eth0 [10.129.2.47/23] from ovn-kubernetes logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:49 +0000 UTC Normal Pod tempo-redmetrics-ingester-0.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:d0a526c0721596880eef4f38ef772ec7a914364cdc7165eaa35f6ba4068a5e7a" already present on machine kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:49 +0000 UTC Normal Pod tempo-redmetrics-ingester-0.spec.containers{tempo} Created Created container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:49 +0000 UTC Normal Pod tempo-redmetrics-ingester-0.spec.containers{tempo} Started Started container tempo kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:51:54 +0000 UTC Warning Pod tempo-redmetrics-compactor-7c559d55d-qpfsc.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet logger.go:42: 06:54:33 | red-metrics | 2023-12-11 06:52:09 +0000 UTC Warning Pod tempo-redmetrics-ingester-0.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 kubelet === CONT kuttl/harness/monitoring logger.go:42: 06:54:33 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:33 | monitoring/5- | Dload Upload Total Spent Left Speed === CONT kuttl/harness/red-metrics logger.go:42: 06:54:33 | red-metrics | Deleting namespace: kuttl-test-ample-rattler === CONT kuttl/harness/monitoring logger.go:42: 06:54:33 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3257 0 --:--:-- --:--:-- --:--:-- 3311 logger.go:42: 06:54:33 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:33 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:34 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:35 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:35 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:35 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3438 0 --:--:-- --:--:-- --:--:-- 3465 logger.go:42: 06:54:35 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:35 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:36 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:36 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:36 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:36 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3339 0 --:--:-- --:--:-- --:--:-- 3386 logger.go:42: 06:54:36 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:36 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:37 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:38 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:38 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:38 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3457 0 --:--:-- --:--:-- --:--:-- 3465 logger.go:42: 06:54:38 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:38 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:39 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:39 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:39 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:39 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3614 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:54:39 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:39 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:40 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:40 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:40 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:40 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3658 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:54:40 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:40 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:41 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:42 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:42 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:42 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3236 0 --:--:-- --:--:-- --:--:-- 3311 logger.go:42: 06:54:42 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:42 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:43 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:43 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:43 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:43 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3699 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:54:43 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:43 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:44 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:45 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:45 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:45 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3557 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:54:45 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:45 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:46 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:46 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:46 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:46 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3635 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:54:46 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:46 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:47 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:48 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:48 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:48 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3733 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:54:48 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:48 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:49 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:49 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:49 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:49 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3646 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:54:49 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:49 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:50 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:50 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:50 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:50 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3233 0 --:--:-- --:--:-- --:--:-- 3239 logger.go:42: 06:54:50 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:50 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:51 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:52 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:52 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:52 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3895 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:54:52 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:52 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:53 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:53 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:53 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:53 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3556 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:54:53 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:53 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:54 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:55 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:55 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:55 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3614 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:54:55 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:55 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:56 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:56 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:56 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:56 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3764 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:54:56 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:56 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:57 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:58 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:58 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:58 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3749 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:54:58 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:58 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:54:59 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:54:59 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:54:59 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:54:59 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3610 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:54:59 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:54:59 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:00 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:00 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:00 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:00 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3734 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:00 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:00 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:01 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:02 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:02 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:02 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3733 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:02 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:02 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:03 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:03 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:03 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:03 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3383 0 --:--:-- --:--:-- --:--:-- 3465 logger.go:42: 06:55:03 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:03 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:04 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:05 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:05 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:05 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3662 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:05 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:05 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:06 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:06 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:06 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:06 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3060 0 --:--:-- --:--:-- --:--:-- 3104 logger.go:42: 06:55:06 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:06 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:07 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:08 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:08 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:08 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3811 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:08 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:08 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:09 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:09 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:09 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:09 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3811 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:09 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:09 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:10 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:10 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:10 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:11 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3790 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:11 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:11 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:12 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:12 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:12 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:12 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3230 0 --:--:-- --:--:-- --:--:-- 3239 logger.go:42: 06:55:12 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:12 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:13 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:13 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:13 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:13 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3369 0 --:--:-- --:--:-- --:--:-- 3386 logger.go:42: 06:55:13 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:13 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:14 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:15 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:15 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:15 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3668 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:15 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:15 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:16 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:16 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:16 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:16 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3738 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:16 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:16 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:17 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:18 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:18 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:18 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3745 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:18 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:18 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:19 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:19 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:19 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:19 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3690 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:19 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:19 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:20 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:21 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:21 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:21 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3589 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:55:21 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:21 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:22 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:22 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:22 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:22 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3672 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:22 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:22 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:23 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:23 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:23 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:23 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3642 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:23 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:23 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:24 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:25 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:25 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:25 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3778 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:25 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:25 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:26 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:26 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:26 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:26 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3678 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:26 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:26 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:27 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:28 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:28 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:28 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3831 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:55:28 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:28 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:29 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:29 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:29 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:29 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3608 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:55:29 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:29 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:30 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:31 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:31 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:31 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3752 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:31 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:31 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:32 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:32 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:32 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:32 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3688 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:32 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:32 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:33 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:33 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:33 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:33 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3825 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:55:33 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:33 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:34 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:35 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:35 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:35 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3880 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:55:35 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:35 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:36 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:36 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:36 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:36 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3606 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:55:36 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:36 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:37 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:38 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:38 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:38 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3563 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:55:38 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:38 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:39 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:39 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:39 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:39 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3754 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:39 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:39 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:40 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:41 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:41 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:41 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3687 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:41 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:41 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:42 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:42 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:42 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:42 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3712 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:42 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:42 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:43 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:43 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:43 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:43 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3668 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:43 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:43 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:44 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:45 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:45 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:45 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3791 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:45 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:45 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:46 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:46 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:46 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:46 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3865 0 --:--:-- --:--:-- --:--:-- 3921 logger.go:42: 06:55:46 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:46 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:47 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:48 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:48 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:48 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3757 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:48 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:48 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:49 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:49 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:49 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:49 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3019 0 --:--:-- --:--:-- --:--:-- 2980 logger.go:42: 06:55:49 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:49 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:50 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:50 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:50 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:51 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3761 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:51 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:51 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:52 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:52 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:52 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:52 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3275 0 --:--:-- --:--:-- --:--:-- 3311 logger.go:42: 06:55:52 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:52 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:53 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:53 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:53 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:53 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3572 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:55:53 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:53 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:54 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:55 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:55 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:55 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3664 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:55 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:55 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:56 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:56 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:56 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:56 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3792 0 --:--:-- --:--:-- --:--:-- 3820 logger.go:42: 06:55:56 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:56 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:57 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:58 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:58 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:58 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3531 0 --:--:-- --:--:-- --:--:-- 3547 logger.go:42: 06:55:58 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:58 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:55:59 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:55:59 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:55:59 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:55:59 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3743 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:55:59 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:55:59 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:56:00 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:56:00 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:56:00 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:56:01 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3579 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:56:01 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:56:01 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:56:02 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:56:02 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:56:02 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:56:02 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3961 0 --:--:-- --:--:-- --:--:-- 4027 logger.go:42: 06:56:02 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:56:02 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:56:03 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:56:03 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:56:03 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:56:03 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3621 0 --:--:-- --:--:-- --:--:-- 3634 logger.go:42: 06:56:03 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:56:03 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:56:04 | monitoring/5- | running command: [sh -c ./tests/e2e-openshift/monitoring/check_metrics.sh] logger.go:42: 06:56:05 | monitoring/5- | % Total % Received % Xferd Average Speed Time Time Time Current logger.go:42: 06:56:05 | monitoring/5- | Dload Upload Total Spent Left Speed logger.go:42: 06:56:05 | monitoring/5- | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 149 100 149 0 0 3704 0 --:--:-- --:--:-- --:--:-- 3725 logger.go:42: 06:56:05 | monitoring/5- | parse error: Invalid numeric literal at line 1, column 10 logger.go:42: 06:56:05 | monitoring/5- | No metric 'tempo_request_duration_seconds_count' with value present. Exiting with status 1. logger.go:42: 06:56:06 | monitoring/5- | test step failed 5- case.go:364: failed in step 5- case.go:366: exit status 1 logger.go:42: 06:56:06 | monitoring | monitoring events from ns kuttl-test-amazing-serval: logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:55 +0000 UTC Normal Pod generate-traces-4bbn2 Binding Scheduled Successfully assigned kuttl-test-amazing-serval/generate-traces-4bbn2 to ip-10-0-44-120.us-east-2.compute.internal default-scheduler logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:55 +0000 UTC Normal Pod generate-traces-4bbn2 AddedInterface Add eth0 [10.128.2.49/23] from ovn-kubernetes logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:55 +0000 UTC Normal Pod generate-traces-4bbn2.spec.containers{telemetrygen} Pulling Pulling image "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest" kubelet logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:55 +0000 UTC Normal Job.batch generate-traces SuccessfulCreate Created pod: generate-traces-4bbn2 job-controller logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:55 +0000 UTC Normal Pod verify-traces-l8xz2 Binding Scheduled Successfully assigned kuttl-test-amazing-serval/verify-traces-l8xz2 to ip-10-0-96-88.us-east-2.compute.internal default-scheduler logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:55 +0000 UTC Normal Pod verify-traces-l8xz2 AddedInterface Add eth0 [10.131.0.49/23] from ovn-kubernetes logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:55 +0000 UTC Normal Pod verify-traces-l8xz2.spec.containers{verify-traces} Pulling Pulling image "registry.access.redhat.com/ubi9/ubi:9.1" kubelet logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:55 +0000 UTC Normal Job.batch verify-traces SuccessfulCreate Created pod: verify-traces-l8xz2 job-controller logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:56 +0000 UTC Normal Pod generate-traces-4bbn2.spec.containers{telemetrygen} Pulled Successfully pulled image "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest" in 960ms (960ms including waiting) kubelet logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:56 +0000 UTC Normal Pod generate-traces-4bbn2.spec.containers{telemetrygen} Created Created container telemetrygen kubelet logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:52:56 +0000 UTC Normal Pod generate-traces-4bbn2.spec.containers{telemetrygen} Started Started container telemetrygen kubelet logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:53:02 +0000 UTC Normal Pod verify-traces-l8xz2.spec.containers{verify-traces} Pulled Successfully pulled image "registry.access.redhat.com/ubi9/ubi:9.1" in 7.115s (7.115s including waiting) kubelet logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:53:02 +0000 UTC Normal Pod verify-traces-l8xz2.spec.containers{verify-traces} Created Created container verify-traces kubelet logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:53:02 +0000 UTC Normal Pod verify-traces-l8xz2.spec.containers{verify-traces} Started Started container verify-traces kubelet logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:53:05 +0000 UTC Normal Job.batch verify-traces Completed Job completed job-controller logger.go:42: 06:56:06 | monitoring | 2023-12-11 06:55:59 +0000 UTC Normal Job.batch generate-traces Completed Job completed job-controller logger.go:42: 06:56:06 | monitoring | Deleting namespace: kuttl-test-amazing-serval === CONT kuttl harness.go:405: run tests finished harness.go:513: cleaning up harness.go:570: removing temp folder: "" --- FAIL: kuttl (455.63s) --- FAIL: kuttl/harness (0.00s) --- PASS: kuttl/harness/gateway (50.60s) --- PASS: kuttl/harness/generate (59.98s) --- PASS: kuttl/harness/reconcile (118.60s) --- PASS: kuttl/harness/compatibility (133.40s) --- PASS: kuttl/harness/receivers-tls (145.07s) --- PASS: kuttl/harness/route (38.99s) --- PASS: kuttl/harness/custom-ca (118.86s) --- PASS: kuttl/harness/receivers-mtls (147.88s) --- PASS: kuttl/harness/multitenancy (124.23s) --- FAIL: kuttl/harness/red-metrics (272.38s) --- FAIL: kuttl/harness/monitoring (282.78s) FAIL