/bin/bash: line 18: RUN: command not found Cloning into '/tmp/tempo-tests'... Switched to a new branch 'downstream-release' test -s /tmp/tempo-tests/bin/controller-gen-v0.9.2 || @[ -f /tmp/tempo-tests/bin/controller-gen-v0.9.2 ] || { set -e ; TMP_DIR=$(mktemp -d) ; cd $TMP_DIR ; go mod init tmp ; echo "Downloading sigs.k8s.io/controller-tools/cmd/controller-gen" ; go get -d sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.2 ; GOBIN=/tmp/tempo-tests/bin go install -mod=mod sigs.k8s.io/controller-tools/cmd/controller-gen ; APP=$(echo "/tmp/tempo-tests/bin/controller-gen") ; APP_NAME=$(echo "$APP-v0.9.2") ; mv "$APP" "$APP_NAME" ; rm -rf $TMP_DIR ; } bash: line 1: @[: command not found go: creating new go.mod: module tmp Downloading sigs.k8s.io/controller-tools/cmd/controller-gen go: downloading sigs.k8s.io/controller-tools v0.9.2 go: downloading github.com/spf13/cobra v1.4.0 go: downloading golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading k8s.io/api v0.24.0 go: downloading k8s.io/apimachinery v0.24.0 go: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b go: downloading k8s.io/apiextensions-apiserver v0.24.0 go: downloading sigs.k8s.io/yaml v1.3.0 go: downloading github.com/gobuffalo/flect v0.2.5 go: downloading github.com/fatih/color v1.12.0 go: downloading github.com/inconshreveable/mousetrap v1.0.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/mattn/go-colorable v0.1.8 go: downloading github.com/mattn/go-isatty v0.0.12 go: downloading golang.org/x/sys v0.0.0-20220209214540-3681064d5158 go: downloading k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading k8s.io/klog/v2 v2.60.1 go: downloading github.com/google/gofuzz v1.1.0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.1 go: downloading sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 go: downloading golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/go-logr/logr v1.2.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading golang.org/x/text v0.3.7 go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 go: added github.com/fatih/color v1.12.0 go: added github.com/go-logr/logr v1.2.0 go: added github.com/gobuffalo/flect v0.2.5 go: added github.com/gogo/protobuf v1.3.2 go: added github.com/google/gofuzz v1.1.0 go: added github.com/inconshreveable/mousetrap v1.0.0 go: added github.com/json-iterator/go v1.1.12 go: added github.com/mattn/go-colorable v0.1.8 go: added github.com/mattn/go-isatty v0.0.12 go: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: added github.com/modern-go/reflect2 v1.0.2 go: added github.com/spf13/cobra v1.4.0 go: added github.com/spf13/pflag v1.0.5 go: added golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 go: added golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd go: added golang.org/x/sys v0.0.0-20220209214540-3681064d5158 go: added golang.org/x/text v0.3.7 go: added golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717 go: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 go: added gopkg.in/inf.v0 v0.9.1 go: added gopkg.in/yaml.v2 v2.4.0 go: added gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b go: added k8s.io/api v0.24.0 go: added k8s.io/apiextensions-apiserver v0.24.0 go: added k8s.io/apimachinery v0.24.0 go: added k8s.io/klog/v2 v2.60.1 go: added k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 go: added sigs.k8s.io/controller-tools v0.9.2 go: added sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 go: added sigs.k8s.io/structured-merge-diff/v4 v4.2.1 go: added sigs.k8s.io/yaml v1.3.0 /tmp/tempo-tests/bin/controller-gen-v0.9.2 object:headerFile="hack/boilerplate.go.txt" paths="./..." go fmt ./... CGO_ENABLED=0 go build -o bin/manager -ldflags "-X github.com/grafana/tempo-operator/internal/version.buildDate=2023-09-11T06:44:52Z -X github.com/grafana/tempo-operator/internal/version.revision="4b4aa02da71e1a33f8a3b2bf0739036011d58a3e" -X github.com/grafana/tempo-operator/internal/version.operatorVersion=0.2.0 -X github.com/grafana/tempo-operator/internal/version.tempoVersion=2.1.1 -X github.com/grafana/tempo-operator/internal/version.tempoQueryVersion=2.1.1" main.go 2023/09/11 06:45:09 kutt-test config testdirs is overridden with args: [ tests/e2e, tests/e2e-openshift ] === RUN kuttl harness.go:462: starting setup harness.go:252: running tests using configured kubeconfig. harness.go:275: Successful connection to cluster at: https://api.ci-op-jm3pjq6q-39f3d.cspilp.interop.ccitredhat.com:6443 2023/09/11 06:45:10 Namespace:/minio created 2023/09/11 06:45:10 PersistentVolumeClaim:minio/minio created 2023/09/11 06:45:10 Deployment:minio/minio created 2023/09/11 06:45:10 Service:minio/minio created 2023/09/11 06:45:10 Secret:minio/minio-test created harness.go:360: running tests harness.go:73: going to run test suite with timeout of 180 seconds for each step harness.go:372: testsuite: tests/e2e has 3 tests harness.go:73: going to run test suite with timeout of 180 seconds for each step harness.go:372: testsuite: tests/e2e-openshift has 1 tests === RUN kuttl/harness === RUN kuttl/harness/gateway === PAUSE kuttl/harness/gateway === RUN kuttl/harness/generate === PAUSE kuttl/harness/generate === RUN kuttl/harness/reconcile === PAUSE kuttl/harness/reconcile === RUN kuttl/harness/route === PAUSE kuttl/harness/route === CONT kuttl/harness/gateway === CONT kuttl/harness/reconcile === CONT kuttl/harness/route === CONT kuttl/harness/generate logger.go:42: 06:45:10 | generate | Ignoring .gitignore as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$ logger.go:42: 06:45:10 | generate | Ignoring cr.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$ === CONT kuttl/harness/route logger.go:42: 06:45:10 | route | Creating namespace: kuttl-test-winning-alien === CONT kuttl/harness/generate logger.go:42: 06:45:10 | generate | Creating namespace: kuttl-test-steady-earwig === CONT kuttl/harness/reconcile logger.go:42: 06:45:10 | reconcile | Creating namespace: kuttl-test-upright-cockatoo === CONT kuttl/harness/gateway logger.go:42: 06:45:10 | gateway | Creating namespace: kuttl-test-warm-locust === CONT kuttl/harness/generate logger.go:42: 06:45:10 | generate/0-storage-secret | starting test step 0-storage-secret === CONT kuttl/harness/gateway logger.go:42: 06:45:10 | gateway/0-install | starting test step 0-install === CONT kuttl/harness/reconcile logger.go:42: 06:45:10 | reconcile/1-install | starting test step 1-install === CONT kuttl/harness/route logger.go:42: 06:45:10 | route/0-install | starting test step 0-install === CONT kuttl/harness/generate logger.go:42: 06:45:10 | generate/0-storage-secret | Secret:kuttl-test-steady-earwig/minio-test created logger.go:42: 06:45:10 | generate/0-storage-secret | test step completed 0-storage-secret logger.go:42: 06:45:10 | generate/1-generate | starting test step 1-generate logger.go:42: 06:45:10 | generate/1-generate | running command: [../../../bin/manager generate --config ../../../config/overlays/community/controller_manager_config.yaml --cr cr.yaml --output generated.yaml] === CONT kuttl/harness/route logger.go:42: 06:45:10 | route/0-install | Secret:kuttl-test-winning-alien/minio-test created === CONT kuttl/harness/generate logger.go:42: 06:45:10 | generate/1-generate | running command: [kubectl apply -f generated.yaml --namespace kuttl-test-steady-earwig] === CONT kuttl/harness/gateway logger.go:42: 06:45:10 | gateway/0-install | ConfigMap:kuttl-test-warm-locust/dex-config created === CONT kuttl/harness/reconcile logger.go:42: 06:45:10 | reconcile/1-install | Secret:kuttl-test-upright-cockatoo/minio-test created === CONT kuttl/harness/gateway logger.go:42: 06:45:10 | gateway/0-install | Deployment:kuttl-test-warm-locust/dex created === CONT kuttl/harness/route logger.go:42: 06:45:10 | route/0-install | TempoStack:kuttl-test-winning-alien/simplest created === CONT kuttl/harness/reconcile logger.go:42: 06:45:10 | reconcile/1-install | TempoStack:kuttl-test-upright-cockatoo/simplest created === CONT kuttl/harness/gateway logger.go:42: 06:45:10 | gateway/0-install | Service:kuttl-test-warm-locust/dex created === CONT kuttl/harness/route logger.go:42: 06:45:12 | route/0-install | test step completed 0-install logger.go:42: 06:45:12 | route | route events from ns kuttl-test-winning-alien: logger.go:42: 06:45:12 | route | 2023-09-11 06:45:11 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding logger.go:42: 06:45:12 | route | 2023-09-11 06:45:11 +0000 UTC Normal Pod tempo-simplest-distributor-7f85ffbc59-44svf Binding Scheduled Successfully assigned kuttl-test-winning-alien/tempo-simplest-distributor-7f85ffbc59-44svf to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:45:12 | route | 2023-09-11 06:45:11 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-7f85ffbc59 SuccessfulCreate Created pod: tempo-simplest-distributor-7f85ffbc59-44svf logger.go:42: 06:45:12 | route | 2023-09-11 06:45:11 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-7f85ffbc59 to 1 logger.go:42: 06:45:12 | route | 2023-09-11 06:45:11 +0000 UTC Normal Pod tempo-simplest-query-frontend-69b98cffdc-z4s5g Binding Scheduled Successfully assigned kuttl-test-winning-alien/tempo-simplest-query-frontend-69b98cffdc-z4s5g to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:45:12 | route | 2023-09-11 06:45:11 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-69b98cffdc SuccessfulCreate Created pod: tempo-simplest-query-frontend-69b98cffdc-z4s5g logger.go:42: 06:45:12 | route | 2023-09-11 06:45:11 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-69b98cffdc to 1 logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-winning-alien/data-tempo-simplest-ingester-0" logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ExternalProvisioning waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal Pod tempo-simplest-compactor-9c88b897-vh4ss Binding Scheduled Successfully assigned kuttl-test-winning-alien/tempo-simplest-compactor-9c88b897-vh4ss to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-9c88b897 SuccessfulCreate Created pod: tempo-simplest-compactor-9c88b897-vh4ss logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-9c88b897 to 1 logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal Pod tempo-simplest-distributor-7f85ffbc59-44svf AddedInterface Add eth0 [10.131.0.26/23] from ovn-kubernetes logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal Pod tempo-simplest-distributor-7f85ffbc59-44svf.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Claim data-tempo-simplest-ingester-0 Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester success logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal Pod tempo-simplest-querier-7cf5d7df94-n4gfj Binding Scheduled Successfully assigned kuttl-test-winning-alien/tempo-simplest-querier-7cf5d7df94-n4gfj to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-7cf5d7df94 SuccessfulCreate Created pod: tempo-simplest-querier-7cf5d7df94-n4gfj logger.go:42: 06:45:12 | route | 2023-09-11 06:45:12 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-7cf5d7df94 to 1 logger.go:42: 06:45:12 | route | Deleting namespace: kuttl-test-winning-alien === CONT kuttl/harness/generate logger.go:42: 06:45:12 | generate/1-generate | configmap/tempo-generated-tempo created logger.go:42: 06:45:12 | generate/1-generate | serviceaccount/tempo-generated-tempo created logger.go:42: 06:45:12 | generate/1-generate | deployment.apps/tempo-generated-tempo-distributor created logger.go:42: 06:45:12 | generate/1-generate | service/tempo-generated-tempo-distributor created logger.go:42: 06:45:13 | generate/1-generate | statefulset.apps/tempo-generated-tempo-ingester created logger.go:42: 06:45:13 | generate/1-generate | service/tempo-generated-tempo-ingester created logger.go:42: 06:45:13 | generate/1-generate | service/tempo-generated-tempo-gossip-ring created logger.go:42: 06:45:13 | generate/1-generate | deployment.apps/tempo-generated-tempo-query-frontend created logger.go:42: 06:45:13 | generate/1-generate | service/tempo-generated-tempo-query-frontend created logger.go:42: 06:45:13 | generate/1-generate | service/tempo-generated-tempo-query-frontend-discovery created logger.go:42: 06:45:13 | generate/1-generate | deployment.apps/tempo-generated-tempo-querier created logger.go:42: 06:45:13 | generate/1-generate | service/tempo-generated-tempo-querier created logger.go:42: 06:45:13 | generate/1-generate | deployment.apps/tempo-generated-tempo-compactor created === CONT kuttl/harness/reconcile logger.go:42: 06:45:13 | reconcile/1-install | test step completed 1-install logger.go:42: 06:45:13 | reconcile/2-delete-svc | starting test step 2-delete-svc logger.go:42: 06:45:13 | reconcile/2-delete-svc | running command: [kubectl delete service tempo-simplest-querier --namespace kuttl-test-upright-cockatoo] === CONT kuttl/harness/generate logger.go:42: 06:45:13 | generate/1-generate | service/tempo-generated-tempo-compactor created === CONT kuttl/harness/reconcile logger.go:42: 06:45:13 | reconcile/2-delete-svc | service "tempo-simplest-querier" deleted logger.go:42: 06:45:14 | reconcile/2-delete-svc | test step completed 2-delete-svc logger.go:42: 06:45:14 | reconcile/3-update-storage-secret | starting test step 3-update-storage-secret logger.go:42: 06:45:14 | reconcile/3-update-storage-secret | Secret:kuttl-test-upright-cockatoo/minio-test updated === CONT kuttl/harness/gateway logger.go:42: 06:45:14 | gateway/0-install | test step completed 0-install logger.go:42: 06:45:14 | gateway/1-install | starting test step 1-install logger.go:42: 06:45:14 | gateway/1-install | Secret:kuttl-test-warm-locust/minio-test created logger.go:42: 06:45:14 | gateway/1-install | Secret:kuttl-test-warm-locust/oidc-test created logger.go:42: 06:45:14 | gateway/1-install | test step completed 1-install logger.go:42: 06:45:14 | gateway/2-install | starting test step 2-install logger.go:42: 06:45:14 | gateway/2-install | TempoStack:kuttl-test-warm-locust/foo created === CONT kuttl/harness/generate logger.go:42: 06:45:43 | generate/1-generate | test step completed 1-generate logger.go:42: 06:45:43 | generate | generate events from ns kuttl-test-steady-earwig: logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:12 +0000 UTC Normal Pod tempo-generated-tempo-distributor-794b974855-v2d6r Binding Scheduled Successfully assigned kuttl-test-steady-earwig/tempo-generated-tempo-distributor-794b974855-v2d6r to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:12 +0000 UTC Normal ReplicaSet.apps tempo-generated-tempo-distributor-794b974855 SuccessfulCreate Created pod: tempo-generated-tempo-distributor-794b974855-v2d6r logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:12 +0000 UTC Normal Deployment.apps tempo-generated-tempo-distributor ScalingReplicaSet Scaled up replica set tempo-generated-tempo-distributor-794b974855 to 1 logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal PersistentVolumeClaim data-tempo-generated-tempo-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal PersistentVolumeClaim data-tempo-generated-tempo-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-steady-earwig/data-tempo-generated-tempo-ingester-0" logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal PersistentVolumeClaim data-tempo-generated-tempo-ingester-0 ExternalProvisioning waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal Pod tempo-generated-tempo-compactor-8674686d5d-l828n Binding Scheduled Successfully assigned kuttl-test-steady-earwig/tempo-generated-tempo-compactor-8674686d5d-l828n to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal Deployment.apps tempo-generated-tempo-compactor ScalingReplicaSet Scaled up replica set tempo-generated-tempo-compactor-8674686d5d to 1 logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal StatefulSet.apps tempo-generated-tempo-ingester SuccessfulCreate create Claim data-tempo-generated-tempo-ingester-0 Pod tempo-generated-tempo-ingester-0 in StatefulSet tempo-generated-tempo-ingester success logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal StatefulSet.apps tempo-generated-tempo-ingester SuccessfulCreate create Pod tempo-generated-tempo-ingester-0 in StatefulSet tempo-generated-tempo-ingester successful logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal Pod tempo-generated-tempo-querier-575cbcf5d8-jf5w7 Binding Scheduled Successfully assigned kuttl-test-steady-earwig/tempo-generated-tempo-querier-575cbcf5d8-jf5w7 to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal ReplicaSet.apps tempo-generated-tempo-querier-575cbcf5d8 SuccessfulCreate Created pod: tempo-generated-tempo-querier-575cbcf5d8-jf5w7 logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal Deployment.apps tempo-generated-tempo-querier ScalingReplicaSet Scaled up replica set tempo-generated-tempo-querier-575cbcf5d8 to 1 logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw Binding Scheduled Successfully assigned kuttl-test-steady-earwig/tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal ReplicaSet.apps tempo-generated-tempo-query-frontend-68bb5b9959 SuccessfulCreate Created pod: tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:13 +0000 UTC Normal Deployment.apps tempo-generated-tempo-query-frontend ScalingReplicaSet Scaled up replica set tempo-generated-tempo-query-frontend-68bb5b9959 to 1 logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-generated-tempo-compactor-8674686d5d-l828n AddedInterface Add eth0 [10.128.2.33/23] from ovn-kubernetes logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:14 +0000 UTC Normal ReplicaSet.apps tempo-generated-tempo-compactor-8674686d5d SuccessfulCreate Created pod: tempo-generated-tempo-compactor-8674686d5d-l828n logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw AddedInterface Add eth0 [10.128.2.30/23] from ovn-kubernetes logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw.spec.containers{tempo} Pulling Pulling image "docker.io/grafana/tempo:2.1.1" logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:15 +0000 UTC Normal Pod tempo-generated-tempo-compactor-8674686d5d-l828n.spec.containers{tempo} Pulling Pulling image "docker.io/grafana/tempo:2.1.1" logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:16 +0000 UTC Normal PersistentVolumeClaim data-tempo-generated-tempo-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-bb0c4fb4-3118-4ac9-ba76-bfe200b242e2 logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-generated-tempo-compactor-8674686d5d-l828n.spec.containers{tempo} Pulled Successfully pulled image "docker.io/grafana/tempo:2.1.1" in 670.462192ms (670.471202ms including waiting) logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-generated-tempo-compactor-8674686d5d-l828n.spec.containers{tempo} Created Created container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-generated-tempo-compactor-8674686d5d-l828n.spec.containers{tempo} Started Started container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw.spec.containers{tempo} Pulled Successfully pulled image "docker.io/grafana/tempo:2.1.1" in 2.053837074s (2.053847724s including waiting) logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw.spec.containers{tempo} Created Created container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw.spec.containers{tempo} Started Started container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:17 +0000 UTC Normal Pod tempo-generated-tempo-compactor-8674686d5d-l828n.spec.containers{tempo} Pulled Container image "docker.io/grafana/tempo:2.1.1" already present on machine logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:17 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0 Binding Scheduled Successfully assigned kuttl-test-steady-earwig/tempo-generated-tempo-ingester-0 to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:17 +0000 UTC Normal Pod tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw.spec.containers{tempo} Pulled Container image "docker.io/grafana/tempo:2.1.1" already present on machine logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:18 +0000 UTC Warning Pod tempo-generated-tempo-compactor-8674686d5d-l828n.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-generated-tempo-compactor-8674686d5d-l828n_kuttl-test-steady-earwig(9fca4625-4a4d-47b4-b13d-dacfb2a61eb9) logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:18 +0000 UTC Normal Pod tempo-generated-tempo-distributor-794b974855-v2d6r AddedInterface Add eth0 [10.131.0.28/23] from ovn-kubernetes logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:18 +0000 UTC Normal Pod tempo-generated-tempo-querier-575cbcf5d8-jf5w7 AddedInterface Add eth0 [10.128.2.32/23] from ovn-kubernetes logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:18 +0000 UTC Warning Pod tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-generated-tempo-query-frontend-68bb5b9959-b2bbw_kuttl-test-steady-earwig(c5d023b7-5e7a-4d9c-a313-ef42a612bd93) logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-bb0c4fb4-3118-4ac9-ba76-bfe200b242e2" logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-generated-tempo-querier-575cbcf5d8-jf5w7.spec.containers{tempo} Pulled Container image "docker.io/grafana/tempo:2.1.1" already present on machine logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-generated-tempo-querier-575cbcf5d8-jf5w7.spec.containers{tempo} Created Created container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-generated-tempo-querier-575cbcf5d8-jf5w7.spec.containers{tempo} Started Started container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-generated-tempo-distributor-794b974855-v2d6r.spec.containers{tempo} Pulling Pulling image "docker.io/grafana/tempo:2.1.1" logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:21 +0000 UTC Warning Pod tempo-generated-tempo-querier-575cbcf5d8-jf5w7.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-generated-tempo-querier-575cbcf5d8-jf5w7_kuttl-test-steady-earwig(6745fa8e-2aea-4e4d-b951-490cc83227b6) logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-generated-tempo-distributor-794b974855-v2d6r.spec.containers{tempo} Pulled Successfully pulled image "docker.io/grafana/tempo:2.1.1" in 2.084253432s (2.084262602s including waiting) logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-generated-tempo-distributor-794b974855-v2d6r.spec.containers{tempo} Created Created container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-generated-tempo-distributor-794b974855-v2d6r.spec.containers{tempo} Started Started container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0 AddedInterface Add eth0 [10.131.0.38/23] from ovn-kubernetes logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:23 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0.spec.containers{tempo} Pulled Container image "docker.io/grafana/tempo:2.1.1" already present on machine logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:23 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0.spec.containers{tempo} Created Created container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:23 +0000 UTC Normal Pod tempo-generated-tempo-ingester-0.spec.containers{tempo} Started Started container tempo logger.go:42: 06:45:43 | generate | 2023-09-11 06:45:24 +0000 UTC Warning Pod tempo-generated-tempo-ingester-0.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-generated-tempo-ingester-0_kuttl-test-steady-earwig(63447d12-2437-44c5-a9d4-cc6b79c6eb9a) logger.go:42: 06:45:43 | generate | Deleting namespace: kuttl-test-steady-earwig === CONT kuttl/harness/reconcile logger.go:42: 06:47:17 | reconcile/3-update-storage-secret | test step completed 3-update-storage-secret logger.go:42: 06:47:17 | reconcile/4-check-bucket-name | starting test step 4-check-bucket-name logger.go:42: 06:47:17 | reconcile/4-check-bucket-name | running command: [/bin/sh -c kubectl get --namespace kuttl-test-upright-cockatoo configmap tempo-simplest -o yaml | grep "endpoint: minio.minio.svc.cluster.local"] case.go:364: failed in step 4-check-bucket-name case.go:366: exit status 1 logger.go:42: 06:47:17 | reconcile | reconcile events from ns kuttl-test-upright-cockatoo: logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-upright-cockatoo/data-tempo-simplest-ingester-0" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ExternalProvisioning waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal Pod tempo-simplest-compactor-7f48b98f64-t6h5w Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-compactor-7f48b98f64-t6h5w to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-7f48b98f64 to 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal Pod tempo-simplest-distributor-794dd4b4cd-nblwg Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-distributor-794dd4b4cd-nblwg to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-794dd4b4cd SuccessfulCreate Created pod: tempo-simplest-distributor-794dd4b4cd-nblwg logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-794dd4b4cd to 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Claim data-tempo-simplest-ingester-0 Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester success logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulCreate create Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal Pod tempo-simplest-querier-5b7995487c-hkdnn Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-querier-5b7995487c-hkdnn to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-5b7995487c SuccessfulCreate Created pod: tempo-simplest-querier-5b7995487c-hkdnn logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-5b7995487c to 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal Pod tempo-simplest-query-frontend-55cff747fd-8l6xz Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-query-frontend-55cff747fd-8l6xz to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-55cff747fd SuccessfulCreate Created pod: tempo-simplest-query-frontend-55cff747fd-8l6xz logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:13 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-55cff747fd to 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-simplest-compactor-74d54cf849-rvcbw Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-compactor-74d54cf849-rvcbw to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-74d54cf849 SuccessfulCreate Created pod: tempo-simplest-compactor-74d54cf849-rvcbw logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-simplest-compactor-7f48b98f64-t6h5w AddedInterface Add eth0 [10.131.0.31/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-simplest-compactor-7f48b98f64-t6h5w.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-7f48b98f64 SuccessfulCreate Created pod: tempo-simplest-compactor-7f48b98f64-t6h5w logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled up replica set tempo-simplest-compactor-74d54cf849 to 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-simplest-distributor-7bd7f85cf9-mf6s2 Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-distributor-7bd7f85cf9-mf6s2 to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-7bd7f85cf9 SuccessfulCreate Created pod: tempo-simplest-distributor-7bd7f85cf9-mf6s2 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled up replica set tempo-simplest-distributor-7bd7f85cf9 to 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-simplest-querier-5b7995487c-hkdnn AddedInterface Add eth0 [10.131.0.30/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-simplest-querier-5b7995487c-hkdnn.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-simplest-querier-6795f5b4c8-bhkv6 Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-querier-6795f5b4c8-bhkv6 to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-6795f5b4c8 SuccessfulCreate Created pod: tempo-simplest-querier-6795f5b4c8-bhkv6 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled up replica set tempo-simplest-querier-6795f5b4c8 to 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-simplest-query-frontend-55cff747fd-8l6xz AddedInterface Add eth0 [10.131.0.29/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Pod tempo-simplest-query-frontend-b8fd78b69-kmq7v Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-query-frontend-b8fd78b69-kmq7v to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-b8fd78b69 SuccessfulCreate Created pod: tempo-simplest-query-frontend-b8fd78b69-kmq7v logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:14 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled up replica set tempo-simplest-query-frontend-b8fd78b69 to 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:15 +0000 UTC Normal Pod tempo-simplest-compactor-74d54cf849-rvcbw AddedInterface Add eth0 [10.131.0.33/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-simplest-distributor-7bd7f85cf9-mf6s2 AddedInterface Add eth0 [10.131.0.32/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-simplest-query-frontend-55cff747fd-8l6xz.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-simplest-query-frontend-b8fd78b69-kmq7v AddedInterface Add eth0 [10.128.2.35/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:17 +0000 UTC Normal PersistentVolumeClaim data-tempo-simplest-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-c8cde86d-cd63-41bf-83d2-e257fa5de4cc logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:17 +0000 UTC Normal Pod tempo-simplest-compactor-74d54cf849-rvcbw.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:17 +0000 UTC Normal Pod tempo-simplest-distributor-794dd4b4cd-nblwg AddedInterface Add eth0 [10.128.2.31/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:17 +0000 UTC Normal Pod tempo-simplest-ingester-0 Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-ingester-0 to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:17 +0000 UTC Normal Pod tempo-simplest-query-frontend-b8fd78b69-kmq7v.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-simplest-compactor-74d54cf849-rvcbw.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" in 2.14088517s (2.140898871s including waiting) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-simplest-compactor-7f48b98f64-t6h5w.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" in 5.35593729s (5.35594779s including waiting) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-simplest-distributor-794dd4b4cd-nblwg.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-simplest-distributor-7bd7f85cf9-mf6s2.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-simplest-querier-5b7995487c-hkdnn.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" in 5.520779363s (5.520786373s including waiting) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-simplest-querier-6795f5b4c8-bhkv6 AddedInterface Add eth0 [10.128.2.36/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-simplest-query-frontend-55cff747fd-8l6xz.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" in 3.749032254s (3.749040334s including waiting) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-compactor-74d54cf849-rvcbw.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-compactor-74d54cf849-rvcbw.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-compactor-74d54cf849-rvcbw.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-compactor-7f48b98f64-t6h5w.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-compactor-7f48b98f64-t6h5w.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-compactor-7f48b98f64-t6h5w.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-distributor-7bd7f85cf9-mf6s2.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" in 401.767033ms (401.777362ms including waiting) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-distributor-7bd7f85cf9-mf6s2.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-distributor-7bd7f85cf9-mf6s2.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-c8cde86d-cd63-41bf-83d2-e257fa5de4cc" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-querier-5b7995487c-hkdnn.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-querier-5b7995487c-hkdnn.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-querier-5b7995487c-hkdnn.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-query-frontend-55cff747fd-8l6xz.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-query-frontend-55cff747fd-8l6xz.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-simplest-query-frontend-55cff747fd-8l6xz.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:21 +0000 UTC Warning Pod tempo-simplest-compactor-74d54cf849-rvcbw.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-simplest-compactor-74d54cf849-rvcbw_kuttl-test-upright-cockatoo(852655b1-d80a-4dde-b794-8cbb358e3af0) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:21 +0000 UTC Warning Pod tempo-simplest-compactor-7f48b98f64-t6h5w.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-simplest-compactor-7f48b98f64-t6h5w_kuttl-test-upright-cockatoo(0a1f9791-c137-4922-b85f-4b1ff806476b) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:21 +0000 UTC Warning Pod tempo-simplest-querier-5b7995487c-hkdnn.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-simplest-querier-5b7995487c-hkdnn_kuttl-test-upright-cockatoo(82e80511-da66-41b5-b2b8-1899960fbd82) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:21 +0000 UTC Warning Pod tempo-simplest-query-frontend-55cff747fd-8l6xz.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-simplest-query-frontend-55cff747fd-8l6xz_kuttl-test-upright-cockatoo(e9a26bc1-4d13-4b68-9e58-ee50b85d00ba) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:23 +0000 UTC Normal Pod tempo-simplest-querier-6795f5b4c8-bhkv6.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-simplest-distributor-794dd4b4cd-nblwg.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" in 5.238437978s (5.238451557s including waiting) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-simplest-distributor-794dd4b4cd-nblwg.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-simplest-distributor-794dd4b4cd-nblwg.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-simplest-querier-6795f5b4c8-bhkv6.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" in 828.801148ms (828.809218ms including waiting) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-simplest-querier-6795f5b4c8-bhkv6.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-simplest-querier-6795f5b4c8-bhkv6.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-simplest-query-frontend-b8fd78b69-kmq7v.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" in 6.513888197s (6.513895557s including waiting) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-simplest-query-frontend-b8fd78b69-kmq7v.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-simplest-query-frontend-b8fd78b69-kmq7v.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:25 +0000 UTC Normal Pod tempo-simplest-ingester-0 AddedInterface Add eth0 [10.128.2.38/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:25 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:25 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:25 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:25 +0000 UTC Normal Pod tempo-simplest-querier-6795f5b4c8-bhkv6.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:25 +0000 UTC Normal Pod tempo-simplest-query-frontend-b8fd78b69-kmq7v.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:26 +0000 UTC Warning Pod tempo-simplest-querier-6795f5b4c8-bhkv6.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-simplest-querier-6795f5b4c8-bhkv6_kuttl-test-upright-cockatoo(6063c08b-d8a5-453b-9c3b-c07080cc24bd) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:26 +0000 UTC Warning Pod tempo-simplest-query-frontend-b8fd78b69-kmq7v.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-simplest-query-frontend-b8fd78b69-kmq7v_kuttl-test-upright-cockatoo(f54906e3-979e-4b97-9273-d1ddf9b41f11) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:27 +0000 UTC Warning Pod tempo-simplest-ingester-0.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-simplest-ingester-0_kuttl-test-upright-cockatoo(973e529c-f1f4-407b-ba68-e90effe20222) logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:44 +0000 UTC Normal Pod tempo-simplest-distributor-794dd4b4cd-nblwg.spec.containers{tempo} Killing Stopping container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:44 +0000 UTC Normal ReplicaSet.apps tempo-simplest-distributor-794dd4b4cd SuccessfulDelete Deleted pod: tempo-simplest-distributor-794dd4b4cd-nblwg logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:44 +0000 UTC Normal Deployment.apps tempo-simplest-distributor ScalingReplicaSet Scaled down replica set tempo-simplest-distributor-794dd4b4cd to 0 from 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:54 +0000 UTC Warning Pod tempo-simplest-compactor-74d54cf849-rvcbw.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:54 +0000 UTC Warning Pod tempo-simplest-compactor-7f48b98f64-t6h5w.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:45:54 +0000 UTC Warning Pod tempo-simplest-distributor-794dd4b4cd-nblwg.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:04 +0000 UTC Normal Deployment.apps tempo-simplest-query-frontend ScalingReplicaSet Scaled down replica set tempo-simplest-query-frontend-55cff747fd to 0 from 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:05 +0000 UTC Warning Pod tempo-simplest-ingester-0.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:05 +0000 UTC Normal Pod tempo-simplest-querier-5b7995487c-hkdnn.spec.containers{tempo} Killing Stopping container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:05 +0000 UTC Normal ReplicaSet.apps tempo-simplest-querier-5b7995487c SuccessfulDelete Deleted pod: tempo-simplest-querier-5b7995487c-hkdnn logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:05 +0000 UTC Normal Deployment.apps tempo-simplest-querier ScalingReplicaSet Scaled down replica set tempo-simplest-querier-5b7995487c to 0 from 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:05 +0000 UTC Normal Pod tempo-simplest-query-frontend-55cff747fd-8l6xz.spec.containers{tempo} Killing Stopping container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:05 +0000 UTC Normal ReplicaSet.apps tempo-simplest-query-frontend-55cff747fd SuccessfulDelete Deleted pod: tempo-simplest-query-frontend-55cff747fd-8l6xz logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:25 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Killing Stopping container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:25 +0000 UTC Normal StatefulSet.apps tempo-simplest-ingester SuccessfulDelete delete Pod tempo-simplest-ingester-0 in StatefulSet tempo-simplest-ingester successful logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:27 +0000 UTC Normal Pod tempo-simplest-ingester-0 Binding Scheduled Successfully assigned kuttl-test-upright-cockatoo/tempo-simplest-ingester-0 to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:36 +0000 UTC Normal Pod tempo-simplest-ingester-0 AddedInterface Add eth0 [10.128.2.40/23] from ovn-kubernetes logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:36 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:36 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Created Created container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:36 +0000 UTC Normal Pod tempo-simplest-ingester-0.spec.containers{tempo} Started Started container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:44 +0000 UTC Normal Pod tempo-simplest-compactor-7f48b98f64-t6h5w.spec.containers{tempo} Killing Stopping container tempo logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:44 +0000 UTC Normal ReplicaSet.apps tempo-simplest-compactor-7f48b98f64 SuccessfulDelete Deleted pod: tempo-simplest-compactor-7f48b98f64-t6h5w logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:44 +0000 UTC Normal Deployment.apps tempo-simplest-compactor ScalingReplicaSet Scaled down replica set tempo-simplest-compactor-7f48b98f64 to 0 from 1 logger.go:42: 06:47:17 | reconcile | 2023-09-11 06:46:56 +0000 UTC Warning Pod tempo-simplest-ingester-0.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 logger.go:42: 06:47:17 | reconcile | Deleting namespace: kuttl-test-upright-cockatoo === CONT kuttl/harness/gateway logger.go:42: 06:48:15 | gateway/2-install | test step failed 2-install case.go:364: failed in step 2-install case.go:366: --- Deployment:kuttl-test-warm-locust/tempo-foo-gateway +++ Deployment:kuttl-test-warm-locust/tempo-foo-gateway @@ -6,6 +6,199 @@ app.kubernetes.io/instance: foo app.kubernetes.io/managed-by: tempo-operator app.kubernetes.io/name: tempo + managedFields: + - apiVersion: apps/v1 + fieldsType: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: {} + f:app.kubernetes.io/component: {} + f:app.kubernetes.io/instance: {} + f:app.kubernetes.io/managed-by: {} + f:app.kubernetes.io/name: {} + f:ownerReferences: + .: {} + k:{"uid":"d565f9ec-2474-411e-9f9c-08cb07291ceb"}: {} + f:spec: + f:progressDeadlineSeconds: {} + f:replicas: {} + f:revisionHistoryLimit: {} + f:selector: {} + f:strategy: + f:rollingUpdate: + .: {} + f:maxSurge: {} + f:maxUnavailable: {} + f:type: {} + f:template: + f:metadata: + f:annotations: + .: {} + f:tempo.grafana.com/config.hash: {} + f:tempo.grafana.com/rbacConfig.hash: {} + f:tempo.grafana.com/tenantsConfig.hash: {} + f:labels: + .: {} + f:app.kubernetes.io/component: {} + f:app.kubernetes.io/instance: {} + f:app.kubernetes.io/managed-by: {} + f:app.kubernetes.io/name: {} + f:spec: + f:containers: + k:{"name":"tempo-gateway"}: + .: {} + f:args: {} + f:image: {} + f:imagePullPolicy: {} + f:livenessProbe: + .: {} + f:failureThreshold: {} + f:httpGet: + .: {} + f:path: {} + f:port: {} + f:scheme: {} + f:periodSeconds: {} + f:successThreshold: {} + f:timeoutSeconds: {} + f:name: {} + f:ports: + .: {} + k:{"containerPort":8080,"protocol":"TCP"}: + .: {} + f:containerPort: {} + f:name: {} + f:protocol: {} + k:{"containerPort":8081,"protocol":"TCP"}: + .: {} + f:containerPort: {} + f:name: {} + f:protocol: {} + k:{"containerPort":8090,"protocol":"TCP"}: + .: {} + f:containerPort: {} + f:name: {} + f:protocol: {} + f:readinessProbe: + .: {} + f:failureThreshold: {} + f:httpGet: + .: {} + f:path: {} + f:port: {} + f:scheme: {} + f:periodSeconds: {} + f:successThreshold: {} + f:timeoutSeconds: {} + f:resources: {} + f:securityContext: + .: {} + f:allowPrivilegeEscalation: {} + f:capabilities: + .: {} + f:drop: {} + f:readOnlyRootFilesystem: {} + f:terminationMessagePath: {} + f:terminationMessagePolicy: {} + f:volumeMounts: + .: {} + k:{"mountPath":"/etc/tempo-gateway/cm"}: + .: {} + f:mountPath: {} + f:name: {} + f:readOnly: {} + k:{"mountPath":"/etc/tempo-gateway/secret/tenants.yaml"}: + .: {} + f:mountPath: {} + f:name: {} + f:readOnly: {} + f:subPath: {} + k:{"mountPath":"/var/run/ca"}: + .: {} + f:mountPath: {} + f:name: {} + k:{"mountPath":"/var/run/tls/server"}: + .: {} + f:mountPath: {} + f:name: {} + f:dnsPolicy: {} + f:restartPolicy: {} + f:schedulerName: {} + f:securityContext: {} + f:serviceAccount: {} + f:serviceAccountName: {} + f:terminationGracePeriodSeconds: {} + f:volumes: + .: {} + k:{"name":"rbac"}: + .: {} + f:configMap: + .: {} + f:defaultMode: {} + f:items: {} + f:name: {} + f:name: {} + k:{"name":"tempo-foo-ca-bundle"}: + .: {} + f:configMap: + .: {} + f:defaultMode: {} + f:name: {} + f:name: {} + k:{"name":"tempo-foo-gateway-mtls"}: + .: {} + f:name: {} + f:secret: + .: {} + f:defaultMode: {} + f:secretName: {} + k:{"name":"tenant"}: + .: {} + f:name: {} + f:secret: + .: {} + f:defaultMode: {} + f:items: {} + f:secretName: {} + manager: Go-http-client + operation: Update + time: "2023-09-11T06:45:16Z" + - apiVersion: apps/v1 + fieldsType: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: {} + f:deployment.kubernetes.io/revision: {} + f:status: + f:availableReplicas: {} + f:conditions: + .: {} + k:{"type":"Available"}: + .: {} + f:lastTransitionTime: {} + f:lastUpdateTime: {} + f:message: {} + f:reason: {} + f:status: {} + f:type: {} + k:{"type":"Progressing"}: + .: {} + f:lastTransitionTime: {} + f:lastUpdateTime: {} + f:message: {} + f:reason: {} + f:status: {} + f:type: {} + f:observedGeneration: {} + f:readyReplicas: {} + f:replicas: {} + f:updatedReplicas: {} + manager: kube-controller-manager + operation: Update + subresource: status + time: "2023-09-11T06:45:35Z" name: tempo-foo-gateway namespace: kuttl-test-warm-locust ownerReferences: @@ -14,15 +207,29 @@ controller: true kind: TempoStack name: foo + uid: d565f9ec-2474-411e-9f9c-08cb07291ceb spec: + progressDeadlineSeconds: 600 + replicas: 1 + revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/component: gateway app.kubernetes.io/instance: foo app.kubernetes.io/managed-by: tempo-operator app.kubernetes.io/name: tempo + strategy: + rollingUpdate: + maxSurge: 25%! (MISSING)+ maxUnavailable: 25%! (MISSING)+ type: RollingUpdate template: metadata: + annotations: + tempo.grafana.com/config.hash: e2bb049bab1fd69b9cce4cbd7134a84294466b0b345da4f1e5080fc375caeef2 + tempo.grafana.com/rbacConfig.hash: be67a26b8b63cddef8e0e4befa9c5485380ad5c40a42c2dc62148dfcf25a80a2 + tempo.grafana.com/tenantsConfig.hash: aa7bffbf2a5b139337fa3388e1f6925818dc653de2dd6641488b27ea28ab1323 + creationTimestamp: null labels: app.kubernetes.io/component: gateway app.kubernetes.io/instance: foo @@ -30,7 +237,32 @@ app.kubernetes.io/name: tempo spec: containers: - - imagePullPolicy: IfNotPresent + - args: + - --traces.tenant-header=x-scope-orgid + - --web.listen=0.0.0.0:8080 + - --web.internal.listen=0.0.0.0:8081 + - --traces.write.endpoint=tempo-foo-distributor.kuttl-test-warm-locust.svc.cluster.local:4317 + - --traces.read.endpoint=https://tempo-foo-query-frontend.kuttl-test-warm-locust.svc.cluster.local:16686 + - --grpc.listen=0.0.0.0:8090 + - --rbac.config=/etc/tempo-gateway/cm/rbac.yaml + - --tenants.config=/etc/tempo-gateway/secret/tenants.yaml + - --log.level=info + - --tls.internal.server.key-file=/var/run/tls/server/tls.key + - --tls.internal.server.cert-file=/var/run/tls/server/tls.crt + - --traces.tls.key-file=/var/run/tls/server/tls.key + - --traces.tls.cert-file=/var/run/tls/server/tls.crt + - --traces.tls.ca-file=/var/run/ca/service-ca.crt + image: registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:b62f5cedfeb5907b638f14ca6aaeea50f41642980a8a6f87b7061e88d90fac23 + imagePullPolicy: IfNotPresent + livenessProbe: + failureThreshold: 10 + httpGet: + path: /live + port: 8081 + scheme: HTTPS + periodSeconds: 30 + successThreshold: 1 + timeoutSeconds: 2 name: tempo-gateway ports: - containerPort: 8090 @@ -42,6 +274,24 @@ - containerPort: 8080 name: public protocol: TCP + readinessProbe: + failureThreshold: 12 + httpGet: + path: /ready + port: 8081 + scheme: HTTPS + periodSeconds: 5 + successThreshold: 1 + timeoutSeconds: 1 + resources: {} + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File volumeMounts: - mountPath: /etc/tempo-gateway/cm name: rbac @@ -50,6 +300,17 @@ name: tenant readOnly: true subPath: tenants.yaml + - mountPath: /var/run/ca + name: tempo-foo-ca-bundle + - mountPath: /var/run/tls/server + name: tempo-foo-gateway-mtls + dnsPolicy: ClusterFirst + restartPolicy: Always + schedulerName: default-scheduler + securityContext: {} + serviceAccount: tempo-foo + serviceAccountName: tempo-foo + terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 420 @@ -65,6 +326,31 @@ - key: tenants.yaml path: tenants.yaml secretName: tempo-foo-gateway + - configMap: + defaultMode: 420 + name: tempo-foo-ca-bundle + name: tempo-foo-ca-bundle + - name: tempo-foo-gateway-mtls + secret: + defaultMode: 420 + secretName: tempo-foo-gateway-mtls status: + availableReplicas: 1 + conditions: + - lastTransitionTime: "2023-09-11T06:45:35Z" + lastUpdateTime: "2023-09-11T06:45:35Z" + message: Deployment has minimum availability. + reason: MinimumReplicasAvailable + status: "True" + type: Available + - lastTransitionTime: "2023-09-11T06:45:16Z" + lastUpdateTime: "2023-09-11T06:45:35Z" + message: ReplicaSet "tempo-foo-gateway-6ffcd7c57f" has successfully progressed. + reason: NewReplicaSetAvailable + status: "True" + type: Progressing + observedGeneration: 1 readyReplicas: 1 + replicas: 1 + updatedReplicas: 1 case.go:366: resource Deployment:kuttl-test-warm-locust/tempo-foo-gateway: .spec.template.spec.containers.volumeMounts: slice length mismatch: 2 != 4 logger.go:42: 06:48:15 | gateway | gateway events from ns kuttl-test-warm-locust: logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:10 +0000 UTC Normal Pod dex-5c56c4454d-hvdj6 Binding Scheduled Successfully assigned kuttl-test-warm-locust/dex-5c56c4454d-hvdj6 to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:10 +0000 UTC Normal ReplicaSet.apps dex-5c56c4454d SuccessfulCreate Created pod: dex-5c56c4454d-hvdj6 logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:10 +0000 UTC Normal Deployment.apps dex ScalingReplicaSet Scaled up replica set dex-5c56c4454d to 1 logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:11 +0000 UTC Normal Pod dex-5c56c4454d-hvdj6 AddedInterface Add eth0 [10.128.2.27/23] from ovn-kubernetes logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:11 +0000 UTC Normal Pod dex-5c56c4454d-hvdj6.spec.containers{hydra} Pulling Pulling image "dexidp/dex:v2.30.0" logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:13 +0000 UTC Normal Pod dex-5c56c4454d-hvdj6.spec.containers{hydra} Pulled Successfully pulled image "dexidp/dex:v2.30.0" in 2.357945036s (2.357955195s including waiting) logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:13 +0000 UTC Normal Pod dex-5c56c4454d-hvdj6.spec.containers{hydra} Created Created container hydra logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:13 +0000 UTC Normal Pod dex-5c56c4454d-hvdj6.spec.containers{hydra} Started Started container hydra logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:15 +0000 UTC Normal Deployment.apps tempo-foo-distributor ScalingReplicaSet Scaled up replica set tempo-foo-distributor-b46f85d9d to 1 logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal PersistentVolumeClaim data-tempo-foo-ingester-0 WaitForFirstConsumer waiting for first consumer to be created before binding logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal PersistentVolumeClaim data-tempo-foo-ingester-0 Provisioning External provisioner is provisioning volume for claim "kuttl-test-warm-locust/data-tempo-foo-ingester-0" logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal PersistentVolumeClaim data-tempo-foo-ingester-0 ExternalProvisioning waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-foo-compactor-5b8bf68594-4zktz Binding Scheduled Successfully assigned kuttl-test-warm-locust/tempo-foo-compactor-5b8bf68594-4zktz to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal ReplicaSet.apps tempo-foo-compactor-5b8bf68594 SuccessfulCreate Created pod: tempo-foo-compactor-5b8bf68594-4zktz logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal Deployment.apps tempo-foo-compactor ScalingReplicaSet Scaled up replica set tempo-foo-compactor-5b8bf68594 to 1 logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-foo-distributor-b46f85d9d-dl4fw Binding Scheduled Successfully assigned kuttl-test-warm-locust/tempo-foo-distributor-b46f85d9d-dl4fw to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal ReplicaSet.apps tempo-foo-distributor-b46f85d9d SuccessfulCreate Created pod: tempo-foo-distributor-b46f85d9d-dl4fw logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-foo-gateway-6ffcd7c57f-qgtpx Binding Scheduled Successfully assigned kuttl-test-warm-locust/tempo-foo-gateway-6ffcd7c57f-qgtpx to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal ReplicaSet.apps tempo-foo-gateway-6ffcd7c57f SuccessfulCreate Created pod: tempo-foo-gateway-6ffcd7c57f-qgtpx logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal Deployment.apps tempo-foo-gateway ScalingReplicaSet Scaled up replica set tempo-foo-gateway-6ffcd7c57f to 1 logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal StatefulSet.apps tempo-foo-ingester SuccessfulCreate create Claim data-tempo-foo-ingester-0 Pod tempo-foo-ingester-0 in StatefulSet tempo-foo-ingester success logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal StatefulSet.apps tempo-foo-ingester SuccessfulCreate create Pod tempo-foo-ingester-0 in StatefulSet tempo-foo-ingester successful logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-foo-querier-7956b54c65-wbm4l Binding Scheduled Successfully assigned kuttl-test-warm-locust/tempo-foo-querier-7956b54c65-wbm4l to ip-10-0-11-13.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal ReplicaSet.apps tempo-foo-querier-7956b54c65 SuccessfulCreate Created pod: tempo-foo-querier-7956b54c65-wbm4l logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal Deployment.apps tempo-foo-querier ScalingReplicaSet Scaled up replica set tempo-foo-querier-7956b54c65 to 1 logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb Binding Scheduled Successfully assigned kuttl-test-warm-locust/tempo-foo-query-frontend-78d47c4d9c-hrlhb to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal ReplicaSet.apps tempo-foo-query-frontend-78d47c4d9c SuccessfulCreate Created pod: tempo-foo-query-frontend-78d47c4d9c-hrlhb logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:16 +0000 UTC Normal Deployment.apps tempo-foo-query-frontend ScalingReplicaSet Scaled up replica set tempo-foo-query-frontend-78d47c4d9c to 1 logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:19 +0000 UTC Normal PersistentVolumeClaim data-tempo-foo-ingester-0 ProvisioningSucceeded Successfully provisioned volume pvc-222b15f0-a619-4399-b65b-456b6f4c3512 logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:19 +0000 UTC Normal Pod tempo-foo-distributor-b46f85d9d-dl4fw AddedInterface Add eth0 [10.131.0.34/23] from ovn-kubernetes logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-foo-ingester-0 Binding Scheduled Successfully assigned kuttl-test-warm-locust/tempo-foo-ingester-0 to ip-10-0-7-6.us-east-2.compute.internal default-scheduler logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-foo-querier-7956b54c65-wbm4l AddedInterface Add eth0 [10.131.0.35/23] from ovn-kubernetes logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:20 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb AddedInterface Add eth0 [10.128.2.37/23] from ovn-kubernetes logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:21 +0000 UTC Normal Pod tempo-foo-compactor-5b8bf68594-4zktz AddedInterface Add eth0 [10.131.0.36/23] from ovn-kubernetes logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:21 +0000 UTC Normal Pod tempo-foo-distributor-b46f85d9d-dl4fw.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:21 +0000 UTC Normal Pod tempo-foo-distributor-b46f85d9d-dl4fw.spec.containers{tempo} Created Created container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:21 +0000 UTC Normal Pod tempo-foo-distributor-b46f85d9d-dl4fw.spec.containers{tempo} Started Started container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:21 +0000 UTC Normal Pod tempo-foo-querier-7956b54c65-wbm4l.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:21 +0000 UTC Normal Pod tempo-foo-querier-7956b54c65-wbm4l.spec.containers{tempo} Created Created container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:21 +0000 UTC Normal Pod tempo-foo-querier-7956b54c65-wbm4l.spec.containers{tempo} Started Started container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-foo-compactor-5b8bf68594-4zktz.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-foo-compactor-5b8bf68594-4zktz.spec.containers{tempo} Created Created container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-foo-compactor-5b8bf68594-4zktz.spec.containers{tempo} Started Started container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-foo-gateway-6ffcd7c57f-qgtpx AddedInterface Add eth0 [10.131.0.37/23] from ovn-kubernetes logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-foo-gateway-6ffcd7c57f-qgtpx.spec.containers{tempo-gateway} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:b62f5cedfeb5907b638f14ca6aaeea50f41642980a8a6f87b7061e88d90fac23" logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:22 +0000 UTC Normal Pod tempo-foo-ingester-0 SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-222b15f0-a619-4399-b65b-456b6f4c3512" logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:22 +0000 UTC Warning Pod tempo-foo-querier-7956b54c65-wbm4l.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-foo-querier-7956b54c65-wbm4l_kuttl-test-warm-locust(74f8e3b9-68de-4957-b358-c3827edc55d5) logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:23 +0000 UTC Warning Pod tempo-foo-compactor-5b8bf68594-4zktz.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-foo-compactor-5b8bf68594-4zktz_kuttl-test-warm-locust(788fbc51-99d0-41e6-9f14-76589320a73d) logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb.spec.containers{tempo} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb.spec.containers{tempo} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" in 390.712322ms (390.731662ms including waiting) logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb.spec.containers{tempo} Created Created container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb.spec.containers{tempo} Started Started container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:24 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb.spec.containers{tempo-query} Pulling Pulling image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:0da43034f440b8258a48a0697ba643b5643d48b615cdb882ac7f4f1f80aad08e" logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:25 +0000 UTC Normal Pod tempo-foo-ingester-0 AddedInterface Add eth0 [10.128.2.39/23] from ovn-kubernetes logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:25 +0000 UTC Normal Pod tempo-foo-ingester-0.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:25 +0000 UTC Normal Pod tempo-foo-ingester-0.spec.containers{tempo} Created Created container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:26 +0000 UTC Normal Pod tempo-foo-ingester-0.spec.containers{tempo} Started Started container tempo logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:27 +0000 UTC Warning Pod tempo-foo-ingester-0.spec.containers{tempo} BackOff Back-off restarting failed container tempo in pod tempo-foo-ingester-0_kuttl-test-warm-locust(a01c988d-3959-4efd-96e8-3c413b42bf98) logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:31 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb.spec.containers{tempo-query} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:0da43034f440b8258a48a0697ba643b5643d48b615cdb882ac7f4f1f80aad08e" in 6.419795841s (6.419807781s including waiting) logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:31 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb.spec.containers{tempo-query} Created Created container tempo-query logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:31 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb.spec.containers{tempo-query} Started Started container tempo-query logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:31 +0000 UTC Normal Pod tempo-foo-query-frontend-78d47c4d9c-hrlhb.spec.containers{tempo} Pulled Container image "registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a" already present on machine logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:33 +0000 UTC Normal Pod tempo-foo-gateway-6ffcd7c57f-qgtpx.spec.containers{tempo-gateway} Pulled Successfully pulled image "registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:b62f5cedfeb5907b638f14ca6aaeea50f41642980a8a6f87b7061e88d90fac23" in 11.381306127s (11.381319338s including waiting) logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:33 +0000 UTC Normal Pod tempo-foo-gateway-6ffcd7c57f-qgtpx.spec.containers{tempo-gateway} Created Created container tempo-gateway logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:33 +0000 UTC Normal Pod tempo-foo-gateway-6ffcd7c57f-qgtpx.spec.containers{tempo-gateway} Started Started container tempo-gateway logger.go:42: 06:48:15 | gateway | 2023-09-11 06:45:56 +0000 UTC Warning Pod tempo-foo-compactor-5b8bf68594-4zktz.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 logger.go:42: 06:48:15 | gateway | 2023-09-11 06:46:05 +0000 UTC Warning Pod tempo-foo-ingester-0.spec.containers{tempo} Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 logger.go:42: 06:48:15 | gateway | Deleting namespace: kuttl-test-warm-locust === CONT kuttl harness.go:405: run tests finished harness.go:513: cleaning up harness.go:570: removing temp folder: "" --- FAIL: kuttl (221.61s) --- FAIL: kuttl/harness (0.00s) --- PASS: kuttl/harness/route (41.88s) --- PASS: kuttl/harness/generate (73.29s) --- FAIL: kuttl/harness/reconcile (162.79s) --- FAIL: kuttl/harness/gateway (221.24s) FAIL