-
Bug
-
Resolution: Done
-
Undefined
-
None
-
None
-
Quality / Stability / Reliability
-
1
-
False
-
-
False
-
-
-
Tracing Sprint # 261, Tracing Sprint # 266
Version of components:
tempo-operator.v0.13.0-1
Description of the problem:
When we set spec.storage.tls.enabled to true, using Tempo with AWS STS, we get the following error and the changes are not applied.
error: tempostacks.tempo.grafana.com "tmstack" could not be patched: Internal error occurred: failed calling webhook "vtempostack.tempo.grafana.com": failed to call webhook: Post "https://tempo-operator-controller-service.openshift-tempo-operator.svc:443/validate-tempo-grafana-com-v1alpha1-tempostack?timeout=10s": EOF You can run `oc replace -f /var/folders/v_/xjhg70nn5h9g__sf8d7wg99r0000gn/T/oc-edit-307932674.yaml` to try this update again.
These are the operator logs generated.
2024/09/12 05:42:04 http: panic serving 10.128.0.2:56050: runtime error: invalid memory address or nil pointer dereference
goroutine 13612 [running]:
net/http.(*conn).serve.func1()
/usr/lib/golang/src/net/http/server.go:1903 +0xbe
panic({0x1aba640?, 0x310bbd0?})
/usr/lib/golang/src/runtime/panic.go:770 +0x132
github.com/grafana/tempo-operator/internal/handlers/storage.GetStorageParamsForTempoStack({, _}, {, _}, {{{0xc0039775e0, 0x6}, {0xc0039775e6, 0x5}, {0x0, 0x0}, ...}, ...})
/remote-source/tempo-operator/app/internal/handlers/storage/storage.go:35 +0xc76
github.com/grafana/tempo-operator/internal/webhooks.(*validator).validateStorage(, {, _}, {{{0xc0039775e0, 0x6}, {0xc0039775e6, 0x5}, {0x0, 0x0}, {0xc0009dbf20, ...}, ...}, ...})
/remote-source/tempo-operator/app/internal/webhooks/tempostack_webhook.go:221 +0x5e
github.com/grafana/tempo-operator/internal/webhooks.(*validator).validate(0xc0005b2340, {0x209d868, 0xc0009dbdd0}, {0x2086f08?, 0xc000ea2608?})
/remote-source/tempo-operator/app/internal/webhooks/tempostack_webhook.go:462 +0x496
github.com/grafana/tempo-operator/internal/webhooks.(*validator).ValidateUpdate(0xc0005acc80?, {0x209d868?, 0xc0009dbdd0?}, {0x1000?, 0x0?}, {0x2086f08?, 0xc000ea2608?})
/remote-source/tempo-operator/app/internal/webhooks/tempostack_webhook.go:189 +0x2d
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatorForType).Handle(, {, _}, {{{0xc0044bbe90, 0x24}, {
{0xc0028fa7f8, 0x11}, {0xc003977350, 0x8}, {0xc003977360, ...}}, ...}})
/remote-source/tempo-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/webhook/admission/validator_custom.go:101 +0x70b
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(, {, _}, {{{0xc0044bbe90, 0x24}, {{0xc0028fa7f8, 0x11}
, {0xc003977350, 0x8}, {0xc003977360, ...}}, ...}})
/remote-source/tempo-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/webhook/admission/webhook.go:169 +0x1ee
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc0005b1040, {0x7fb8f4033640, 0xc001438f00}, 0xc00305be60)
/remote-source/tempo-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/webhook/admission/http.go:119 +0xb12
sigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1({0x7fb8f4033640, 0xc001438f00}, 0xc00305be60)
/remote-source/tempo-operator/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:60 +0xcb
net/http.HandlerFunc.ServeHTTP(0x3118c50?, {0x7fb8f4033640?, 0xc001438f00?}, 0x423265?)
/usr/lib/golang/src/net/http/server.go:2171 +0x29
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x2094370?, 0xc0044c6c40?}, 0xc00305be60)
/remote-source/tempo-operator/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:147 +0xc3
net/http.HandlerFunc.ServeHTTP(0x81f906?, {0x2094370?, 0xc0044c6c40?}, 0x41b265?)
/usr/lib/golang/src/net/http/server.go:2171 +0x29
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x2094370, 0xc0044c6c40}, 0xc00305be60)
/remote-source/tempo-operator/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:109 +0xc2
net/http.HandlerFunc.ServeHTTP(0xc0005e8b60?, {0x2094370?, 0xc0044c6c40?}, 0x10?)
/usr/lib/golang/src/net/http/server.go:2171 +0x29
net/http.(*ServeMux).ServeHTTP(0x41b265?, {0x2094370, 0xc0044c6c40}, 0xc00305be60)
/usr/lib/golang/src/net/http/server.go:2688 +0x1ad
net/http.serverHandler.ServeHTTP({0x20899f0?}, {0x2094370?, 0xc0044c6c40?}, 0x6?)
/usr/lib/golang/src/net/http/server.go:3142 +0x8e
net/http.(*conn).serve(0xc002f85b90, {0x209d868, 0xc0000dd1d0})
/usr/lib/golang/src/net/http/server.go:2044 +0x5e8
created by net/http.(*Server).Serve in goroutine 75
/usr/lib/golang/src/net/http/server.go:3290 +0x4b4
Steps to reproduce the issue:
- Install the Tempo operator on a STS enabled OCP cluster.
- Create a Tempo instance with AWS STS. https://docs.openshift.com/container-platform/4.16/observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-installing.html#distr-tracing-tempo-object-storage-setup-aws-sts-install_dist-tracing-tempo-installing
- Set the spec.storage.tls.enabled to true in the TempoStack CR.
- The change will not be applied with the above errors in description.
Additional note:
To workaround this we set the field using extraConfig:
spec:
extraConfig:
tempo:
storage:
trace:
s3:
insecure: false