-
Bug
-
Resolution: Done
-
Undefined
-
None
-
None
-
None
Version of components:
tempo-operator.v0.13.0-1
Description of the problem:
When we set spec.storage.tls.enabled to true, using Tempo with AWS STS, we get the following error and the changes are not applied.
error: tempostacks.tempo.grafana.com "tmstack" could not be patched: Internal error occurred: failed calling webhook "vtempostack.tempo.grafana.com": failed to call webhook: Post "https://tempo-operator-controller-service.openshift-tempo-operator.svc:443/validate-tempo-grafana-com-v1alpha1-tempostack?timeout=10s": EOF You can run `oc replace -f /var/folders/v_/xjhg70nn5h9g__sf8d7wg99r0000gn/T/oc-edit-307932674.yaml` to try this update again.
These are the operator logs generated.
2024/09/12 05:42:04 http: panic serving 10.128.0.2:56050: runtime error: invalid memory address or nil pointer dereference goroutine 13612 [running]: net/http.(*conn).serve.func1() /usr/lib/golang/src/net/http/server.go:1903 +0xbe panic({0x1aba640?, 0x310bbd0?}) /usr/lib/golang/src/runtime/panic.go:770 +0x132 github.com/grafana/tempo-operator/internal/handlers/storage.GetStorageParamsForTempoStack({, _}, {, _}, {{{0xc0039775e0, 0x6}, {0xc0039775e6, 0x5}, {0x0, 0x0}, ...}, ...}) /remote-source/tempo-operator/app/internal/handlers/storage/storage.go:35 +0xc76 github.com/grafana/tempo-operator/internal/webhooks.(*validator).validateStorage(, {, _}, {{{0xc0039775e0, 0x6}, {0xc0039775e6, 0x5}, {0x0, 0x0}, {0xc0009dbf20, ...}, ...}, ...}) /remote-source/tempo-operator/app/internal/webhooks/tempostack_webhook.go:221 +0x5e github.com/grafana/tempo-operator/internal/webhooks.(*validator).validate(0xc0005b2340, {0x209d868, 0xc0009dbdd0}, {0x2086f08?, 0xc000ea2608?}) /remote-source/tempo-operator/app/internal/webhooks/tempostack_webhook.go:462 +0x496 github.com/grafana/tempo-operator/internal/webhooks.(*validator).ValidateUpdate(0xc0005acc80?, {0x209d868?, 0xc0009dbdd0?}, {0x1000?, 0x0?}, {0x2086f08?, 0xc000ea2608?}) /remote-source/tempo-operator/app/internal/webhooks/tempostack_webhook.go:189 +0x2d sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatorForType).Handle(, {, _}, {{{0xc0044bbe90, 0x24}, { {0xc0028fa7f8, 0x11}, {0xc003977350, 0x8}, {0xc003977360, ...}}, ...}}) /remote-source/tempo-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/webhook/admission/validator_custom.go:101 +0x70b sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(, {, _}, {{{0xc0044bbe90, 0x24}, {{0xc0028fa7f8, 0x11} , {0xc003977350, 0x8}, {0xc003977360, ...}}, ...}}) /remote-source/tempo-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/webhook/admission/webhook.go:169 +0x1ee sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc0005b1040, {0x7fb8f4033640, 0xc001438f00}, 0xc00305be60) /remote-source/tempo-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/webhook/admission/http.go:119 +0xb12 sigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1({0x7fb8f4033640, 0xc001438f00}, 0xc00305be60) /remote-source/tempo-operator/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:60 +0xcb net/http.HandlerFunc.ServeHTTP(0x3118c50?, {0x7fb8f4033640?, 0xc001438f00?}, 0x423265?) /usr/lib/golang/src/net/http/server.go:2171 +0x29 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x2094370?, 0xc0044c6c40?}, 0xc00305be60) /remote-source/tempo-operator/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:147 +0xc3 net/http.HandlerFunc.ServeHTTP(0x81f906?, {0x2094370?, 0xc0044c6c40?}, 0x41b265?) /usr/lib/golang/src/net/http/server.go:2171 +0x29 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x2094370, 0xc0044c6c40}, 0xc00305be60) /remote-source/tempo-operator/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:109 +0xc2 net/http.HandlerFunc.ServeHTTP(0xc0005e8b60?, {0x2094370?, 0xc0044c6c40?}, 0x10?) /usr/lib/golang/src/net/http/server.go:2171 +0x29 net/http.(*ServeMux).ServeHTTP(0x41b265?, {0x2094370, 0xc0044c6c40}, 0xc00305be60) /usr/lib/golang/src/net/http/server.go:2688 +0x1ad net/http.serverHandler.ServeHTTP({0x20899f0?}, {0x2094370?, 0xc0044c6c40?}, 0x6?) /usr/lib/golang/src/net/http/server.go:3142 +0x8e net/http.(*conn).serve(0xc002f85b90, {0x209d868, 0xc0000dd1d0}) /usr/lib/golang/src/net/http/server.go:2044 +0x5e8 created by net/http.(*Server).Serve in goroutine 75 /usr/lib/golang/src/net/http/server.go:3290 +0x4b4
Steps to reproduce the issue:
- Install the Tempo operator on a STS enabled OCP cluster.
- Create a Tempo instance with AWS STS. https://docs.openshift.com/container-platform/4.16/observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-installing.html#distr-tracing-tempo-object-storage-setup-aws-sts-install_dist-tracing-tempo-installing
- Set the spec.storage.tls.enabled to true in the TempoStack CR.
- The change will not be applied with the above errors in description.
Additional note:
To workaround this we set the field using extraConfig:
spec:
extraConfig:
tempo:
storage:
trace:
s3:
insecure: false