Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-4747

HTTP panic when toggling spec.storage.tls.enabled to true, when using Tempo with AWS STS

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • rhosdt-3.4
    • None
    • None
    • None
    • Tracing Sprint # 261

      Version of components:

      tempo-operator.v0.13.0-1

      Description of the problem:

      When we set spec.storage.tls.enabled to true, using Tempo with AWS STS, we get the following error and the changes are not applied. 

      error: tempostacks.tempo.grafana.com "tmstack" could not be patched: Internal error occurred: failed calling webhook "vtempostack.tempo.grafana.com": failed to call webhook: Post "https://tempo-operator-controller-service.openshift-tempo-operator.svc:443/validate-tempo-grafana-com-v1alpha1-tempostack?timeout=10s": EOF
      You can run `oc replace -f /var/folders/v_/xjhg70nn5h9g__sf8d7wg99r0000gn/T/oc-edit-307932674.yaml` to try this update again.
      

      These are the operator logs generated. 

      2024/09/12 05:42:04 http: panic serving 10.128.0.2:56050: runtime error: invalid memory address or nil pointer dereference
      goroutine 13612 [running]:
      net/http.(*conn).serve.func1()
      /usr/lib/golang/src/net/http/server.go:1903 +0xbe
      panic({0x1aba640?, 0x310bbd0?})
      /usr/lib/golang/src/runtime/panic.go:770 +0x132
      github.com/grafana/tempo-operator/internal/handlers/storage.GetStorageParamsForTempoStack({, _}, {, _}, {{{0xc0039775e0, 0x6}, {0xc0039775e6, 0x5}, {0x0, 0x0}, ...}, ...})
      /remote-source/tempo-operator/app/internal/handlers/storage/storage.go:35 +0xc76
      github.com/grafana/tempo-operator/internal/webhooks.(*validator).validateStorage(, {, _}, {{{0xc0039775e0, 0x6}, {0xc0039775e6, 0x5}, {0x0, 0x0}, {0xc0009dbf20, ...}, ...}, ...})
      /remote-source/tempo-operator/app/internal/webhooks/tempostack_webhook.go:221 +0x5e
      github.com/grafana/tempo-operator/internal/webhooks.(*validator).validate(0xc0005b2340, {0x209d868, 0xc0009dbdd0}, {0x2086f08?, 0xc000ea2608?})
      /remote-source/tempo-operator/app/internal/webhooks/tempostack_webhook.go:462 +0x496
      github.com/grafana/tempo-operator/internal/webhooks.(*validator).ValidateUpdate(0xc0005acc80?, {0x209d868?, 0xc0009dbdd0?}, {0x1000?, 0x0?}, {0x2086f08?, 0xc000ea2608?})
      /remote-source/tempo-operator/app/internal/webhooks/tempostack_webhook.go:189 +0x2d
      sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatorForType).Handle(, {, _}, {{{0xc0044bbe90, 0x24}, {
      {0xc0028fa7f8, 0x11}, {0xc003977350, 0x8}, {0xc003977360, ...}}, ...}})
      
      /remote-source/tempo-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/webhook/admission/validator_custom.go:101 +0x70b
      
      sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(, {, _}, {{{0xc0044bbe90, 0x24}, {{0xc0028fa7f8, 0x11}
      , {0xc003977350, 0x8}, {0xc003977360, ...}}, ...}})
      /remote-source/tempo-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/webhook/admission/webhook.go:169 +0x1ee
      sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc0005b1040, {0x7fb8f4033640, 0xc001438f00}, 0xc00305be60)
      /remote-source/tempo-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/webhook/admission/http.go:119 +0xb12
      sigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1({0x7fb8f4033640, 0xc001438f00}, 0xc00305be60)
      /remote-source/tempo-operator/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:60 +0xcb
      net/http.HandlerFunc.ServeHTTP(0x3118c50?, {0x7fb8f4033640?, 0xc001438f00?}, 0x423265?)
      /usr/lib/golang/src/net/http/server.go:2171 +0x29
      github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x2094370?, 0xc0044c6c40?}, 0xc00305be60)
      /remote-source/tempo-operator/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:147 +0xc3
      net/http.HandlerFunc.ServeHTTP(0x81f906?, {0x2094370?, 0xc0044c6c40?}, 0x41b265?)
      /usr/lib/golang/src/net/http/server.go:2171 +0x29
      github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x2094370, 0xc0044c6c40}, 0xc00305be60)
      /remote-source/tempo-operator/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:109 +0xc2
      net/http.HandlerFunc.ServeHTTP(0xc0005e8b60?, {0x2094370?, 0xc0044c6c40?}, 0x10?)
      /usr/lib/golang/src/net/http/server.go:2171 +0x29
      net/http.(*ServeMux).ServeHTTP(0x41b265?, {0x2094370, 0xc0044c6c40}, 0xc00305be60)
      /usr/lib/golang/src/net/http/server.go:2688 +0x1ad
      net/http.serverHandler.ServeHTTP({0x20899f0?}, {0x2094370?, 0xc0044c6c40?}, 0x6?)
      /usr/lib/golang/src/net/http/server.go:3142 +0x8e
      net/http.(*conn).serve(0xc002f85b90, {0x209d868, 0xc0000dd1d0})
      /usr/lib/golang/src/net/http/server.go:2044 +0x5e8
      created by net/http.(*Server).Serve in goroutine 75
      /usr/lib/golang/src/net/http/server.go:3290 +0x4b4
      

      Steps to reproduce the issue:

      1. Install the Tempo operator on a STS enabled OCP cluster.
      2. Create a Tempo instance with AWS STS. https://docs.openshift.com/container-platform/4.16/observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-installing.html#distr-tracing-tempo-object-storage-setup-aws-sts-install_dist-tracing-tempo-installing
      3. Set the spec.storage.tls.enabled to true in the TempoStack CR.
      4. The change will not be applied with the above errors in description.

      Additional note:
      To workaround this we set the field using extraConfig:

      spec:
        extraConfig:
          tempo:
            storage:
              trace:
                s3:
                  insecure: false 

              rvargasp@redhat.com Ruben Vargas Palma
              rhn-support-ikanse Ishwar Kanse
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: