-
Bug
-
Resolution: Done
-
Undefined
-
None
-
None
-
None
Version of components:
OCP version 4.17.0-0.nightly-2024-09-08-135628
tempo-operator.v0.13.0
Description of the issue:
Tempo query container is crashing with the following error when the Tempo query is scaled.
"error":"open /var/run/tls/server/tls.crt: no such file or directory"
Steps to reproduce the issue:
1. Install Tempo operator built off the latest upstream branch.
2. Install OpenTelemetry collector.
3. Run component-replicas test with --skip-delete.
chainsaw test --skip-delete --config .chainsaw-openshift.yaml tests/e2e-openshift/component-replicas
4. Check the TempoStack instance pods and Tempo query pod logs.
% oc get pods NAME READY STATUS RESTARTS AGE dev-collector-77c7db7789-6hqzs 1/1 Running 0 11m generate-traces-grpc-krcmb 0/1 Completed 0 11m generate-traces-http-59cgf 0/1 Completed 0 11m minio-66774b6457-5j82z 1/1 Running 0 14m tempo-cmpreps-compactor-76cf774dbc-rx99w 1/1 Running 0 14m tempo-cmpreps-compactor-76cf774dbc-xj5l4 1/1 Running 0 13m tempo-cmpreps-distributor-5865cfb945-464gx 1/1 Running 0 14m tempo-cmpreps-distributor-5865cfb945-l56jv 1/1 Running 0 13m tempo-cmpreps-gateway-5bdf5bf65c-546q8 2/2 Running 0 13m tempo-cmpreps-gateway-5bdf5bf65c-fd62s 2/2 Running 0 14m tempo-cmpreps-ingester-0 1/1 Running 0 14m tempo-cmpreps-ingester-1 1/1 Running 0 13m tempo-cmpreps-querier-7d98775b64-drqsp 1/1 Running 0 14m tempo-cmpreps-querier-7d98775b64-qvrtf 1/1 Running 0 13m tempo-cmpreps-query-frontend-857d97d89b-gjw4r 3/3 Running 4 (6m22s ago) 13m tempo-cmpreps-query-frontend-857d97d89b-wpl6p 2/3 CrashLoopBackOff 4 (61s ago) 14m
% oc logs tempo-cmpreps-query-frontend-857d97d89b-wpl6p -c tempo-query
{"@level":"error","@message":"failed to init tracer backend","@module":"jaeger-tempo","@timestamp":"2024-09-09T08:04:00.142664Z","error":"error opening /var/run/ca/service-ca.crt CA"} {"@level":"error","@message":"failed to load TLS credentials","@module":"jaeger-tempo","@timestamp":"2024-09-09T08:04:00.142776Z","error":"open /var/run/tls/server/tls.crt: no such file or directory"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xba9ab0]
goroutine 17 [running]:
github.com/grafana/tempo/cmd/tempo-query/tempo.(*Backend).apiSchema(...)
/drone/src/cmd/tempo-query/tempo/plugin.go:182
github.com/grafana/tempo/cmd/tempo-query/tempo.(*Backend).FindTraceIDs(0x0, {0xe878f8?, 0xc000290450?}, 0xc0000cd800)
/drone/src/cmd/tempo-query/tempo/plugin.go:350 +0x110
github.com/grafana/tempo/cmd/tempo-query/tempo.(*Backend).FindTraces(0x0, 0xc000290360, {0xe8bc30, 0xc00028a230})
/drone/src/cmd/tempo-query/tempo/plugin.go:311 +0x135
github.com/jaegertracing/jaeger/proto-gen/storage_v1._SpanReaderPlugin_FindTraces_Handler({0xd09240, 0x0}, {0xe8a410, 0xc00028c080})
/drone/src/vendor/github.com/jaegertracing/jaeger/proto-gen/storage_v1/storage.pb.go:1513 +0x107
github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1({0xd09240, 0x0}, {0xe8a530, 0xc0002be000}, 0xc0002840f0, 0xdb6be0)
/drone/src/vendor/github.com/opentracing-contrib/go-grpc/server.go:114 +0x34a
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0000b7600, {0xe878f8, 0xc0002901e0}, {0xe8cc60, 0xc0001b2180}, 0xc0002b6000, 0xc00020e8a0, 0x1462f60, 0x0)
/drone/src/vendor/google.golang.org/grpc/server.go:1680 +0x11e7
google.golang.org/grpc.(*Server).handleStream(0xc0000b7600, {0xe8cc60, 0xc0001b2180}, 0xc0002b6000)
/drone/src/vendor/google.golang.org/grpc/server.go:1794 +0xe3a
google.golang.org/grpc.(*Server).serveStreams.func2.1()
/drone/src/vendor/google.golang.org/grpc/server.go:1029 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 9
/drone/src/vendor/google.golang.org/grpc/server.go:1040 +0x125
