Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-11118

Kiali v1.73 causes panic error in tempo-query container with Tempo 0.18

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • OSSM 2.6.10
    • Kiali
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      With newer Tempo version (Tempo operator 0.18), there is a panic error in the `tempo-sample-query-frontend` pod in `tempo-query` container.

      panic: invalid Go type model.TraceID for field jaeger.storage.v1.GetTraceRequest.trace_idgoroutine 61 [running]:
      google.golang.org/protobuf/internal/impl.newSingularConverter({0xfeec48, 0xe0ece0}, {0xfee348, 0xc000276388})
          google.golang.org/protobuf@v1.36.6/internal/impl/convert.go:142 +0xd47
      google.golang.org/protobuf/internal/impl.NewConverter({0xfeec48, 0xe0ece0}, {0xfee348, 0xc000276388})
          google.golang.org/protobuf@v1.36.6/internal/impl/convert.go:60 +0x8f
      google.golang.org/protobuf/internal/impl.fieldInfoForScalar({0xfee348, 0xc000276388}, {{0xde1daa, 0x7}, {0x0, 0x0}, {0xfeec48, 0xe0ece0}, {0xde1db3, 0x88}, ...}, ...)
          google.golang.org/protobuf@v1.36.6/internal/impl/message_reflect_field.go:268 +0x1d7
      google.golang.org/protobuf/internal/impl.(*MessageInfo).makeKnownFieldsFunc(0xc000166a80, {0x58, {0xfeec48, 0xd16dc0}, 0x40, {0xfeec48, 0xd05100}, 0xffffffffffffffff, {0x0, 0x0}, ...})
          google.golang.org/protobuf@v1.36.6/internal/impl/message_reflect.go:78 +0x74a
      google.golang.org/protobuf/internal/impl.(*MessageInfo).makeReflectFuncs(0xc000166a80, {0xfeec48, 0xe2cca0}, {0x58, {0xfeec48, 0xd16dc0}, 0x40, {0xfeec48, 0xd05100}, 0xffffffffffffffff, ...})
          google.golang.org/protobuf@v1.36.6/internal/impl/message_reflect.go:42 +0x78
      google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce(0xc000166a80)
          google.golang.org/protobuf@v1.36.6/internal/impl/message.go:92 +0x1d0
      google.golang.org/protobuf/internal/impl.(*MessageInfo).init(...)
          google.golang.org/protobuf@v1.36.6/internal/impl/message.go:71
      google.golang.org/protobuf/internal/impl.(*messageReflectWrapper).ProtoMethods(0xfd3920?)
          google.golang.org/protobuf@v1.36.6/internal/impl/message_reflect_gen.go:162 +0x25
      google.golang.org/protobuf/proto.protoMethods(...)
          google.golang.org/protobuf@v1.36.6/proto/proto_methods.go:19
      google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0xfd5138, 0xc00017d2c0}, 0x2710, 0x0}, {0xc0001231c5, 0x2c, ...}, ...)
          google.golang.org/protobuf@v1.36.6/proto/decode.go:100 +0xe2
      google.golang.org/protobuf/proto.Unmarshal({0xc0001231c5, 0x2c, 0x2c}, {0xfd3920?, 0xc000354500?})
          google.golang.org/protobuf@v1.36.6/proto/decode.go:62 +0x6e
      google.golang.org/grpc/encoding/proto.(*codecV2).Unmarshal(0x101?, {0xc000354470?, 0x1?, 0x1?}, {0xe4fa60?, 0xc0001412c0?})
          google.golang.org/grpc@v1.71.1/encoding/proto/proto.go:80 +0xfa
      google.golang.org/grpc.recv(0xc000391e30?, {0x7f1050ac1928, 0x18cadc0}, {0xfd1780?, 0xc000141140?}, {0x0?, 0x0?}, {0xe4fa60, 0xc0001412c0}, 0x400000, ...)
          google.golang.org/grpc@v1.71.1/rpc_util.go:911 +0x11f
      google.golang.org/grpc.(*serverStream).RecvMsg(0xc00027c460, {0xe4fa60, 0xc0001412c0})
          google.golang.org/grpc@v1.71.1/stream.go:1760 +0x192
      github.com/jaegertracing/jaeger/proto-gen/storage_v1._SpanReaderPlugin_GetTrace_Handler({0xe3f2a0, 0xc0001ee000}, {0xfdfd88, 0xc00027c460})
          github.com/jaegertracing/jaeger@v1.67.0/proto-gen/storage_v1/storage.pb.go:1473 +0x57
      google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001f6000, {0xfdbd18, 0xc00027faa0}, 0xc000141140, 0xc0001d46c0, 0x1839f20, 0x0)
          google.golang.org/grpc@v1.71.1/server.go:1695 +0x1252
      google.golang.org/grpc.(*Server).handleStream(0xc0001f6000, {0xfdbf50, 0xc000207520}, 0xc000141140)
          google.golang.org/grpc@v1.71.1/server.go:1819 +0xb47
      google.golang.org/grpc.(*Server).serveStreams.func2.1()
          google.golang.org/grpc@v1.71.1/server.go:1035 +0x7f
      created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 24
          google.golang.org/grpc@v1.71.1/server.go:1046 +0x11d 

      Every time when a user clicks on any trace in the Kiali UI, an exception is thrown in the tempo-query container, which also causes the unavailability of Tempo for some period of time.
      See video in the attachments.
      Screencast From 2025-10-01 10-40-16.mp4

      This is not happening with the older Tempo (Tempo version 0.16) nor with using native Tempo. (However, Kiali v1.73 doesn't support Tempo natively, so the Jaeger provider in Tempo is used instead ) 

              rh-ee-jcordoba Josune Cordoba Torrecilla
              mkralik@redhat.com Matej Kralik
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: