Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-3139

Jaeger UI only shows service names of traces of the last 15 minutes, traces of other services cannot be queried


    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • rhosdt-3.2
    • None
    • None
    • None
    • 8
    • False
    • None
    • False
    • Tracing Sprint # 240, Tracing Sprint # 241, Tracing Sprint # 242, Tracing Sprint # 243, Tracing Sprint # 244, Tracing Sprint # 245, Tracing Sprint # 246, Tracing Sprint # 248

      The Jaeger UI gets the values for its services dropdown from this Tempo API: https://grafana.com/docs/tempo/latest/api_docs/#search-tag-values
      Unfortunately, this API only queries the ingester caches, which keep traces 15 minutes (by default) in their cache after sending it to the backend object storage.

      The traces are queryable and not lost, however in the Jaeger UI the service name needs to be selected first before the search button gets enabled.

      In case a service is generating traces every 15 mins, the service name can be selected in the Jaeger UI and all historical traces from the object storage backend will be shown.
      Otherwise, if this service didn't generate traces in the last 15 minutes, the Jaeger UI won't show the service name and it's not possible to search for it in the Jaeger UI (uploading a JSON when searching might work though).

      Ruben asked in the Tempo channel, currently there is no way to change that except increasing the complete_block_timeout setting so that ingesters keep traces longer in their memory cache, however that will increase memory usage of the ingesters.

            rvargasp@redhat.com Ruben Vargas Palma
            agerstma@redhat.com Andreas Gerstmayr
            0 Vote for this issue
            6 Start watching this issue