Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-6009

Request for fetch traces is timeouted after 30s even though I set a higher timeout in external_services.tracing.query_timeout

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • OSSM 2.6.0
    • OSSM 2.5.0
    • Kiali
    • None
    • False
    • None
    • False
    • Hide
      The default tracing time out query in Kiali is 30 seconds. It is possible to increase the Tracing timeout with the following configuration:

      `.spec.external_services.tracing.query_timeout: 40`
      `.spec.server.write_timeout: 40`
      openshift has default 30s timeout on routes. The Kiali route needs this annotation `haproxy.router.openshift.io/timeout=50s` (https://docs.openshift.com/container-platform/4.16/networking/routes/route-configuration.html#nw-configuring-route-timeouts_route-configuration)
      Show
      The default tracing time out query in Kiali is 30 seconds. It is possible to increase the Tracing timeout with the following configuration: `.spec.external_services.tracing.query_timeout: 40` `.spec.server.write_timeout: 40` openshift has default 30s timeout on routes. The Kiali route needs this annotation `haproxy.router.openshift.io/timeout=50s` ( https://docs.openshift.com/container-platform/4.16/networking/routes/route-configuration.html#nw-configuring-route-timeouts_route-configuration )
    • Hide
      1. Set up OSSM 2.5 with tempo tracing
      2. Setup bookinfo with kiali traffic generator
      3. Wait at least 15minutes to generate traffic
      4. Open Kiali UI
      5. Go to Application->httpbin->httpbin->Outbound Metrics or Traces
      Show
      Set up OSSM 2.5 with tempo tracing Setup bookinfo with kiali traffic generator Wait at least 15minutes to generate traffic Open Kiali UI Go to Application->httpbin->httpbin->Outbound Metrics or Traces

      Request for fetch traces in Kiali time outed (with Tempo) after 30 seconds even though I set a higher timeout in `external_services.tracing.query_timeout: 60`. It looks like the request between Kiali UI <-> Kiali API is timeouted. Can I set also timeout for that?

      E.g.
      when I don't set any external_services.tracing.query_timeout, the default one is used (5s). So, in that case, the connection between Kiali UI -> Kiali API works but a request between Kiali API -> Jaeger API is timeouted.
      When I look in the browser console, there is a request to Kiali API 

      GET https://kiali-istio-system.apps.user-rhos-d-2.servicemesh.rhqeaws.com/api/namespaces/mkralik/apps/httpbin/spans?startMicros=1709125095129000

      which returns

      error	'Get "http://tempo-sample-query-frontend.tracing-system.svc.cluster.local:16686/api/traces?end=1709128675000000&limit=100&service=httpbin.mkralik&start=1709125095000000": net/http: request canceled (Client.Timeout exceeded while awaiting headers)'

      to workaround that, I have set `external_services.tracing.query_timeout: 60` which helps when the request takes lower than 30s. But it doesn't work for requests that take longer.

        1. image-2024-07-08-12-44-16-609.png
          image-2024-07-08-12-44-16-609.png
          42 kB
        2. image-2024-07-09-16-18-06-303.png
          image-2024-07-09-16-18-06-303.png
          24 kB
        3. Screenshot from 2024-07-08 11-52-51.png
          Screenshot from 2024-07-08 11-52-51.png
          352 kB

              rh-ee-jcordoba Josune Cordoba Torrecilla
              mkralik@redhat.com Matej Kralik
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: