Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-2288

EO seeing "failed to get index settings" errors with Jaeger indices

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Minor Minor
    • None
    • jaeger-operator-1.24
    • None
    • None
    • 2
    • False
    • False
    • Tracing Sprint # 216, Tracing Sprint # 217

      ###ENV

      $ oc get csv -n service-mesh
      NAME DISPLAY VERSION REPLACES PHASE
      container-security-operator.v3.3.4 Quay Container Security 3.3.4 container-security-operator.v3.3.1 Succeeded
      elasticsearch-operator.5.2.2-21 OpenShift Elasticsearch Operator 5.2.2-21 elasticsearch-operator.5.2.1-5 Succeeded
      jaeger-operator.v1.24.1 Red Hat OpenShift Jaeger 1.24.1 jaeger-operator.v1.20.3 Succeeded
      kiali-operator.v1.24.6 Kiali Operator 1.24.6 kiali-operator.v1.24.5 Succeeded
      redhat-openshift-pipelines-operator.v1.1.2 Red Hat OpenShift Pipelines Operator 1.1.2 openshift-pipelines-operator.v1.1.1 Succeeded
      servicemeshoperator.v2.0.8 Red Hat OpenShift Service Mesh 2.0.8-0 servicemeshoperator.v2.0.2 Succeeded

      ###ISSUE

      It's possible to see continue in the Elasticsearch Operator logs one error indicating that's not able to get the dynamic index setting for the jaeger indices like these:

      {"_ts":"2021-10-15T12:03:30.72868073Z","_level":"0","_component":"elasticsearch-operator","_message":"failed to get index settings","_error":{"cause":{"Value":"string","Type":{},"Offset":81,"Struct":"IndexMapperSettings","Field":"settings.index.mapper.dynamic"},"destination_type":"estypes.IndexSettings","index":"jaeger-service-2021-10-15","msg":"failed to decode response body"},"cluster":"elasticsearch","index":"jaeger-service-2021-10-15","namespace":"service-mesh"}

      It seems that jaeger is able to continue collecting logs and rotating the indices, but these kind of messages are filling the Elasticsearch Operator logs.

      $ oc logs jaeger-es-index-cleaner-27282235-db2p6
      Removing jaeger-service-2021-11-07
      Removing jaeger-span-2021-11-07

       

      Also, for adding more information, the EO has no issues connecting with the Elasticsearch running in the service mesh namespace:

      $ curl -vv -k -H "Authorization: Bearer $TOKEN" https://elasticsearch.service-mesh.svc:9200/_cat/indices

      •   Trying 172.30.21.42...
      • TCP_NODELAY set
      • Connected to elasticsearch.service-mesh.svc (172.30.21.42) port 9200 (#0)
      • ALPN, offering h2
      • ALPN, offering http/1.1
      • successfully set certificate verify locations:
      •   CAfile: /etc/pki/tls/certs/ca-bundle.crt
          CApath: none
        ...
        < HTTP/1.1 200 OK
        < Content-Length: 258
        < Content-Type: text/plain; charset=UTF-8
        < Gap-Upstream-Address: localhost:9200
        < Date: Fri, 15 Oct 2021 11:56:23 GMT

        green open jaeger-span-2021-10-15    72Uzfh10Twy-qxXdjSi30A 3 1 72086 0  9.7mb  4.5mb
        green open jaeger-service-2021-10-15 h0Y1etnoQbmnAu1XTHEIOQ 3 1    11 0 37.5kb 18.7kb
        green open .security                 uXA0G_dFSUucxZQ5Cu4mZg 1 1     6 2 64.1kb   31kb

              kearls@redhat.com Kevin Earls (Inactive)
              rhn-support-ocasalsa Oscar Casal Sanchez
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: