Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2138

EO Self-generated certificates issue with Kibana when "logging.openshift.io/elasticsearch-cert-management: true" annotation is used

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • Logging 5.3.4
    • Logging 5.3.0
    • Log Storage
    • None
    • False
    • False
    • NEW
    • NEW

      Description

      KIbana can't establish connection to the Elasticsearch with error:

      {"type":"log","@timestamp":"2021-12-21T15:24:17Z","tags":["warning","elasticsearch","admin"],"pid":116,"message":"Unable to revive connection: https://elasticsearch.openshift-logging.svc:9200/"}
      {"type":"log","@timestamp":"2021-12-21T15:24:17Z","tags":["warning","elasticsearch","admin"],"pid":116,"message":"No living connections"}
      

      In same time in Elasticsearch/proxy container got error:

      2021/12/21 15:33:51 http: TLS handshake error from 10.131.0.119:44492: tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Logging Signing CA")
      

      In Elasticsearch operator pod logs:

      {"_ts":"2021-12-21T14:24:30.262503062Z","_level":"0","_component":"elasticsearch-operator_controller_kibana-controller","_message":"Reconciler error","_error":{"msg":"did not receive hashvalue for trusted CA value"},"name":"kibana","namespace":"openshift-logging"}
      

      Route Kibana also unavailable.

      How to reproduce

      Deploy CLO from PR: pull/1265

      Workaround:

      oc delete deployment/kibana secret/kibana secret/kibana-proxy -n openshift-logging 
      

      after recreating deployment and secrets, kibana pod can connect to the Elasticsearch and router works well.

      Possible issue

      A possible problem can be in race-condition for certificate generation, looks like Kibana and Elasticsearch certificates signed by different CA (signing-elasticsearch secret updated several times).

              vparfono Vitalii Parfonov
              vparfono Vitalii Parfonov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: