Details
-
Bug
-
Resolution: Done
-
Major
-
Logging 5.3.2
-
None
-
False
-
False
-
NEW
-
NEW
-
Logging (LogExp) - Sprint 212
Description
Description
KIbana can't establish connection to the Elasticsearch with error:
{"type":"log","@timestamp":"2021-12-21T15:24:17Z","tags":["warning","elasticsearch","admin"],"pid":116,"message":"Unable to revive connection: https://elasticsearch.openshift-logging.svc:9200/"}
{"type":"log","@timestamp":"2021-12-21T15:24:17Z","tags":["warning","elasticsearch","admin"],"pid":116,"message":"No living connections"}
In same time in Elasticsearch/proxy container got error:
2021/12/21 15:33:51 http: TLS handshake error from 10.131.0.119:44492: tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Logging Signing CA")
In Elasticsearch operator pod logs:
{"_ts":"2021-12-21T14:24:30.262503062Z","_level":"0","_component":"elasticsearch-operator_controller_kibana-controller","_message":"Reconciler error","_error":{"msg":"did not receive hashvalue for trusted CA value"},"name":"kibana","namespace":"openshift-logging"}
Route Kibana also unavailable.
How to reproduce
Deploy CLO from PR: pull/1265
Workaround:
oc delete deployment/kibana secret/kibana secret/kibana-proxy -n openshift-logging
after recreating deployment and secrets, kibana pod can connect to the Elasticsearch and router works well.
Possible issue
A possible problem can be in race-condition for certificate generation, looks like Kibana and Elasticsearch certificates signed by different CA (signing-elasticsearch secret updated several times).
