Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
None
-
False
-
False
-
NEW
-
NEW
-
Logging (Core) - Sprint 213, Logging (Core) - Sprint 214, Logging (Core) - Sprint 215
Description
DescriptionÂ
Fluentd pods are failing to start after updating OCP version from 4.7.32 to 4.8.18 with cluster-logging.5.2.2-21.
-The respective fluentd pods are reporting the following error:
2021-12-03T15:33:54.470042002+08:00 2021-12-03 15:33:54 +0800 [error]: unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: EVP lib"
2021-12-03T15:33:54.470042002+08:00 2021-12-03 15:33:54 +0800 [error]: /usr/share/ruby/net/protocol.rb:44:in `connect_nonblock'
2021-12-03T15:33:54.470042002+08:00 2021-12-03 15:33:54 +0800 [error]: /usr/share/ruby/net/protocol.rb:44:in `ssl_socket_connect'
2021-12-03T15:33:54.470084194+08:00 2021-12-03 15:33:54 +0800 [error]: /usr/share/ruby/net/http.rb:985:in `connect'
2021-12-03T15:33:54.470084194+08:00 2021-12-03 15:33:54 +0800 [error]: /usr/share/ruby/net/http.rb:920:in `do_start'
2021-12-03T15:33:54.470084194+08:00 2021-12-03 15:33:54 +0800 [error]: /usr/share/ruby/net/http.rb:909:in `start'
-Checking the connectivity and certificates between fluentd and internal ES through curl and openssl, there are no issues:
$ curl -vvv --cacert /var/run/ocp-collector/secrets/fluentd/ca-bundle.crt --key /var/run/ocp-collector/secrets/fluentd/tls.key --cert /var/run/ocp-collector/secrets/fluentd/tls.crt
https://elasticsearch.openshift-logging.svc:9200/_cat/health
$ openssl s_client -connect elasticsearch.openshift-logging.svc:9200 -cert /var/run/ocp-collector/secrets/fluentd/tls.crt -key /var/run/ocp-collector/secrets/fluentd/tls.key -CAfile /var/run/ocp-collector/secrets/fluentd/ca-bundle.crt
-Client reinstall cluster logging but the error is still the same.
-Changing fluentd logs to debug mode and reboot fluent pods cannot see anything relevant.
-Asking the following outputs inside fluentd pods we couldn't see anything wrongly configured either:
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- /var/run/secrets/kubernetes.io/serviceaccount/token
Version-Release number of selected component (if applicable):
- ClusterLogging Operator 5.2.2-21
- OpenShift Container Platform 4.8.18
Actual results:
Fluentd pods are failing to start with [error]: unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: EVP lib", for this reason, fluentd cannot send logs to an internal ES.
Expected results:
Fluentd pods work properly.