-
Bug
-
Resolution: Done
-
Critical
-
Logging 5.8.0
Description of problem:
Fluentd collector pods can't forward logs to default lokistack and raise SSL errors:
POD_IPS: 10.130.0.82, PROM_BIND_IP: 0.0.0.0 Setting each total_size_limit for 4 buffers to 4809148876 bytes Setting queued_chunks_limit_size for each buffer to 573 Setting chunk_limit_size for each buffer to 8388608 2023-09-26 12:29:36 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp 2023-09-26 12:29:36 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp 2023-09-26 12:31:03 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified 2023-09-26 12:31:03 +0000 [warn]: Webrick ignores given TLS version 2023-09-26 12:31:06 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=0 next_retry_time=2023-09-26 12:31:07 +0000 chunk="606423f8e85b5cb5c09d9ecc4fad1d37" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 peeraddr=172.30.202.120:8080 state=error: certificate verify failed (self-signed certificate in certificate chain)" 2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/net-protocol-0.2.1/lib/net/protocol.rb:46:in `connect_nonblock' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/net-protocol-0.2.1/lib/net/protocol.rb:46:in `ssl_socket_connect' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/ruby/net/http.rb:1048:in `connect' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/ruby/net/http.rb:976:in `do_start' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/ruby/net/http.rb:965:in `start' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/ruby/net/http.rb:627:in `start' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:245:in `loki_http_request' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:162:in `write' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:1225:in `try_flush' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:1538:in `flush_thread_run' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start' 2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Version-Release number of selected component (if applicable):
registry.redhat.io/openshift-logging/cluster-logging-rhel9-operator@sha256:4ee056a1a25e8d85e3a08a54b94b98e3f704bc939d393698c7a20b970925ae12
How reproducible:
Always
Steps to Reproduce:
- Deploy lokistack
- Create CL with:
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance namespace: openshift-logging spec: collection: type: fluentd logStore: lokistack: name: lokistack-dev type: lokistack managementState: Managed visualization: type: ocp-console
Actual results:
SSL errors on fluentd collector pods
Expected results:
Logs should be sent to lokistack without error.