Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4559

SSL errors when forwarding logs to LokiStack using fluentd

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • VERIFIED
    • Log Collection - Sprint 242

      Description of problem:

      Fluentd collector pods can't forward logs to default lokistack and raise SSL errors:

      POD_IPS: 10.130.0.82, PROM_BIND_IP: 0.0.0.0
      Setting each total_size_limit for 4 buffers to 4809148876 bytes
      Setting queued_chunks_limit_size for each buffer to 573
      Setting chunk_limit_size for each buffer to 8388608
      2023-09-26 12:29:36 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
      2023-09-26 12:29:36 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
      2023-09-26 12:31:03 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified
      2023-09-26 12:31:03 +0000 [warn]: Webrick ignores given TLS version
      2023-09-26 12:31:06 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=0 next_retry_time=2023-09-26 12:31:07 +0000 chunk="606423f8e85b5cb5c09d9ecc4fad1d37" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 peeraddr=172.30.202.120:8080 state=error: certificate verify failed (self-signed certificate in certificate chain)"
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/net-protocol-0.2.1/lib/net/protocol.rb:46:in `connect_nonblock'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/net-protocol-0.2.1/lib/net/protocol.rb:46:in `ssl_socket_connect'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/ruby/net/http.rb:1048:in `connect'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/ruby/net/http.rb:976:in `do_start'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/ruby/net/http.rb:965:in `start'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/ruby/net/http.rb:627:in `start'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:245:in `loki_http_request'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:162:in `write'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:1225:in `try_flush'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
        2023-09-26 12:31:06 +0000 [warn]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

      Version-Release number of selected component (if applicable):

      registry.redhat.io/openshift-logging/cluster-logging-rhel9-operator@sha256:4ee056a1a25e8d85e3a08a54b94b98e3f704bc939d393698c7a20b970925ae12

      How reproducible:

      Always

      Steps to Reproduce:

      1. Deploy lokistack
      2. Create CL with:
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        collection:
          type: fluentd
        logStore:
          lokistack:
            name: lokistack-dev
          type: lokistack
        managementState: Managed
        visualization:
          type: ocp-console 

      Actual results:

      SSL errors on fluentd collector pods

      Expected results:

      Logs should be sent to lokistack without error.

      Additional info:

              vparfono Vitalii Parfonov
              rhn-support-kbharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: