Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3303

Loki pods are down with TLS handshake error

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Blocker Blocker
    • None
    • Logging 5.5.5
    • Log Storage, Loki
    • False
    • None
    • False
    • NEW
    • NEW

      Description of problem:

      Loki pods are throwing TLS handshake error

      Version-Release number of selected component (if applicable):

      Logging 5.5.6
      OCP 4.12

      How reproducible:

      Always

      Steps to Reproduce:

      1. Deploy CLO and Loki

      $ oc get csv
      NAME                    DISPLAY                     VERSION   REPLACES   PHASE
      cluster-logging.5.5.6   Red Hat OpenShift Logging   5.5.6                Succeeded
      loki-operator.5.5.6     Loki Operator               5.5.6                Succeeded
      

      2. Create bucket secret for log storage

      3. Provision LokiStack CR

      apiVersion: loki.grafana.com/v1
      kind: LokiStack
      metadata:
        name: lokistack-sample
        namespace: openshift-logging
      spec:
        managementState: Managed
        replicationFactor: 1
        size: 1x.extra-small
        storage:
          secret:
            name: s3-secret
            type: s3
        storageClassName: gp3-csi
        tenants:
          mode: openshift-logging

      4. Create a CL instance to forward logs to LokiStack instance.

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
      name: "instance"
      namespace: openshift-logging
      spec:
      managementState: "Managed"
      logStore:
      type: "lokistack"
      lokistack:
      name: lokistack-sample
      collection:
      type: "fluentd"

      Actual results:

      Collector pods throwing below error:

      2022-12-23 12:03:20 +0000 [warn]: [default_loki_infra] failed to flush the buffer. retry_times=8 next_retry_time=2022-12-23 12:04:18 +0000 chunk="5f07d3d79e5d4d2c9f01b87049c5125f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="502 Bad Gateway "
        2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:168:in `write'
        2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
        2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
        2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
        2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
      2022-12-23 12:03:21 +0000 [warn]: [default_loki_infra] failed to flush the buffer. retry_times=8 next_retry_time=2022-12-23 12:04:23 +0000 chunk="5f07972e8e3fc3a1aa0dada286d80339" error_class=Errno::ECONNRESET error="Connection reset by peer"
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/openssl/buffering.rb:383:in `syswrite_nonblock'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/openssl/buffering.rb:383:in `write_nonblock'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:277:in `block in write0'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:275:in `each'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:275:in `each_with_index'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:275:in `write0'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:248:in `block in write'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:265:in `writing'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:247:in `write'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http/generic_request.rb:189:in `send_request_with_body'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http/generic_request.rb:123:in `exec'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:1521:in `block in transport_request'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:1519:in `catch'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:1519:in `transport_request'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:1492:in `request'
        2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:226:in `block in loki_http_request'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:933:in `start'
        2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:606:in `start'
        2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:226:in `loki_http_request'
        2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:156:in `write'
        2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
        2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
        2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
        2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

      Loki pods throwing below error:

      2022-12-23 12:05:57.923481 I | http: TLS handshake error from 10.129.2.13:50562: remote error: tls: bad certificate
      

      Expected results:

      Loki components should be up and running without any error. 

              Unassigned Unassigned
              gkarager Giriyamma Karagere Ramaswamy (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: