Details
-
Bug
-
Resolution: Cannot Reproduce
-
Blocker
-
None
-
Logging 5.5.5
-
False
-
None
-
False
-
NEW
-
NEW
Description
Description of problem:
Loki pods are throwing TLS handshake error
Version-Release number of selected component (if applicable):
Logging 5.5.6
OCP 4.12
How reproducible:
Always
Steps to Reproduce:
1. Deploy CLO and Loki
$ oc get csv
NAME DISPLAY VERSION REPLACES PHASE
cluster-logging.5.5.6 Red Hat OpenShift Logging 5.5.6 Succeeded
loki-operator.5.5.6 Loki Operator 5.5.6 Succeeded
2. Create bucket secret for log storage
3. Provision LokiStack CR
apiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: lokistack-sample namespace: openshift-logging spec: managementState: Managed replicationFactor: 1 size: 1x.extra-small storage: secret: name: s3-secret type: s3 storageClassName: gp3-csi tenants: mode: openshift-logging
4. Create a CL instance to forward logs to LokiStack instance.
apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: openshift-logging spec: managementState: "Managed" logStore: type: "lokistack" lokistack: name: lokistack-sample collection: type: "fluentd"
Actual results:
Collector pods throwing below error:
2022-12-23 12:03:20 +0000 [warn]: [default_loki_infra] failed to flush the buffer. retry_times=8 next_retry_time=2022-12-23 12:04:18 +0000 chunk="5f07d3d79e5d4d2c9f01b87049c5125f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="502 Bad Gateway " 2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:168:in `write' 2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush' 2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run' 2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start' 2022-12-23 12:03:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2022-12-23 12:03:21 +0000 [warn]: [default_loki_infra] failed to flush the buffer. retry_times=8 next_retry_time=2022-12-23 12:04:23 +0000 chunk="5f07972e8e3fc3a1aa0dada286d80339" error_class=Errno::ECONNRESET error="Connection reset by peer" 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/openssl/buffering.rb:383:in `syswrite_nonblock' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/openssl/buffering.rb:383:in `write_nonblock' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:277:in `block in write0' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:275:in `each' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:275:in `each_with_index' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:275:in `write0' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:248:in `block in write' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:265:in `writing' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/protocol.rb:247:in `write' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http/generic_request.rb:189:in `send_request_with_body' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http/generic_request.rb:123:in `exec' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:1521:in `block in transport_request' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:1519:in `catch' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:1519:in `transport_request' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:1492:in `request' 2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:226:in `block in loki_http_request' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:933:in `start' 2022-12-23 12:03:21 +0000 [warn]: /usr/share/ruby/net/http.rb:606:in `start' 2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:226:in `loki_http_request' 2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:156:in `write' 2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush' 2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run' 2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start' 2022-12-23 12:03:21 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Loki pods throwing below error:
2022-12-23 12:05:57.923481 I | http: TLS handshake error from 10.129.2.13:50562: remote error: tls: bad certificate
Expected results:
Loki components should be up and running without any error.