Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4791

Fluentd collector Pods no longer picks up the log collector SAs Secret as a fallback

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      Before this update, forwarding with a legacy forwarder to an internal Lokistack would produce SSL certificate errors. With this update, the logcollector service account is used as the default service account for authentication, using it's associated token and ca.crt.
      Show
      Before this update, forwarding with a legacy forwarder to an internal Lokistack would produce SSL certificate errors. With this update, the logcollector service account is used as the default service account for authentication, using it's associated token and ca.crt.
    • Bug Fix
    • ?
    • Log Collection - Sprint 245

    Description

      Description of problem:

      We have fluentd writing to a Loki instance deployed by LokiStack on the same cluster. Since the 5.8.0 release, which was upgraded from 5.7.7 we see errors in the collector logs as it attempts to push logs via the Loki gateway.

       

      2023-11-09 12:55:52 +0000 [warn]: [loki_app] failed to flush the buffer. retry_times=50 next_retry_time=2023-11-09 12:56:53 +0000 chunk="609b715724186ee6731bb08a50d01c55" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 peeraddr=172.30.195.210:8080 state=error: certificate verify failed (self-signed certificate in certificate chain)"
      

      We have inspected the old configuration which used to contain the following 

          ca_cert /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
          bearer_token_file /var/run/secrets/kubernetes.io/serviceaccount/token
      

      This appears to have been picked up as a fallback via https://github.com/openshift/cluster-logging-operator/blob/master/internal/generator/fluentd/output/loki/loki.go#L164-L171

       

      The new generated config no longer contains the required ca_cert and bearer_token references. We do not configure a specific Secret in the ClusterLogForwarder

      Version-Release number of selected component (if applicable):

      How reproducible:

      Loki ingestion is broken since upgrade across all three of our clusters.

      Actual results:

       

      Expected results:

      Additional info:

      Attachments

        Activity

          People

            rh-ee-calee Calvin Lee
            pgough@redhat.com Philip Gough
            Anping Li Anping Li
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: