Description of problem:

We have fluentd writing to a Loki instance deployed by LokiStack on the same cluster. Since the 5.8.0 release, which was upgraded from 5.7.7 we see errors in the collector logs as it attempts to push logs via the Loki gateway.

2023-11-09 12:55:52 +0000 [warn]: [loki_app] failed to flush the buffer. retry_times=50 next_retry_time=2023-11-09 12:56:53 +0000 chunk="609b715724186ee6731bb08a50d01c55" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 peeraddr=172.30.195.210:8080 state=error: certificate verify failed (self-signed certificate in certificate chain)"

We have inspected the old configuration which used to contain the following 

    ca_cert /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    bearer_token_file /var/run/secrets/kubernetes.io/serviceaccount/token

This appears to have been picked up as a fallback via https://github.com/openshift/cluster-logging-operator/blob/master/internal/generator/fluentd/output/loki/loki.go#L164-L171

The new generated config no longer contains the required ca_cert and bearer_token references. We do not configure a specific Secret in the ClusterLogForwarder

Version-Release number of selected component (if applicable):

How reproducible:

Loki ingestion is broken since upgrade across all three of our clusters.

Actual results:

Expected results:

Additional info: