Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: Logging 5.1.1
Affects Version/s: Logging 5.1, Logging 5.2
Component/s: Log Collection
Labels:
- devel_ack+

Blocked:
False
Ready:
False
Epic Link:
Basic authentication against external Elasticsearch
Docs QE Status:
NEW
QE Status:
NEW
Release Note Text:
Undefined
Market:

Sprint:
Logging (Core) - Sprint 202, Logging (Core) - Sprint 203

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

When forward log to external elasticsearch with secret, CLO always sets below configurations in fluent.conf even when these keys are not existed:

      client_key '/var/run/ocp-collector/secrets/pipelinesecret/tls.key'
      client_cert '/var/run/ocp-collector/secrets/pipelinesecret/tls.crt'
      ca_file '/var/run/ocp-collector/secrets/pipelinesecret/ca-bundle.crt'
      ssl_version TLSv1_2

Besides, the `scheme` is always set to `https` even when it is `http` in clf/instance.
Version-Release number of selected component (if applicable):

latest code cloned from https://github.com/openshift/cluster-logging-operator master branch
How reproducible:

Always

Steps to Reproduce:
1. deploy CLO

2. deploy an external elasticsearch server, enable user authentication, don't enable http ssl and transport ssl

3. create a secret in openshift-logging namespace which contains username and password

oc create secret generic $secret-name --from-literal=username=$username --from-literal=password=$password

4. create clf/instance to forward logs to the external elasticsearch server with the secret created in step 3

cat << EOF | oc create -f -
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
  - name: secure-es
    secret:
      name: pipelinesecret
    type: elasticsearch
    url: http://elasticsearch-server.a2h8p.svc:9200
  pipelines:
  - inputRefs:
    - application
    - infrastructure
    - audit
    name: forward-logs
    outputRefs:
    - secure-es
EOF

5. deploy fluentd pods

6. check fluent.conf in cm/fluentd and secret keys in fluentd pod

      @type elasticsearch
      @id secure_es
      host elasticsearch-server.a2h8p.svc
      port 9200
      verify_es_version_at_startup false
    user "#{File.exists?('/var/run/ocp-collector/secrets/pipelinesecret/username') ? open('/var/run/ocp-collector/secrets/pipelinesecret/username','r') do |f|f.read end : ''}"
    password "#{File.exists?('/var/run/ocp-collector/secrets/pipelinesecret/password') ? open('/var/run/ocp-collector/secrets/pipelinesecret/password','r') do |f|f.read end : ''}"
    scheme https
      ssl_version TLSv1_2
      target_index_key viaq_index_name
      id_key viaq_msg_id
      remove_keys viaq_index_name
      client_key '/var/run/ocp-collector/secrets/pipelinesecret/tls.key'
      client_cert '/var/run/ocp-collector/secrets/pipelinesecret/tls.crt'
      ca_file '/var/run/ocp-collector/secrets/pipelinesecret/ca-bundle.crt'
      type_name _doc

$ oc rsh fluentd-r9z2x
sh-4.4# ls /var/run/ocp-collector/secrets/pipelinesecret/
password  username
sh-4.4#

7. check logs in external elasticsearch, no logs received

8. check fluentd pod log, lots of error logs:

  2021-05-14 01:15:14 +0000 [warn]: suppressed same stacktrace
2021-05-14 01:16:18 +0000 [warn]: [secure_es] failed to flush the buffer. retry_time=12 next_retry_seconds=2021-05-14 01:17:19 +0000 chunk="5c23fd61a8ab91a09364881da73ec885" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-server.a2h8p.svc\", :port=>9200, :scheme=>\"https\", :user=>\"qitang\", :password=>\"obfuscated\"}): Problem with the local SSL certificate"
  2021-05-14 01:16:18 +0000 [warn]: suppressed same stacktrace
2021-05-14 01:16:18 +0000 [warn]: [secure_es] failed to flush the buffer. retry_time=13 next_retry_seconds=2021-05-14 01:17:14 +0000 chunk="5c23fd61ca56f4a6f7b18d59b716f973" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-server.a2h8p.svc\", :port=>9200, :scheme=>\"https\", :user=>\"qitang\", :password=>\"obfuscated\"}): Problem with the local SSL certificate"
  2021-05-14 01:16:18 +0000 [warn]: suppressed same stacktrace

Actual results:

please find it in step 6
Expected results:

      @type elasticsearch
      @id secure_es
      host elasticsearch-server.a2h8p.svc
      port 9200
      verify_es_version_at_startup false
    user "#{File.exists?('/var/run/ocp-collector/secrets/pipelinesecret/username') ? open('/var/run/ocp-collector/secrets/pipelinesecret/username','r') do |f|f.read end : ''}"
    password "#{File.exists?('/var/run/ocp-collector/secrets/pipelinesecret/password') ? open('/var/run/ocp-collector/secrets/pipelinesecret/password','r') do |f|f.read end : ''}"
    scheme http
      target_index_key viaq_index_name
      id_key viaq_msg_id
      remove_keys viaq_index_name
      type_name _doc