Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1372

CLO always set some incorrect default values in fluent.conf when forward log to external elasticsearch with secret

    XMLWordPrintable

Details

    • Logging (Core) - Sprint 202, Logging (Core) - Sprint 203

    Description

      Description of problem:

      When forward log to external elasticsearch with secret, CLO always sets below configurations in fluent.conf even when these keys are not existed:

            client_key '/var/run/ocp-collector/secrets/pipelinesecret/tls.key'
      client_cert '/var/run/ocp-collector/secrets/pipelinesecret/tls.crt'
      ca_file '/var/run/ocp-collector/secrets/pipelinesecret/ca-bundle.crt'
      ssl_version TLSv1_2

      Besides, the `scheme` is always set to `https` even when it is `http` in clf/instance.
      Version-Release number of selected component (if applicable):

      latest code cloned from https://github.com/openshift/cluster-logging-operator master branch
      How reproducible:

      Always

      Steps to Reproduce:
      1. deploy CLO

      2. deploy an external elasticsearch server, enable user authentication, don't enable http ssl and transport ssl

      3. create a secret in openshift-logging namespace which contains username and password

      oc create secret generic $secret-name --from-literal=username=$username --from-literal=password=$password

      4. create clf/instance to forward logs to the external elasticsearch server with the secret created in step 3

      cat << EOF | oc create -f -
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
  - name: secure-es
    secret:
      name: pipelinesecret
    type: elasticsearch
    url: http://elasticsearch-server.a2h8p.svc:9200
  pipelines:
  - inputRefs:
    - application
    - infrastructure
    - audit
    name: forward-logs
    outputRefs:
    - secure-es
EOF

      5. deploy fluentd pods

      6. check fluent.conf in cm/fluentd and secret keys  in fluentd pod

            @type elasticsearch
      @id secure_es
      host elasticsearch-server.a2h8p.svc
      port 9200
      verify_es_version_at_startup false
    user "#{File.exists?('/var/run/ocp-collector/secrets/pipelinesecret/username') ? open('/var/run/ocp-collector/secrets/pipelinesecret/username','r') do |f|f.read end : ''}"
    password "#{File.exists?('/var/run/ocp-collector/secrets/pipelinesecret/password') ? open('/var/run/ocp-collector/secrets/pipelinesecret/password','r') do |f|f.read end : ''}"
    scheme https
      ssl_version TLSv1_2
      target_index_key viaq_index_name
      id_key viaq_msg_id
      remove_keys viaq_index_name
      client_key '/var/run/ocp-collector/secrets/pipelinesecret/tls.key'
      client_cert '/var/run/ocp-collector/secrets/pipelinesecret/tls.crt'
      ca_file '/var/run/ocp-collector/secrets/pipelinesecret/ca-bundle.crt'
      type_name _doc
      $ oc rsh fluentd-r9z2x
sh-4.4# ls /var/run/ocp-collector/secrets/pipelinesecret/
password  username
sh-4.4#

      7. check logs in external elasticsearch, no logs received

      8. check fluentd pod log, lots of error logs:

        2021-05-14 01:15:14 +0000 [warn]: suppressed same stacktrace
2021-05-14 01:16:18 +0000 [warn]: [secure_es] failed to flush the buffer. retry_time=12 next_retry_seconds=2021-05-14 01:17:19 +0000 chunk="5c23fd61a8ab91a09364881da73ec885" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-server.a2h8p.svc\", :port=>9200, :scheme=>\"https\", :user=>\"qitang\", :password=>\"obfuscated\"}): Problem with the local SSL certificate"
  2021-05-14 01:16:18 +0000 [warn]: suppressed same stacktrace
2021-05-14 01:16:18 +0000 [warn]: [secure_es] failed to flush the buffer. retry_time=13 next_retry_seconds=2021-05-14 01:17:14 +0000 chunk="5c23fd61ca56f4a6f7b18d59b716f973" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-server.a2h8p.svc\", :port=>9200, :scheme=>\"https\", :user=>\"qitang\", :password=>\"obfuscated\"}): Problem with the local SSL certificate"
  2021-05-14 01:16:18 +0000 [warn]: suppressed same stacktrace

      Actual results:

      please find it in step 6
      Expected results:

            @type elasticsearch
      @id secure_es
      host elasticsearch-server.a2h8p.svc
      port 9200
      verify_es_version_at_startup false
    user "#{File.exists?('/var/run/ocp-collector/secrets/pipelinesecret/username') ? open('/var/run/ocp-collector/secrets/pipelinesecret/username','r') do |f|f.read end : ''}"
    password "#{File.exists?('/var/run/ocp-collector/secrets/pipelinesecret/password') ? open('/var/run/ocp-collector/secrets/pipelinesecret/password','r') do |f|f.read end : ''}"
    scheme http
      target_index_key viaq_index_name
      id_key viaq_msg_id
      remove_keys viaq_index_name
      type_name _doc

      Additional info:

      Attachments

        Activity

          People

            ikarpukh Igor Karpukhin (Inactive)
            qitang@redhat.com Qiaoling Tang
            Qiaoling Tang Qiaoling Tang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: