Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2843

tls.cert, tls.key and passphrase are not passed to the fluentd configuration when forwarding logs using syslog over TLS

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      In previous releases of OpenShift Logging, when forwarding logs using syslog over TLS, the configuration for the fluentd log forwarder only contained the 'ca-bundle.crt' entry, but not the 'tls.crt', 'tls.key' or 'passphrase' entries, even if specified in the corresponding ClusterLogForwarder output. This issues has now been addressed and ca-bundle.crt, tls.crt, tls.key and passphrase are all passed to the configuration of the fluentd log forwarder.
      Show
      In previous releases of OpenShift Logging, when forwarding logs using syslog over TLS, the configuration for the fluentd log forwarder only contained the 'ca-bundle.crt' entry, but not the 'tls.crt', 'tls.key' or 'passphrase' entries, even if specified in the corresponding ClusterLogForwarder output. This issues has now been addressed and ca-bundle.crt, tls.crt, tls.key and passphrase are all passed to the configuration of the fluentd log forwarder.
    • Hide

      Deploy RHOL

      $ oc get csv -n openshift-logging
      NAME                           DISPLAY                            VERSION   REPLACES   PHASE
      cluster-logging.5.4.3          Red Hat OpenShift Logging          5.4.3                Succeeded
      elasticsearch-operator.5.4.3   OpenShift Elasticsearch Operator   5.4.3                Succeeded
       

      Create a secret containing `ca-bundle.crt`, `tls.crt` and `tls.key` for being used later in the clusterLogForwarder for sending to syslog:

      $ oc get secret syslog -o yaml 
      apiVersion: v1
      data:
        ca-bundle.crt: xxxxx
        tls.crt: xxxxx
        tls.key: xxxxx
      kind: Secret
      metadata:
        creationTimestamp: "2022-07-20T15:01:27Z"
        name: syslog
        namespace: openshift-logging
        resourceVersion: "7350054"
        uid: 5a4f01b3-e6f8-4f55-a284-9976217810a6
      type: Opaque

      Configure clusterLogForwarder for sending logs to rsyslog using tls 

      $ cat clusterlogforwarder.yaml
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        labels:
          app.kubernetes.io/instance: logging
        name: instance
        namespace: openshift-logging
      spec:
        outputs:
          - name: syslog
            secret:
              name: syslog
            syslog:
              addLogSource: true
              facility: syslog
              rfc: RFC3164
              severity: informational
            type: syslog
            url: 'tls://localhost:514'
        pipelines:
          - inputRefs:
              - application
            name: application-logs
            outputRefs:
              - syslog
       $ oc create -f clusterlogforwarder.yaml

      Check the fluentd configuration generated that it only contains the `ca-bundle.crt` and not the `tls.key` and `tls.crt`:

      $ oc get cm collector -o json |jq '.data."fluent.conf"' -r
      ...
        <match journal.** system.var.log**>
          @type remote_syslog
          @id syslog_journal
          host localhost
          port 514
          rfc rfc3164
          facility syslog
          severity informational
          program ${$.systemd.u.SYSLOG_IDENTIFIER}
          protocol tcp
          packet_size 4096
          hostname "#{ENV['NODE_NAME']}"
          tls true
          ca_file '/var/run/ocp-collector/secrets/syslog/ca-bundle.crt'
          timeout 60
          timeout_exception true
          keep_alive true
          keep_alive_idle 75
          keep_alive_cnt 9
          keep_alive_intvl 7200
          <format>
            @type json
          </format>
          <buffer $.systemd.u.SYSLOG_IDENTIFIER>
            @type file
            path '/var/lib/fluentd/syslog_journal'
            flush_mode interval
            flush_interval 1s
            flush_thread_count 2
            retry_type exponential_backoff
            retry_wait 1s
            retry_max_interval 60s
            retry_timeout 60m
            queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32'}"
            total_limit_size "#{ENV['TOTAL_LIMIT_SIZE_PER_BUFFER'] || '8589934592'}"
            chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '8m'}"
            overflow_action block
            disable_chunk_backup true
          </buffer>
        </match>  <match **>
          @type remote_syslog
          @id syslog
          host localhost
          port 514
          rfc rfc3164
          facility syslog
          severity informational
          protocol tcp
          packet_size 4096
          hostname "#{ENV['NODE_NAME']}"
          tls true
          ca_file '/var/run/ocp-collector/secrets/syslog/ca-bundle.crt'
          timeout 60
          timeout_exception true
          keep_alive true
          keep_alive_idle 75
          keep_alive_cnt 9
          keep_alive_intvl 7200
          <format>
            @type json
          </format>
          <buffer>
            @type file
            path '/var/lib/fluentd/syslog'
            flush_mode interval
            flush_interval 1s
            flush_thread_count 2
            retry_type exponential_backoff
            retry_wait 1s
            retry_max_interval 60s
            retry_timeout 60m
            queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32'}"
            total_limit_size "#{ENV['TOTAL_LIMIT_SIZE_PER_BUFFER'] || '8589934592'}"
            chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '8m'}"
            overflow_action block
            disable_chunk_backup true
          </buffer>
        </match>
      </label>
      ....
      Show
      Deploy RHOL $ oc get csv -n openshift-logging NAME                           DISPLAY                            VERSION   REPLACES   PHASE cluster-logging.5.4.3          Red Hat OpenShift Logging          5.4.3                Succeeded elasticsearch- operator .5.4.3   OpenShift Elasticsearch Operator   5.4.3                Succeeded Create a secret containing `ca-bundle.crt`, `tls.crt` and `tls.key` for being used later in the clusterLogForwarder for sending to syslog: $ oc get secret syslog -o yaml  apiVersion: v1 data:   ca-bundle.crt: xxxxx   tls.crt: xxxxx   tls.key: xxxxx kind: Secret metadata:   creationTimestamp: "2022-07-20T15:01:27Z"   name: syslog   namespace: openshift-logging   resourceVersion: "7350054"   uid: 5a4f01b3-e6f8-4f55-a284-9976217810a6 type: Opaque Configure clusterLogForwarder for sending logs to rsyslog using tls   $ cat clusterlogforwarder.yaml apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata:   labels:     app.kubernetes.io/instance: logging   name: instance   namespace: openshift-logging spec:   outputs:     - name: syslog       secret:         name: syslog       syslog:         addLogSource: true         facility: syslog         rfc: RFC3164         severity: informational       type: syslog       url: 'tls: //localhost:514'   pipelines:     - inputRefs:         - application       name: application-logs       outputRefs:         - syslog $ oc create -f clusterlogforwarder.yaml Check the fluentd configuration generated that it only contains the `ca-bundle.crt` and not the `tls.key` and `tls.crt`: $ oc get cm collector -o json |jq '.data. "fluent.conf" ' -r ... <match journal.** system. var .log**>     @type remote_syslog     @id syslog_journal     host localhost     port 514     rfc rfc3164     facility syslog     severity informational     program ${$.systemd.u.SYSLOG_IDENTIFIER}     protocol tcp     packet_size 4096     hostname "#{ENV[ 'NODE_NAME' ]}"     tls true     ca_file '/ var /run/ocp-collector/secrets/syslog/ca-bundle.crt'     timeout 60     timeout_exception true     keep_alive true     keep_alive_idle 75     keep_alive_cnt 9     keep_alive_intvl 7200     <format>       @type json     </format>     <buffer $.systemd.u.SYSLOG_IDENTIFIER>       @type file       path '/ var /lib/fluentd/syslog_journal'       flush_mode interval       flush_interval 1s       flush_thread_count 2       retry_type exponential_backoff       retry_wait 1s       retry_max_interval 60s       retry_timeout 60m       queued_chunks_limit_size "#{ENV[ 'BUFFER_QUEUE_LIMIT' ] || '32' }"       total_limit_size "#{ENV[ 'TOTAL_LIMIT_SIZE_PER_BUFFER' ] || '8589934592' }"       chunk_limit_size "#{ENV[ 'BUFFER_SIZE_LIMIT' ] || '8m' }"       overflow_action block       disable_chunk_backup true     </buffer>   </match>  <match **>     @type remote_syslog     @id syslog     host localhost     port 514     rfc rfc3164     facility syslog     severity informational     protocol tcp     packet_size 4096     hostname "#{ENV[ 'NODE_NAME' ]}"     tls true     ca_file '/ var /run/ocp-collector/secrets/syslog/ca-bundle.crt'     timeout 60     timeout_exception true     keep_alive true     keep_alive_idle 75     keep_alive_cnt 9     keep_alive_intvl 7200     <format>       @type json     </format>     <buffer>       @type file       path '/ var /lib/fluentd/syslog'       flush_mode interval       flush_interval 1s       flush_thread_count 2       retry_type exponential_backoff       retry_wait 1s       retry_max_interval 60s       retry_timeout 60m       queued_chunks_limit_size "#{ENV[ 'BUFFER_QUEUE_LIMIT' ] || '32' }"       total_limit_size "#{ENV[ 'TOTAL_LIMIT_SIZE_PER_BUFFER' ] || '8589934592' }"       chunk_limit_size "#{ENV[ 'BUFFER_SIZE_LIMIT' ] || '8m' }"       overflow_action block       disable_chunk_backup true     </buffer>   </match> </label> ....
    • Log Collection - Sprint 224, Log Collection - Sprint 225, Log Collection - Sprint 226

      When defined for forwarding logs using the syslog protocol with tls and the secret contains the `ca-bundle.crt`, `tls.key` and `tls.crt`, the real configuration generated for fluentd defines only `ca-bundle.crt` omitting `tls.key` and `tls.cert`.

            [LOG-2843] tls.cert, tls.key and passphrase are not passed to the fluentd configuration when forwarding logs using syslog over TLS

            CPaaS Service Account mentioned this issue in merge request !728 of openshift-logging / Log Collection Midstream on branch openshift-logging-5.5-rhel-8_upstream_382d8900e3c32fe8434190b9d9cb1343:

            Updated US source to: 1c84d31 Merge pull request #1839 from syedriko/syedriko-log2843-take-2-release-5.5-cherrypick

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in merge request !728 of openshift-logging / Log Collection Midstream on branch openshift-logging-5.5-rhel-8_ upstream _382d8900e3c32fe8434190b9d9cb1343 : Updated US source to: 1c84d31 Merge pull request #1839 from syedriko/syedriko-log2843-take-2-release-5.5-cherrypick

            CPaaS Service Account mentioned this issue in merge request !701 of openshift-logging / Log Collection Midstream on branch openshift-logging-5.5-rhel-8_upstream_942709b698c7a35b98c3d504ecc75746:

            Updated US source to: 817f413 Merge pull request #1835 from openshift-cherrypick-robot/cherry-pick-1643-to-release-5.5

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in merge request !701 of openshift-logging / Log Collection Midstream on branch openshift-logging-5.5-rhel-8_ upstream _942709b698c7a35b98c3d504ecc75746 : Updated US source to: 817f413 Merge pull request #1835 from openshift-cherrypick-robot/cherry-pick-1643-to-release-5.5

            rhn-support-ocasalsa I've cloned this JIRA for 5.5 -  https://issues.redhat.com/browse/LOG-3533 and started cherry-picking. Do we need to go as far back as 5.4?

            Sergey Yedrikov added a comment - rhn-support-ocasalsa I've cloned this JIRA for 5.5 -   https://issues.redhat.com/browse/LOG-3533 and started cherry-picking. Do we need to go as far back as 5.4?

            Test passed using cluster-logging-rhel8-operator/images/v5.6.0-16. 

                 tls true
                 client_cert_key '/var/run/ocp-collector/secrets/rsyslog/tls.key'
                 client_cert '/var/run/ocp-collector/secrets/rsyslog/tls.crt'
                 ca_file '/var/run/ocp-collector/secrets/rsyslog/ca-bundle.crt'
                 client_cert_key_password "#{File.exists?('/var/run/ocp-collector/secrets/rsyslog/passphrase') ? open('/var/run/ocp-collector/secrets/rsyslog/passphrase', 'r') do |f|f.read end : ''}" 

            Qiaoling Tang added a comment - Test passed using cluster-logging-rhel8-operator/images/v5.6.0-16.       tls true      client_cert_key '/ var /run/ocp-collector/secrets/rsyslog/tls.key'      client_cert '/ var /run/ocp-collector/secrets/rsyslog/tls.crt'      ca_file '/ var /run/ocp-collector/secrets/rsyslog/ca-bundle.crt'      client_cert_key_password "#{File.exists?( '/ var /run/ocp-collector/secrets/rsyslog/passphrase' ) ? open( '/ var /run/ocp-collector/secrets/rsyslog/passphrase' , 'r' ) do |f|f.read end : ''}"

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in merge request !195 of openshift-logging / Log Collection Midstream on branch openshift-logging-5.6-rhel-8_ upstream _1caae4555e1335676a8e91aa825c9a14 : Updated 2 upstream sources

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in merge request !192 of openshift-logging / Log Collection Midstream on branch openshift-logging-5.6-rhel-8_ upstream _eb8fc3a95e5914f9e191c6540bd06be1 : Updated 2 upstream sources

            qitang@redhat.com The PR that added support for passphrase has merged in the upstream, look for it on your end!

            Sergey Yedrikov added a comment - qitang@redhat.com The PR that added support for passphrase has merged in the upstream, look for it on your end!

            qitang@redhat.com Thanks for catching, I'll add passphrase.

            Sergey Yedrikov added a comment - qitang@redhat.com Thanks for catching, I'll add passphrase.

            syedriko_sub@redhat.com I test using latest cluster-logging.v5.6.0, I can see `client_cert_key` and `client_cert` are added into fluent.conf when there have `tls.key` and `tls.crt` in the secret. However, when I add `passphrase` into the secret, I don't see the passphrase in fluent.conf. 

            $ oc get secret rsyslog -oyaml
            apiVersion: v1
            data:
              ca-bundle.crt: LS0tLS1CRUdJTiS0tCg==
              passphrase: dGVzdHJlZGFodA==
              tls.crt: LS0tLS1CRUdJTitLQo=
              tls.key: LS0tLS1CRUBLRVktLS0tLQo= 
            
            $ cat fluent.conf 
                tls true
                client_cert_key '/var/run/ocp-collector/secrets/rsyslog/tls.key'
                client_cert '/var/run/ocp-collector/secrets/rsyslog/tls.crt'
                ca_file '/var/run/ocp-collector/secrets/rsyslog/ca-bundle.crt'
                timeout 60

            Qiaoling Tang added a comment - syedriko_sub@redhat.com I test using latest cluster-logging.v5.6.0, I can see `client_cert_key` and `client_cert` are added into fluent.conf when there have `tls.key` and `tls.crt` in the secret. However, when I add `passphrase` into the secret, I don't see the passphrase in fluent.conf.  $ oc get secret rsyslog -oyaml apiVersion: v1 data:   ca-bundle.crt: LS0tLS1CRUdJTiS0tCg==   passphrase: dGVzdHJlZGFodA==   tls.crt: LS0tLS1CRUdJTitLQo=   tls.key: LS0tLS1CRUBLRVktLS0tLQo= $ cat fluent.conf      tls true     client_cert_key '/ var /run/ocp-collector/secrets/rsyslog/tls.key'     client_cert '/ var /run/ocp-collector/secrets/rsyslog/tls.crt'     ca_file '/ var /run/ocp-collector/secrets/rsyslog/ca-bundle.crt'     timeout 60

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in merge request !116 of openshift-logging / Log Collection Midstream on branch openshift-logging-5.6-rhel-8_ upstream _cc5ac5e5bfea83f255c33d63fbec3183 : Updated 3 upstream sources

              syedriko_sub@redhat.com Sergey Yedrikov
              rhn-support-ocasalsa Oscar Casal Sanchez
              Qiaoling Tang Qiaoling Tang
              Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: