Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1652

The fluentd doesn't use the new username/password after changing username/password in the pipeline secret.

XMLWordPrintable

    • False
    • False
    • NEW
    • NEW
    • Hide
      There is currently a known issue: If you forward logs to an external Elasticsearch server, and you change a configured value in the pipeline secret, such as a username and password; Then the fluentd forwarder loads the new secret but uses the old value to connect to external ES. This issue happens because the (Red Hat OpenShift Logging?) Operator does not currently monitor secrets for content changes.

      Workaround: If you change the secret, you can force the Fluentd pods to redeploy by entering`oc delete pod -l component=fluentd`.
      Show
      There is currently a known issue: If you forward logs to an external Elasticsearch server, and you change a configured value in the pipeline secret, such as a username and password; Then the fluentd forwarder loads the new secret but uses the old value to connect to external ES. This issue happens because the (Red Hat OpenShift Logging?) Operator does not currently monitor secrets for content changes. Workaround: If you change the secret, you can force the Fluentd pods to redeploy by entering`oc delete pod -l component=fluentd`.
    • Bug Fix

      Description of problem:

      Forward logs to external ES with username/password, then change the username in the pipeline secret, the fluentd can load the new secret but it uses the old user name to connect to external ES.

      Version-Release number of selected component (if applicable):

      cluster-logging.5.2.0-23

      How reproducible:

      Always

      Steps to Reproduce:
      1. deploy external ES, enable user authentication, add users `test2`, set password to `redhat`
      2. forward logs to external ES with user test2

          outputs:
    - name: secure-es
      secret:
        name: test2
      type: elasticsearch
      url: http://elasticsearch-server.bo3dc.svc:9200

      3. change the username to `test1` in the secret/test2

      oc set data secret/test2 --from-literal=username=test1 --from-literal=password=redhat

      4. remove user test2 from external ES and add user `test1` with password `redhat`

      5. check the username in fluentd, it's already changed to `test1` but no fluentd pods restart

      $ oc exec fluentd-trl4g -- cat /var/run/ocp-collector/secrets/test2/username 
Defaulted container "fluentd" out of: fluentd, logfilesmetricexporter
test1

      6. check the fluentd pod logs, fluentd can't connect to ES because it uses the old username `test2`:

      2021-08-06 02:18:44 +0000 [warn]: [secure_es] failed to flush the buffer. retry_time=27 next_retry_seconds=2021-08-06 02:19:47 +0000 chunk="5c8da7a2a1c0f535443d0b077f59ac7d" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-server.bo3dc.svc\", :port=>9200, :scheme=>\"http\", :user=>\"test2\", :password=>\"obfuscated\"}): [401] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [test2] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [test2] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}"
  2021-08-06 02:18:44 +0000 [warn]: suppressed same stacktrace

      Actual results:

      Expected results:

      Additional info: 

      workaround:

      oc delete pod -l component=fluentd

              Unassigned Unassigned
              qitang@redhat.com Qiaoling Tang
              Qiaoling Tang Qiaoling Tang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: