Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1652

The fluentd doesn't use the new username/password after changing username/password in the pipeline secret.

    XMLWordPrintable

Details

    • False
    • False
    • NEW
    • NEW
    • Hide
      There is currently a known issue: If you forward logs to an external Elasticsearch server, and you change a configured value in the pipeline secret, such as a username and password; Then the fluentd forwarder loads the new secret but uses the old value to connect to external ES. This issue happens because the (Red Hat OpenShift Logging?) Operator does not currently monitor secrets for content changes.

      Workaround: If you change the secret, you can force the Fluentd pods to redeploy by entering`oc delete pod -l component=fluentd`.
      Show
      There is currently a known issue: If you forward logs to an external Elasticsearch server, and you change a configured value in the pipeline secret, such as a username and password; Then the fluentd forwarder loads the new secret but uses the old value to connect to external ES. This issue happens because the (Red Hat OpenShift Logging?) Operator does not currently monitor secrets for content changes. Workaround: If you change the secret, you can force the Fluentd pods to redeploy by entering`oc delete pod -l component=fluentd`.
    • Bug Fix

    Description

      Description of problem:

      Forward logs to external ES with username/password, then change the username in the pipeline secret, the fluentd can load the new secret but it uses the old user name to connect to external ES.

      Version-Release number of selected component (if applicable):

      cluster-logging.5.2.0-23

      How reproducible:

      Always

      Steps to Reproduce:
      1. deploy external ES, enable user authentication, add users `test2`, set password to `redhat`
      2. forward logs to external ES with user test2

          outputs:
    - name: secure-es
      secret:
        name: test2
      type: elasticsearch
      url: http://elasticsearch-server.bo3dc.svc:9200

      3. change the username to `test1` in the secret/test2

      oc set data secret/test2 --from-literal=username=test1 --from-literal=password=redhat

      4. remove user test2 from external ES and add user `test1` with password `redhat`

      5. check the username in fluentd, it's already changed to `test1` but no fluentd pods restart

      $ oc exec fluentd-trl4g -- cat /var/run/ocp-collector/secrets/test2/username 
Defaulted container "fluentd" out of: fluentd, logfilesmetricexporter
test1

      6. check the fluentd pod logs, fluentd can't connect to ES because it uses the old username `test2`:

      2021-08-06 02:18:44 +0000 [warn]: [secure_es] failed to flush the buffer. retry_time=27 next_retry_seconds=2021-08-06 02:19:47 +0000 chunk="5c8da7a2a1c0f535443d0b077f59ac7d" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-server.bo3dc.svc\", :port=>9200, :scheme=>\"http\", :user=>\"test2\", :password=>\"obfuscated\"}): [401] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [test2] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [test2] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}"
  2021-08-06 02:18:44 +0000 [warn]: suppressed same stacktrace

      Actual results:

      Expected results:

      Additional info: 

      workaround:

      oc delete pod -l component=fluentd

      Attachments

        Issue Links

          is documented by

          Task - Represents a small unit of work that is not end-user facing. RHDEVDOCS-3223 Create RN Known Issue for LOG-1652 "The fluentd doesn't use the new username/password..."

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is related to

          Story - Used to indicate the smallest unit of work for a team to complete and is end user-facing. Created by Jira Software - do not edit or delete. LOG-974 Enable username and password for the Elasticsearch output type

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          links to

          GitHub openshift/openshift-docs#35445: RHDEVDOCS-3226 Logging 5.2 docs aggregated branch

          GitHub openshift/openshift-docs#37537: Rhdevdocs 3226 rn update

          Activity

            People

              Unassigned Unassigned
              qitang@redhat.com Qiaoling Tang
              Qiaoling Tang Qiaoling Tang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: