Uploaded image for project: 'Docs for Red Hat Developers'
  1. Docs for Red Hat Developers
  2. RHDEVDOCS-3223

Create RN Known Issue for LOG-1652 "The fluentd doesn't use the new username/password..."

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Critical Critical
    • Logging 5.2
    • Logging 5.2
    • Logging

      Description of problem:

      Forward logs to external ES with username/password, then change the username in the pipeline secret, the fluentd can load the new secret but it uses the old user name to connect to external ES.

      Version-Release number of selected component (if applicable):

      cluster-logging.5.2.0-23

      How reproducible:

      Always

      Steps to Reproduce:
      1. deploy external ES, enable user authentication, add users `test2`, set password to `redhat`
      2. forward logs to external ES with user test2

          outputs:
    - name: secure-es
      secret:
        name: test2
      type: elasticsearch
      url: http://elasticsearch-server.bo3dc.svc:9200

      3. change the username to `test1` in the secret/test2

      oc set data secret/test2 --from-literal=username=test1 --from-literal=password=redhat

      4. remove user test2 from external ES and add user `test1` with password `redhat`

      5. check the username in fluentd, it's already changed to `test1` but no fluentd pods restart

      $ oc exec fluentd-trl4g -- cat /var/run/ocp-collector/secrets/test2/username 
Defaulted container "fluentd" out of: fluentd, logfilesmetricexporter
test1

      6. check the fluentd pod logs, fluentd can't connect to ES because it uses the old username `test2`:

      2021-08-06 02:18:44 +0000 [warn]: [secure_es] failed to flush the buffer. retry_time=27 next_retry_seconds=2021-08-06 02:19:47 +0000 chunk="5c8da7a2a1c0f535443d0b077f59ac7d" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-server.bo3dc.svc\", :port=>9200, :scheme=>\"http\", :user=>\"test2\", :password=>\"obfuscated\"}): [401] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [test2] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [test2] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}"
  2021-08-06 02:18:44 +0000 [warn]: suppressed same stacktrace

      Actual results:

      Expected results:

      Additional info: 

      workaround:

      oc delete pod -l component=fluentd

              rdlugyhe Rolfe Dlugy-Hegwer
              rdlugyhe Rolfe Dlugy-Hegwer
              Rolfe Dlugy-Hegwer Rolfe Dlugy-Hegwer
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: