Details
-
Task
-
Resolution: Done
-
Critical
-
Logging 5.2
-
devex docs #206 Aug 19-Sep 9
-
3
-
Release Notes
-
Low
-
Undefined
-
Not Yet Documented
Description
Description of problem:
Forward logs to external ES with username/password, then change the username in the pipeline secret, the fluentd can load the new secret but it uses the old user name to connect to external ES.
Version-Release number of selected component (if applicable):
cluster-logging.5.2.0-23
How reproducible:
Always
Steps to Reproduce:
1. deploy external ES, enable user authentication, add users `test2`, set password to `redhat`
2. forward logs to external ES with user test2
outputs:
- name: secure-es
secret:
name: test2
type: elasticsearch
url: http://elasticsearch-server.bo3dc.svc:9200
3. change the username to `test1` in the secret/test2
oc set data secret/test2 --from-literal=username=test1 --from-literal=password=redhat
4. remove user test2 from external ES and add user `test1` with password `redhat`
5. check the username in fluentd, it's already changed to `test1` but no fluentd pods restart
$ oc exec fluentd-trl4g -- cat /var/run/ocp-collector/secrets/test2/username Defaulted container "fluentd" out of: fluentd, logfilesmetricexporter test1
6. check the fluentd pod logs, fluentd can't connect to ES because it uses the old username `test2`:
2021-08-06 02:18:44 +0000 [warn]: [secure_es] failed to flush the buffer. retry_time=27 next_retry_seconds=2021-08-06 02:19:47 +0000 chunk="5c8da7a2a1c0f535443d0b077f59ac7d" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-server.bo3dc.svc\", :port=>9200, :scheme=>\"http\", :user=>\"test2\", :password=>\"obfuscated\"}): [401] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [test2] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [test2] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}" 2021-08-06 02:18:44 +0000 [warn]: suppressed same stacktrace
Actual results:
Expected results:
Additional info:
workaround:
oc delete pod -l component=fluentd
Attachments
Issue Links
- documents
-
LOG-1652 The fluentd doesn't use the new username/password after changing username/password in the pipeline secret.
-
- Closed
-
- relates to
-
-
- Closed
-
- links to