Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3460

Log forward to Elasticsearch using user/password fails

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • Impediment
    • NEW
    • Hide

      1. Deploy RHOL 5.5

      2. Create a secret for being containing only user and password for being used to log forward to an external elasticsearch

      $ oc get secret external-elasticsearch -o yaml -n openshift-logging 
apiVersion: v1
data:
  password: MTIgYnl0ZXMgbG9uZw==
  username: MTQgYnl0ZXMgbG9uZw==
kind: Secret

...

      3. Configure log forwarding to one external Elasticsearch using the previous secret

      $ oc get clusterlogforwarder instance -o yaml -n openshift-logging 
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance 
  namespace: openshift-logging 
spec:
  outputs:
  - name: elasticsearch 
    type: elasticsearch
    url: http://elasticsearch.example.com:9200
    secret: 
      name: external-elasticsearch
  pipelines:
   - name: apps
     inputRefs:
      - application
     outputRefs:
      - elasticsearch
...

      Check the collector logs failing with the error:

      $ oc logs <collector> -c collector -n openshift-logging
2022-12-26 10:29:34 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_times=6 next_retry_time=2022-12-26 10:30:28 +0000 chunk="5f0b5b13770faab407f0596dbbb6aace" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.example.com\", :port=>9200, :scheme=>\"http\", :user=>\"<elastic username>\\n\", :password=>\"obfuscated\"}): [401] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [<elastic username>\\n] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [<elastic username>\\n] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}"

       

       

      Show
      1. Deploy RHOL 5.5 2. Create a secret for being containing only user and password for being used to log forward to an external elasticsearch $ oc get secret external-elasticsearch -o yaml -n openshift-logging  apiVersion: v1 data:   password: MTIgYnl0ZXMgbG9uZw==   username: MTQgYnl0ZXMgbG9uZw== kind: Secret ... 3. Configure log forwarding to one external Elasticsearch using the previous secret $ oc get clusterlogforwarder instance -o yaml -n openshift-logging apiVersion: "logging.openshift.io/v1" kind: ClusterLogForwarder metadata:   name: instance    namespace: openshift-logging  spec:   outputs:   - name: elasticsearch      type: elasticsearch     url: http: //elasticsearch.example.com:9200   secret:        name: external-elasticsearch   pipelines:    - name: apps      inputRefs:       - application      outputRefs:       - elasticsearch ... Check the collector logs failing with the error: $ oc logs <collector> -c collector -n openshift-logging 2022-12-26 10:29:34 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_times=6 next_retry_time=2022-12-26 10:30:28 +0000 chunk= "5f0b5b13770faab407f0596dbbb6aace" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error= "could not push logs to Elasticsearch cluster ({:host=>\" elasticsearch.example.com\ ", :port=>9200, :scheme=>\" http\ ", :user=>\" <elastic username>\\n\ ", :password=>\" obfuscated\ "}): [401] {\" error\ ":{\" root_cause\ ":[{\" type\ ":\" security_exception\ ",\" reason\ ":\" unable to authenticate user [<elastic username>\\n] for REST request [/_bulk]\ ",\" header\ ":{\" WWW-Authenticate\ ":[\" Basic realm=\\\ "security\\\" charset=\\\ "UTF-8\\\" \ ",\" Bearer realm=\\\ "security\\\" \ ",\" ApiKey\ "]}}],\" type\ ":\" security_exception\ ",\" reason\ ":\" unable to authenticate user [<elastic username>\\n] for REST request [/_bulk]\ ",\" header\ ":{\" WWW-Authenticate\ ":[\" Basic realm=\\\ "security\\\" charset=\\\ "UTF-8\\\" \ ",\" Bearer realm=\\\ "security\\\" \ ",\" ApiKey\ "]}},\" status\ ":401}"    
    • Log Collection - Sprint 233
    • Moderate
    • Customer Escalated

      RHOL version: current latest 5.5.5

      Log forwarding to an Elasticsearch when using username and password is not working with error:

      $ oc logs <collector> -c collector -n openshift-logging
2022-12-26 10:29:34 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_times=6 next_retry_time=2022-12-26 10:30:28 +0000 chunk="5f0b5b13770faab407f0596dbbb6aace" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.example.com\", :port=>9200, :scheme=>\"http\", :user=>\"<elastic username>\\n\", :password=>\"obfuscated\"}): [401] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [<elastic username>\\n] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [<elastic username>\\n] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}"

      The username and the password are valid, for two reasons:

      1. Using curl works
      $ oc -n openshift-logging rsh <collector pod>
sh-4.4# curl -u $(cat /var/run/ocp-collector/secrets/external-elasticsearch/username):$(cat /var/run/ocp-collector/secrets/external-elasticsearch/password) http://elasticsearch.example.com:9200/_cat/nodes
10.88.0.5 59 87 10 0.77 0.82 0.95 mdi * m8Ua3d5

      2. When using for workarounding it, fluentd starts to deliver the logs. The WORKAROUND is:

      Move to Unmanaged the CLO for not modifying the configurations

      $ oc project openshift-logging
$ oc -n openshift-logging patch clusterlogging/instance -p '\{"spec":{"managementState": "Unmanaged"}}' --type=merge

      Backup the collector cm

      $ oc -n openshift-logging get cm collector > cm_collector.yml
$ oc -n openshift-logging edit cm collector

      Needed to change in the 2 sections below:

            <match retry_elasticsearch>
        @type elasticsearch
        @id retry_elasticsearch
        host elasticsearch.example.com
...
        user "#\{File.exists?('/var/run/ocp-collector/secrets/external-elasticsearch/username') ? open('/var/run/ocp-collector/secrets/external-elasticsearch/username','r') do |f|f.read end : ''}"   
        password "#\{File.exists?('/var/run/ocp-collector/secrets/external-elasticsearch/password') ? open('/var/run/ocp-collector/secrets/external-elasticsearch/password','r') do |f|f.read end : ''}"

      And this:  

        <match **>
        @type elasticsearch
        @id elasticsearch
        host elasticsearch.example.com
...
        user "#\{File.exists?('/var/run/ocp-collector/secrets/external-elasticsearch/username') ? open('/var/run/ocp-collector/secrets/external-elasticsearch/username','r') do |f|f.read end : ''}"
        password "#\{File.exists?('/var/run/ocp-collector/secrets/external-elasticsearch/password') ? open('/var/run/ocp-collector/secrets/external-elasticsearch/password','r') do |f|f.read end : ''}"

      Modify the lines below in the previous entries:

              user "#\{File.exists?('/var/run/ocp-collector/secrets/external-elasticsearch/username') ? open('/var/run/ocp-collector/secrets/external-elasticsearch/username','r') do |f|f.read end : ''}"
        password "#\{File.exists?('/var/run/ocp-collector/secrets/external-elasticsearch/password') ? open('/var/run/ocp-collector/secrets/external-elasticsearch/password','r') do |f|f.read end : ''}"

      By:

              user "<username>"
        password "<password>"

      Restart the collectors to take the new configuration:

      $ oc -n openshift-logging delete pods -l component=collector

       

       

       

       

       

       

       

              rh-ee-calee Calvin Lee
              rhn-support-ocasalsa Oscar Casal Sanchez
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: