Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5602

Vector forwarding authentication issue when password contains special characters

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, Vector configurations with field values that included a single dollar sign ($) were not parsed correctly. After this update, field values containing a single dollar sign are automatically replaced with two dollar signs ($$), allowing Vector to parse the values correctly.
      Show
      Before this update, Vector configurations with field values that included a single dollar sign ($) were not parsed correctly. After this update, field values containing a single dollar sign are automatically replaced with two dollar signs ($$), allowing Vector to parse the values correctly.
    • Bug Fix
    • High
    • Hide

      1) Deploy an external third-party system. In the troubleshooting tests I deployed an external ES using user+password with special characters 

      user: elastic
      password: 4xpXpbq&rmPCF576N$Bz
      

      Checking that the external third-party is properly configured:

      sh-5.1$ curl -u elastic:4xpXpbq&rmPCF576N$Bz http://elasticsearch-server.usermanaged-es.svc:9200/_cat/nodes 127.0.0.1 11 98 13 0.31 0.31 0.32 mdi * elasticsearch-server

      3) Deploy the logging stack with Vector as collector

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        creationTimestamp: "2024-05-30T13:24:16Z"
        generation: 3
        name: instance
        namespace: openshift-logging
        resourceVersion: "583997"
        uid: 3248b178-e196-46f2-ac31-7b04805e04d9
      spec:
        outputs:
        - elasticsearch:
            version: 8
          name: es-created-by-user
          secret:
            name: elasticsearchsecret
          type: elasticsearch
          url: http://elasticsearch-server.usermanaged-es.svc:9200
        pipelines:
        - inputRefs:
          - application
          - infrastructure
          - audit 
          name: forward-to-external-es
          outputRefs:
          - es-created-by-user
      

      4) Check the collector logs:

      2024-05-30T18:08:31.455319Z ERROR sink{component_kind="sink" component_id=output_es_created_by_user component_type=elasticsearch}: vector::sinks::util::retries: Not retriable; dropping the request. reason="client-side error, 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk?timeout=2147483648s]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk?timeout=2147483648s]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}" internal_log_rate_limit=true

      5) Check the user/password in Vector.toml config file:

      cat vector.toml
      # Basic Auth Config
      [sinks.output_es_created_by_user.auth]
      strategy = "basic"
      user = "elastic"
      password = "4xpXpbq&rmPCF576N$Bz"
      

      6)Modifying the collector from Vector to Fluentd, the evidence is that the logs are being forwarder:

      sh-5.1# cd /var/run/ocp-collector/secrets/
      sh-5.1# ls
      elasticsearchsecret
      sh-5.1# cd elasticsearchsecret/
      sh-5.1# ls
      password  username
      sh-5.1# cat password 
      4xpXpbq&rmPCF576N$Bz
      
      sh-5.1# curl -u elastic:'4xpXpbq&rmPCF576N$Bz' http://elasticsearch-server.usermanaged-es.svc:9200/_cat/indices
      yellow open audit-write UpQtBGnMRLyeAxrh7-k5oA 1 1 84079 0  65.5mb  65.5mb
      yellow open app-write   2WYeHu7gSFmxK0CHAOpNog 1 1   905 0 690.6kb 690.6kb
      yellow open infra-write j91j1Of_R5uRtk8MMygD8w 1 1 90453 0  57.3mb  57.3mb
      sh-5.1# 
      sh-5.1# curl -u elastic:'4xpXpbq&rmPCF576N$Bz' http://elasticsearch-server.usermanaged-es.svc:9200/_cat/indices
      yellow open audit-write UpQtBGnMRLyeAxrh7-k5oA 1 1 84079 0 96.1mb 96.1mb
      yellow open app-write   2WYeHu7gSFmxK0CHAOpNog 1 1   905 0  5.3mb  5.3mb
      yellow open infra-write j91j1Of_R5uRtk8MMygD8w 1 1 90453 0 81.1mb 81.1mb
      

       

       

       

       

       

      Show
      1) Deploy an external third-party system. In the troubleshooting tests I deployed an external ES using user+password with special characters  user: elastic password: 4xpXpbq&rmPCF576N$Bz Checking that the external third-party is properly configured: sh-5.1$ curl -u elastic:4xpXpbq&rmPCF576N$Bz http: //elasticsearch-server.usermanaged-es.svc:9200/_cat/nodes 127.0.0.1 11 98 13 0.31 0.31 0.32 mdi * elasticsearch-server 3) Deploy the logging stack with Vector as collector apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata:   creationTimestamp: "2024-05-30T13:24:16Z"   generation: 3   name: instance   namespace: openshift-logging   resourceVersion: "583997"   uid: 3248b178-e196-46f2-ac31-7b04805e04d9 spec:   outputs:   - elasticsearch:       version: 8     name: es-created-by-user     secret:       name: elasticsearchsecret     type: elasticsearch     url: http: //elasticsearch-server.usermanaged-es.svc:9200   pipelines:   - inputRefs:     - application     - infrastructure - audit     name: forward-to-external-es     outputRefs:     - es-created-by-user 4) Check the collector logs: 2024-05-30T18:08:31.455319Z ERROR sink{component_kind= "sink" component_id=output_es_created_by_user component_type=elasticsearch}: vector::sinks::util::retries: Not retriable; dropping the request. reason= "client-side error, 401 Unauthorized: {\" error\ ":{\" root_cause\ ":[{\" type\ ":\" security_exception\ ",\" reason\ ":\" unable to authenticate user [elastic] for REST request [/_bulk?timeout=2147483648s]\ ",\" header\ ":{\" WWW-Authenticate\ ":[\" Basic realm=\\\ "security\\\" charset=\\\ "UTF-8\\\" \ ",\" ApiKey\ "]}}],\" type\ ":\" security_exception\ ",\" reason\ ":\" unable to authenticate user [elastic] for REST request [/_bulk?timeout=2147483648s]\ ",\" header\ ":{\" WWW-Authenticate\ ":[\" Basic realm=\\\ "security\\\" charset=\\\ "UTF-8\\\" \ ",\" ApiKey\ "]}},\" status\ ":401}" internal_log_rate_limit= true 5) Check the user/password in Vector.toml config file: cat vector.toml # Basic Auth Config [sinks.output_es_created_by_user.auth] strategy = "basic" user = "elastic" password = "4xpXpbq&rmPCF576N$Bz" 6)Modifying the collector from Vector to Fluentd, the evidence is that the logs are being forwarder: sh-5.1# cd / var /run/ocp-collector/secrets/ sh-5.1# ls elasticsearchsecret sh-5.1# cd elasticsearchsecret/ sh-5.1# ls password  username sh-5.1# cat password  4xpXpbq&rmPCF576N$Bz sh-5.1# curl -u elastic: '4xpXpbq&rmPCF576N$Bz' http: //elasticsearch-server.usermanaged-es.svc:9200/_cat/indices yellow open audit-write UpQtBGnMRLyeAxrh7-k5oA 1 1 84079 0  65.5mb  65.5mb yellow open app-write   2WYeHu7gSFmxK0CHAOpNog 1 1   905 0 690.6kb 690.6kb yellow open infra-write j91j1Of_R5uRtk8MMygD8w 1 1 90453 0  57.3mb  57.3mb sh-5.1#  sh-5.1# curl -u elastic: '4xpXpbq&rmPCF576N$Bz' http: //elasticsearch-server.usermanaged-es.svc:9200/_cat/indices yellow open audit-write UpQtBGnMRLyeAxrh7-k5oA 1 1 84079 0 96.1mb 96.1mb yellow open app-write   2WYeHu7gSFmxK0CHAOpNog 1 1   905 0  5.3mb  5.3mb yellow open infra-write j91j1Of_R5uRtk8MMygD8w 1 1 90453 0 81.1mb 81.1mb          
    • Log Collection - Sprint 258, Log Collection - Sprint 259
    • Important
    • Customer Escalated, Customer Reported

      Description of problem:

      Forwarding the logs using Vector to an external third-party with user/password authentication is observed a 401 Error un collector pods even being the user and password corrects in the Vector configuration. 

      If we modify the collector from Vector to Fluentd, we observe the logs are sent properly. 

      I suspect  Vector stores the username/password with double quotes and is cannot read the special values correctly.

      In real cases, this has happened with a migration from Fluentd to Vector and testing, the same problem occurs when it is a new logging deployment with Vector and using special characters in the password.

       

      Version-Release number of selected component (if applicable):

      Red Hat Openshift Logging 5.9

       

              rh-ee-calee Calvin Lee
              acandelp Adrian Candel
              Kabir Bharti Kabir Bharti
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: