Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4281

Fluentd is unable to forward logs to external ipv6 elasticsearch instance

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Before this fix, IPv6 addresses would not be parsed correctly after evaluating host/hosts for placeholders. With this fix, IPv6 addresses are correctly parsed.
    • Bug Fix
    • Log Collection - Sprint 243, Log Collection - Sprint 244, Log Collection - Sprint 245
    • Customer Escalated

    Description

      Description of problem:

      Forwarding logs to external ipv6 elasticsearch instance fails when using fluentd as log collector. Collector pods stream below logs:

      2023-06-29 16:55:05 +0000 [warn]: [elasticsearch_secure] failed to flush the buffer. retry_times=5 next_retry_time=2023-06-29 16:55:35 +0000 chunk="5ff478dd6b35c5b24fccf3c705fc07e2" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (\{:host=>nil, :port=>nil, :scheme=>\"fd00\", :user=>\"elastic\", :password=>\"obfuscated\"}): no implicit conversion of nil into String"
      
        2023-06-29 16:55:05 +0000 [warn]: suppressed same stacktrace
      
      2023-06-29 16:55:35 +0000 [warn]: [elasticsearch_secure] failed to flush the buffer. retry_times=6 next_retry_time=2023-06-29 16:56:34 +0000 chunk="5ff478dd6b35c5b24fccf3c705fc07e2" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (\{:host=>nil, :port=>nil, :scheme=>\"fd00\", :user=>\"elastic\", :password=>\"obfuscated\"}): no implicit conversion of nil into String"
      
        2023-06-29 16:55:35 +0000 [warn]: suppressed same stacktrace
      
      

      Version-Release number of selected component (if applicable):

      All Red Hat OpeShift Logging Operator Versions

      How reproducible:

      100%

      Steps to Reproduce:

      1. Create a ipv6 RHOCP cluster (singlestack will be ok here)
      2. Install Red Hat OpenShift Logging Operator.
      3. Setup a virtual machine which has ipv6 and has elasticsearch up and running (setup this instance in the same network where RHOCP was running to reduce the infra complexity)
      4. Configure CLF on RHOCP cluster to send logs to this externally hosted elasticsearch instance. 

      Actual results:

      The logs are not forwarded to external elasticsearch instance and collector pods stream below error:

      
      2023-06-29 16:55:05 +0000 [warn]: [elasticsearch_secure] failed to flush the buffer. retry_times=5 next_retry_time=2023-06-29 16:55:35 +0000 chunk="5ff478dd6b35c5b24fccf3c705fc07e2" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (\{:host=>nil, :port=>nil, :scheme=>\"fd00\", :user=>\"elastic\", :password=>\"obfuscated\"}): no implicit conversion of nil into String"
      
      

      Expected results:

      The logs should flow normally to external elasticsearch instance.

      Additional info:

      Curl request from collector pods is completed successfully. See below:

      1. curl -u elastic --cacert /var/run/ocp-collector/secrets/es-secret/ca-bundle.crt [https://[IPv6]:|https://[ipv6]/]<Port>

      Enter host password for user 'elastic':

       

      {

        "name" : "node-1",

        "cluster_name" : "elasticsearch-test",

        "cluster_uuid" : "9MYXL3VBT9-k9rtY1nvxfw",

        "version" :

      {     "number" : "7.17.9",     "build_flavor" : "default",     "build_type" : "rpm",     "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",     "build_date" : "2023-01-31T05:34:43.305517834Z",     "build_snapshot" : false,     "lucene_version" : "8.11.1",     "minimum_wire_compatibility_version" : "6.8.0",     "minimum_index_compatibility_version" : "6.0.0-beta1"   }

      ,

        "tagline" : "You Know, for Search"

      }

       

      Attachments

        Activity

          People

            rh-ee-calee Calvin Lee
            rhn-support-dgautam Dhruv Gautam
            Kabir Bharti Kabir Bharti
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: