Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4811

Fluentd is unable to forward logs to external ipv6 elasticsearch instance

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • NEW
    • Before this fix, IPv6 addresses would not be parsed correctly after evaluating host/hosts for placeholders. With this fix, IPv6 addresses are correctly parsed.
    • Bug Fix
    • Log Collection - Sprint 245
    • Customer Escalated

    Description

      This is a clone of LOG-4281 for fix in Logging v5.8.z

      Description of problem:

      Forwarding logs to external ipv6 elasticsearch instance fails when using fluentd as log collector. Collector pods stream below logs:

      2023-06-29 16:55:05 +0000 [warn]: [elasticsearch_secure] failed to flush the buffer. retry_times=5 next_retry_time=2023-06-29 16:55:35 +0000 chunk="5ff478dd6b35c5b24fccf3c705fc07e2" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (\{:host=>nil, :port=>nil, :scheme=>\"fd00\", :user=>\"elastic\", :password=>\"obfuscated\"}): no implicit conversion of nil into String"
      
        2023-06-29 16:55:05 +0000 [warn]: suppressed same stacktrace
      
      2023-06-29 16:55:35 +0000 [warn]: [elasticsearch_secure] failed to flush the buffer. retry_times=6 next_retry_time=2023-06-29 16:56:34 +0000 chunk="5ff478dd6b35c5b24fccf3c705fc07e2" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (\{:host=>nil, :port=>nil, :scheme=>\"fd00\", :user=>\"elastic\", :password=>\"obfuscated\"}): no implicit conversion of nil into String"
      
        2023-06-29 16:55:35 +0000 [warn]: suppressed same stacktrace
      
      

      Version-Release number of selected component (if applicable):

      All Red Hat OpeShift Logging Operator Versions

      How reproducible:

      100%

      Steps to Reproduce:

      1. Create a ipv6 RHOCP cluster (singlestack will be ok here)
      2. Install Red Hat OpenShift Logging Operator.
      3. Setup a virtual machine which has ipv6 and has elasticsearch up and running (setup this instance in the same network where RHOCP was running to reduce the infra complexity)
      4. Configure CLF on RHOCP cluster to send logs to this externally hosted elasticsearch instance. 

      Actual results:

      The logs are not forwarded to external elasticsearch instance and collector pods stream below error:

      2023-06-29 16:55:05 +0000 [warn]: [elasticsearch_secure] failed to flush the buffer. retry_times=5 next_retry_time=2023-06-29 16:55:35 +0000 chunk="5ff478dd6b35c5b24fccf3c705fc07e2" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (\{:host=>nil, :port=>nil, :scheme=>\"fd00\", :user=>\"elastic\", :password=>\"obfuscated\"}): no implicit conversion of nil into String"
      
      

      Expected results:

      The logs should flow normally to external elasticsearch instance.

      Additional info:

      Curl request from collector pods is completed successfully. See below:

      1. curl -u elastic --cacert /var/run/ocp-collector/secrets/es-secret/ca-bundle.crt [https://[IPv6]:|https://[ipv6]/]<Port>

      Enter host password for user 'elastic':

       

      {

        "name" : "node-1",

        "cluster_name" : "elasticsearch-test",

        "cluster_uuid" : "9MYXL3VBT9-k9rtY1nvxfw",

        "version" :

      {     "number" : "7.17.9",     "build_flavor" : "default",     "build_type" : "rpm",     "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",     "build_date" : "2023-01-31T05:34:43.305517834Z",     "build_snapshot" : false,     "lucene_version" : "8.11.1",     "minimum_wire_compatibility_version" : "6.8.0",     "minimum_index_compatibility_version" : "6.0.0-beta1"   }

      ,

        "tagline" : "You Know, for Search"

      }

       

      Attachments

        Issue Links

          Activity

            People

              rh-ee-calee Calvin Lee
              rhn-support-dgautam Dhruv Gautam
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: