Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4706

Fluentd CrashLoopBackOff on IPv6 single stack and dual stack clusters

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • NEW
    • Before this change, Fluentd collector pods were in a CrashLoopBackOff state due to binding of the prometheus server on IPv6 clusters. With this change, the collectors spin up properly on IPv6 clusters.
    • Bug Fix
    • Log Collection - Sprint 244, Log Collection - Sprint 245

    Description

      Description of problem:

      On IPv6 single stack and dual stack (IPv4 and IPv6) clusters, fluentd pod [error]: unexpected error error_class=SocketError error="getaddrinfo: Name or service not known"

      Version-Release number of selected component (if applicable):

      4.14.0-0.nightly-2023-10-18-004928

      How reproducible:
      always

      Steps to Reproduce:

      1.  Deploy EFK.
        apiVersion: "logging.openshift.io/v1"
        kind: "ClusterLogging"
        metadata:
          name: "instance"
          namespace: openshift-logging
        spec:
          managementState: "Managed"
          logStore:
            type: "elasticsearch"
            elasticsearch:
              nodeCount: 1
              resources:
                limits:
                  memory: 2Gi
                requests:
                  cpu: 200m
                  memory: 2Gi
              storage: {}
              redundancyPolicy: "ZeroRedundancy"
          visualization:
            type: "kibana"
            kibana:
              replicas: 1
          collection:
            type: "fluentd"
            fluentd: {}
        

      2. check the collector pods status 

       

      oc get pods
      NAME                                            READY   STATUS             RESTARTS       AGE
      cluster-logging-operator-7c865f66df-6xzhq       1/1     Running            0              122m
      collector-2lz27                                 0/1     CrashLoopBackOff   3 (45s ago)    4m22s
      collector-nhxmz                                 0/1     CrashLoopBackOff   4 (70s ago)    4m22s
      collector-tgc8z                                 0/1     Error              4 (107s ago)   4m22s
      collector-x9mmv                                 1/1     Running            4 (62s ago)    4m22s
      collector-z92r6                                 0/1     CrashLoopBackOff   3 (35s ago)    4m22s
      
      
      
      
      $ oc logs collector-tgc8z
      POD_IPS: fd01:0:0:4::2d, PROM_BIND_IP: [::]
      Setting each total_size_limit for 3 buffers to 23966594048 bytes
      Setting queued_chunks_limit_size for each buffer to 2857
      Setting chunk_limit_size for each buffer to 8388608
      2023-10-23 03:45:50 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
      2023-10-23 03:45:50 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
      2023-10-23 03:46:37 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified
      2023-10-23 03:46:37 +0000 [warn]: Webrick ignores given TLS version
      2023-10-23 03:46:37 +0000 [error]: unexpected error error_class=SocketError error="getaddrinfo: Name or service not known"
        2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:227:in `getaddrinfo'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:227:in `foreach'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:763:in `tcp_server_sockets'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/utils.rb:60:in `create_listeners'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/ssl.rb:169:in `listen'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/server.rb:111:in `initialize'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/httpserver.rb:47:in `initialize'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server/compat/server.rb:47:in `new'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server/compat/server.rb:47:in `initialize'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:96:in `new'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:96:in `http_server_create_https_server'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:67:in `http_server_create_http_server'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluent-plugin-prometheus-2.0.3/lib/fluent/plugin/in_prometheus.rb:109:in `start'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:203:in `block in start'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:191:in `each'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:191:in `block in lifecycle'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `each'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `lifecycle'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:202:in `start'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:248:in `start'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in `block in run_worker'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in `main_process'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in `run_worker'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
        2023-10-23 03:46:37 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        2023-10-23 03:46:37 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>'
        2023-10-23 03:46:37 +0000 [error]: /usr/bin/fluentd:25:in `load'
        2023-10-23 03:46:37 +0000 [error]: /usr/bin/fluentd:25:in `<main>'
      2023-10-23 03:46:37 +0000 [error]: unexpected error error_class=SocketError error="getaddrinfo: Name or service not known"
        2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:227:in `getaddrinfo'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:227:in `foreach'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:763:in `tcp_server_sockets'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/utils.rb:60:in `create_listeners'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/ssl.rb:169:in `listen'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/server.rb:111:in `initialize'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/httpserver.rb:47:in `initialize'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server/compat/server.rb:47:in `new'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server/compat/server.rb:47:in `initialize'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:96:in `new'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:96:in `http_server_create_https_server'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:67:in `http_server_create_http_server'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluent-plugin-prometheus-2.0.3/lib/fluent/plugin/in_prometheus.rb:109:in `start'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:203:in `block in start'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:191:in `each'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:191:in `block in lifecycle'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `each'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `lifecycle'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:202:in `start'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:248:in `start'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in `block in run_worker'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in `main_process'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in `run_worker'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
        2023-10-23 03:46:37 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        2023-10-23 03:46:37 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>'
        2023-10-23 03:46:37 +0000 [error]: /usr/bin/fluentd:25:in `load'
        2023-10-23 03:46:37 +0000 [error]: /usr/bin/fluentd:25:in `<main>'
      

       

      Actual results:

      Expected results:

      Additional info:

      Attachments

        Activity

          People

            rh-ee-calee Calvin Lee
            anli@redhat.com Anping Li
            Anping Li Anping Li
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: