Loading...

Type: Bug
Resolution: Done-Errata
Priority: Critical
Fix Version/s: Logging 5.8.1
Affects Version/s: Logging 5.8.0
Component/s: Log Collection
Labels:
- devel_ack+
- regression

Blocked:
False
Blocked Reason:
None
Ready:
False
Docs QE Status:
NEW
QE Status:
NEW
Release Note Text:
Before this change, Fluentd collector pods were in a CrashLoopBackOff state due to binding of the prometheus server on IPv6 clusters. With this change, the collectors spin up properly on IPv6 clusters.
Release Note Type:
Bug Fix
Intelligence Requested:
Market:
RH Private Keywords:

Sprint:
Log Collection - Sprint 244, Log Collection - Sprint 245
Severity:
Moderate

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

On IPv6 single stack and dual stack (IPv4 and IPv6) clusters, fluentd pod [error]: unexpected error error_class=SocketError error="getaddrinfo: Name or service not known"

Version-Release number of selected component (if applicable):

4.14.0-0.nightly-2023-10-18-004928

How reproducible:
always

Steps to Reproduce:

Deploy EFK.

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: openshift-logging
spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    elasticsearch:
      nodeCount: 1
      resources:
        limits:
          memory: 2Gi
        requests:
          cpu: 200m
          memory: 2Gi
      storage: {}
      redundancyPolicy: "ZeroRedundancy"
  visualization:
    type: "kibana"
    kibana:
      replicas: 1
  collection:
    type: "fluentd"
    fluentd: {}

2. check the collector pods status

oc get pods
NAME                                            READY   STATUS             RESTARTS       AGE
cluster-logging-operator-7c865f66df-6xzhq       1/1     Running            0              122m
collector-2lz27                                 0/1     CrashLoopBackOff   3 (45s ago)    4m22s
collector-nhxmz                                 0/1     CrashLoopBackOff   4 (70s ago)    4m22s
collector-tgc8z                                 0/1     Error              4 (107s ago)   4m22s
collector-x9mmv                                 1/1     Running            4 (62s ago)    4m22s
collector-z92r6                                 0/1     CrashLoopBackOff   3 (35s ago)    4m22s




$ oc logs collector-tgc8z
POD_IPS: fd01:0:0:4::2d, PROM_BIND_IP: [::]
Setting each total_size_limit for 3 buffers to 23966594048 bytes
Setting queued_chunks_limit_size for each buffer to 2857
Setting chunk_limit_size for each buffer to 8388608
2023-10-23 03:45:50 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2023-10-23 03:45:50 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2023-10-23 03:46:37 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified
2023-10-23 03:46:37 +0000 [warn]: Webrick ignores given TLS version
2023-10-23 03:46:37 +0000 [error]: unexpected error error_class=SocketError error="getaddrinfo: Name or service not known"
  2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:227:in `getaddrinfo'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:227:in `foreach'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:763:in `tcp_server_sockets'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/utils.rb:60:in `create_listeners'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/ssl.rb:169:in `listen'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/server.rb:111:in `initialize'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/httpserver.rb:47:in `initialize'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server/compat/server.rb:47:in `new'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server/compat/server.rb:47:in `initialize'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:96:in `new'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:96:in `http_server_create_https_server'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:67:in `http_server_create_http_server'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluent-plugin-prometheus-2.0.3/lib/fluent/plugin/in_prometheus.rb:109:in `start'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:203:in `block in start'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:191:in `each'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:191:in `block in lifecycle'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `each'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `lifecycle'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:202:in `start'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:248:in `start'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in `block in run_worker'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in `main_process'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in `run_worker'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
  2023-10-23 03:46:37 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-10-23 03:46:37 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>'
  2023-10-23 03:46:37 +0000 [error]: /usr/bin/fluentd:25:in `load'
  2023-10-23 03:46:37 +0000 [error]: /usr/bin/fluentd:25:in `<main>'
2023-10-23 03:46:37 +0000 [error]: unexpected error error_class=SocketError error="getaddrinfo: Name or service not known"
  2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:227:in `getaddrinfo'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:227:in `foreach'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/ruby/socket.rb:763:in `tcp_server_sockets'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/utils.rb:60:in `create_listeners'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/ssl.rb:169:in `listen'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/server.rb:111:in `initialize'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/webrick-1.8.1/lib/webrick/httpserver.rb:47:in `initialize'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server/compat/server.rb:47:in `new'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server/compat/server.rb:47:in `initialize'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:96:in `new'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:96:in `http_server_create_https_server'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/plugin_helper/http_server.rb:67:in `http_server_create_http_server'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluent-plugin-prometheus-2.0.3/lib/fluent/plugin/in_prometheus.rb:109:in `start'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:203:in `block in start'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:191:in `each'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:191:in `block in lifecycle'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `each'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `lifecycle'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:202:in `start'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:248:in `start'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in `block in run_worker'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in `main_process'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in `run_worker'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
  2023-10-23 03:46:37 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-10-23 03:46:37 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-10-23 03:46:37 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>'
  2023-10-23 03:46:37 +0000 [error]: /usr/bin/fluentd:25:in `load'
  2023-10-23 03:46:37 +0000 [error]: /usr/bin/fluentd:25:in `<main>'