Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: Logging 5.8.5
Affects Version/s: Logging 5.8.0
Component/s: Log Collection
Labels:
- RHEL9
- devel_ack+
- fluentd
- regression

Blocked:
False
Blocked Reason:
None
Ready:
False
Docs QE Status:
NEW
QE Status:
NEW
Release Note Text:
Before this change, the collector was unable to read private certificate keys on FIPS enabled clusters. This change updates the OpenSSL gem to allow reading private certificates
Release Note Type:
Bug Fix
Intelligence Requested:
Market:

Sprint:
Log Collection - Sprint 234, Log Collection - Sprint 235, Log Collection - Sprint 244, Log Collection - Sprint 245, Log Collection - Sprint 246, Log Collection - Sprint 247, Log Collection - Sprint 250, Log Collection - Sprint 251

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

Deploy logging 5.6.5 on a cluster which has enabled FIPS, then check pods' status. When using fluentd as the collector, collector pods can't start and stuck in CrashLoopBackOff status:

$ oc get pod
NAME                                            READY   STATUS             RESTARTS        AGE
cluster-logging-operator-75f94b5648-tr9x9       1/1     Running            0               16m
collector-4rt5r                                 1/2     CrashLoopBackOff   6 (4m29s ago)   14m
collector-g7864                                 1/2     CrashLoopBackOff   7 (2m56s ago)   14m
collector-kdbxr                                 1/2     CrashLoopBackOff   7 (29s ago)     14m
collector-wkgfl                                 1/2     CrashLoopBackOff   7 (2m48s ago)   14m
collector-wtq6x                                 2/2     Running            7 (8m2s ago)    14m
collector-x8k8q                                 1/2     CrashLoopBackOff   7 (2m59s ago)   14m

And raise below errors:

2023-04-07 07:35:06 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified
2023-04-07 07:35:06 +0000 [error]: unexpected error error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey"
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/cert_option.rb:89:in `read'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/cert_option.rb:89:in `cert_option_load'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/cert_option.rb:65:in `cert_option_server_validate!'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/cert_option.rb:27:in `cert_option_create_context'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/http_server/ssl_context_builder.rb:32:in `build'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/http_server.rb:94:in `http_server_create_https_server'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/http_server.rb:67:in `http_server_create_http_server'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-prometheus-2.0.3/lib/fluent/plugin/in_prometheus.rb:109:in `start'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:203:in `block in start'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `each'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `block in lifecycle'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `each'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `lifecycle'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:202:in `start'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/engine.rb:248:in `start'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/engine.rb:147:in `run'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:720:in `block in run_worker'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in `main_process'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in `run_worker'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in `<top (required)>'
  2023-04-07 07:35:06 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-04-07 07:35:06 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/bin/fluentd:25:in `load'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/bin/fluentd:25:in `<main>'
2023-04-07 07:35:06 +0000 [error]: unexpected error error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey"
  2023-04-07 07:35:06 +0000 [error]: suppressed same stacktrace