Loading...

Type: Bug
Resolution: Cannot Reproduce
Priority: Normal
Fix Version/s: Logging 5.2.z, Logging 5.5.0
Affects Version/s: Logging 5.2
Component/s: Log Collection
Labels:
- devel_ack+

Blocked:
False
Ready:
False
Docs QE Status:
NEW
QE Status:
VERIFIED
Release Note Text:
undefined
Market:

Sprint:
Logging (Core) - Sprint 211, Logging (Core) - Sprint 212, Logging (Core) - Sprint 213, Logging (Core) - Sprint 214

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

Launch a cluster with FIPS enabled, deploy fluentd server following https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-43250 , forward logs to fluentd server with mTLS and enable tls_client_private_key_passphrase, check the logging fluentd pod logs, there are lots of errors:

2021-08-25 02:07:50 +0000 [warn]: failed to flush the buffer. retry_time=0 next_retry_seconds=2021-08-25 02:07:51 +0000 chunk="5ca58659f3ce4f21ca1e31cd865bdd91" error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey: no start line"
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/socket.rb:155:in `read'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/socket.rb:155:in `socket_create_tls'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:352:in `create_transfer_socket'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/connection_manager.rb:86:in `call'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/connection_manager.rb:86:in `block in connect_keepalive'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/socket_cache.rb:40:in `block in checkout_or'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/socket_cache.rb:34:in `synchronize'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/socket_cache.rb:34:in `checkout_or'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/connection_manager.rb:85:in `connect_keepalive'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/connection_manager.rb:42:in `connect'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:732:in `connect'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:606:in `send_data'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:336:in `block in write'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/load_balancer.rb:46:in `block in select_healthy_node'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/load_balancer.rb:37:in `times'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/load_balancer.rb:37:in `select_healthy_node'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:336:in `write'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1125:in `try_flush'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1431:in `flush_thread_run'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2021-08-25 02:07:50 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-08-25 02:07:51 +0000 [warn]: failed to flush the buffer. retry_time=1 next_retry_seconds=2021-08-25 02:07:52 +0000 chunk="5ca58659f3ce4f21ca1e31cd865bdd91" error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey: no start line"
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/socket.rb:155:in `read'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/socket.rb:155:in `socket_create_tls'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:352:in `create_transfer_socket'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/connection_manager.rb:86:in `call'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/connection_manager.rb:86:in `block in connect_keepalive'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/socket_cache.rb:40:in `block in checkout_or'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/socket_cache.rb:34:in `synchronize'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/socket_cache.rb:34:in `checkout_or'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/connection_manager.rb:85:in `connect_keepalive'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/connection_manager.rb:42:in `connect'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:732:in `connect'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:606:in `send_data'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:336:in `block in write'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/load_balancer.rb:46:in `block in select_healthy_node'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/load_balancer.rb:37:in `times'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward/load_balancer.rb:37:in `select_healthy_node'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:336:in `write'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1125:in `try_flush'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1431:in `flush_thread_run'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2021-08-25 02:07:51 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-08-25 02:07:51 +0000 [warn]: failed to flush the buffer. retry_time=2 next_retry_seconds=2021-08-25 02:07:54 +0000 chunk="5ca5865df79543025731bf2407ee6698" error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey: no start line"
  2021-08-25 02:07:51 +0000 [warn]: suppressed same stacktrace
2021-08-25 02:07:54 +0000 [warn]: failed to flush the buffer. retry_time=3 next_retry_seconds=2021-08-25 02:07:57 +0000 chunk="5ca5865df79543025731bf2407ee6698" error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey: no start line"
  2021-08-25 02:07:54 +0000 [warn]: suppressed same stacktrace
2021-08-25 02:07:54 +0000 [warn]: failed to flush the buffer. retry_time=4 next_retry_seconds=2021-08-25 02:08:01 +0000 chunk="5ca58659f3ce4f21ca1e31cd865bdd91" error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey: no start line"
2021-08-25 02:07:56 +0000 [error]: Timer detached. title=:in_collected_tail_monitor
2021-08-25 02:08:01 +0000 [warn]: failed to flush the buffer. retry_time=5 next_retry_seconds=2021-08-25 02:08:16 +0000 chunk="5ca58659f3ce4f21ca1e31cd865bdd91" error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey: no start line"
  2021-08-25 02:08:01 +0000 [warn]: suppressed same stacktrace
2021-08-25 02:08:01 +0000 [warn]: failed to flush the buffer. retry_time=6 next_retry_seconds=2021-08-25 02:08:34 +0000 chunk="5ca5865df79543025731bf2407ee6698" error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey: no start line"
  2021-08-25 02:08:01 +0000 [warn]: suppressed same stacktrace

logs can't be forwarded to fluentd server

Version-Release number of selected component (if applicable):

cluster-logging.5.2.0-43

How reproducible:

Always

Steps to Reproduce:
refer to https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-43250

Actual results:

Expected results:

Additional info:

no such issue when test forward to fluentd server with mTLS but don't enable tls_client_private_key_passphrase in the same cluster

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates