Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: Logging 5.6.6
Affects Version/s: Logging 5.5.5
Component/s: Log Collection
Labels:
- collector
- devel_ack+
- kafka
- logging

Blocked:
False
Blocked Reason:
None
Ready:
False
Docs QE Status:
NEW
QE Status:
VERIFIED
Release Note Text:

Hide
Before this change when configuring clusterlogforward to write to a kafka output topic which matched a key in the payload, it would cause the messages to be dropped due to an error. This fixes that by prefixing fluent's buffer name with an underscore

Show
Before this change when configuring clusterlogforward to write to a kafka output topic which matched a key in the payload, it would cause the messages to be dropped due to an error. This fixes that by prefixing fluent's buffer name with an underscore
Release Note Type:
Bug Fix
Steps to Reproduce:
Hide

1) Deploy CLF instance

apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - kafka: brokers: - tls://nwapplog1.euro.dreibanken.at:9093 - tls://nwapplog2.euro.dreibanken.at:9093 topic: openshift name: kafka-nwapplog type: kafka pipelines: - inputRefs: - application name: application-to-remote outputRefs: - kafka-nwapplog

2) Create a new project

$ oc create namespace test

3) Label the namespace

$ oc label namespace test "app.test=test" $ oc label namespace test "app=config-server"

4) Deploy an application:

$ oc new-app rails-postgresql-example

5) Check collector logs allocated in the same node as the application was deployed:

2022-12-28 08:00:24 +0000 [warn]: emit transaction failed: error_class=ArgumentError error="comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" location="/usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'" tag="kubernetes.var.log.pods.test_rails-postgresql-example-1-build_f833abef-c772-42d8-bb04-2016823ba47a.git-clone.0.log" 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `write' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1023:in `block in handle_stream_with_standard_format' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:931:in `write_guard' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1022:in `handle_stream_with_standard_format' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:925:in `execute_chunking' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:851:in `emit_buffered' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:615:in `receive_lines' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1089:in `block in handle_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1125:in `with_io' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1058:in `handle_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `block in on_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `synchronize' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `on_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:816:in `on_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:401:in `block in setup_watcher' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:734:in `on_change' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2022-12-28 08:00:24 +0000 [warn]: emit transaction failed: error_class=ArgumentError error="comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" location="/usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'" tag="kubernetes.var.log.pods.test_rails-postgresql-example-1-build_f833abef-c772-42d8-bb04-2016823ba47a.git-clone.0.log"
Show
1) Deploy CLF instance apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - kafka: brokers: - tls: //nwapplog1.euro.dreibanken.at:9093 - tls: //nwapplog2.euro.dreibanken.at:9093 topic: openshift name: kafka-nwapplog type: kafka pipelines: - inputRefs: - application name: application-to-remote outputRefs: - kafka-nwapplog 2) Create a new project $ oc create namespace test 3) Label the namespace $ oc label namespace test "app.test=test" $ oc label namespace test "app=config-server" 4) Deploy an application: $ oc new -app rails-postgresql-example 5) Check collector logs allocated in the same node as the application was deployed: 2022-12-28 08:00:24 +0000 [warn]: emit transaction failed: error_class=ArgumentError error= "comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" location= "/usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'" tag= "kubernetes. var .log.pods.test_rails-postgresql-example-1-build_f833abef-c772-42d8-bb04-2016823ba47a.git-clone.0.log" 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `write' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1023:in `block in handle_stream_with_standard_format' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:931:in `write_guard' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1022:in `handle_stream_with_standard_format' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:925:in `execute_chunking' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:851:in `emit_buffered' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:615:in `receive_lines' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1089:in `block in handle_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1125:in `with_io' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1058:in `handle_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `block in on_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `synchronize' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `on_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:816:in `on_notify' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:401:in `block in setup_watcher' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:734:in `on_change' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start' 2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2022-12-28 08:00:24 +0000 [warn]: emit transaction failed: error_class=ArgumentError error= "comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" location= "/usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'" tag= "kubernetes. var .log.pods.test_rails-postgresql-example-1-build_f833abef-c772-42d8-bb04-2016823ba47a.git-clone.0.log"
Intelligence Requested:
Market:

Sprint:
Log Collection - Sprint 233, Log Collection - Sprint 234, Log Collection - Sprint 235, Log Collection - Sprint 236
Severity:
Moderate

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

-RHOL Version 5.5.5

"emit transaction failed: error_class=ArgumentError error="comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" is shown in collector pods when trying to send application logs to an external kafka.

I experience this error when namespaces have specific labels (maybe it could be related with--> https://issues.redhat.com/browse/LOG-3341)