Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4027

[release-5.7] emit transaction failed: error_class=ArgumentError error when sending logs to kafka

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this change when configuring clusterlogforward to write to a kafka output topic which matched a key in the payload, it would cause the messages to be dropped due to an error. This fixes that by prefixing fluent's buffer name with an underscore
      Show
      Before this change when configuring clusterlogforward to write to a kafka output topic which matched a key in the payload, it would cause the messages to be dropped due to an error. This fixes that by prefixing fluent's buffer name with an underscore
    • Bug Fix
    • Hide

      1) Deploy CLF instance

       

      apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance 
  namespace: openshift-logging 
spec:
  outputs:
  - kafka:
      brokers:
      - tls://server1.example.com:9093
      - tls://server2.example.com:9093
      topic: openshift
    name: kafka-nwapplog
    type: kafka
  pipelines:
  - inputRefs:
    - application
    name: application-to-remote
    outputRefs:
    - kafka-nwapplog

       

      2) Create a new project

       

      $ oc create namespace test

      3) Label the namespace

       

      $ oc label namespace test "app.test=test"
$ oc label namespace test "app=config-server"

       

      4) Deploy an application:

      $ oc new-app rails-postgresql-example

      5) Check collector logs allocated in the same node as the application was deployed:

      2022-12-28 08:00:24 +0000 [warn]: emit transaction failed: error_class=ArgumentError error="comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" location="/usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'" tag="kubernetes.var.log.pods.test_rails-postgresql-example-1-build_f833abef-c772-42d8-bb04-2016823ba47a.git-clone.0.log"
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `write'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1023:in `block in handle_stream_with_standard_format'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:931:in `write_guard'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1022:in `handle_stream_with_standard_format'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:925:in `execute_chunking'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:851:in `emit_buffered'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1089:in `block in handle_notify'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1125:in `with_io'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1058:in `handle_notify'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `block in on_notify'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `synchronize'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `on_notify'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:816:in `on_notify'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:401:in `block in setup_watcher'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:734:in `on_change'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-12-28 08:00:24 +0000 [warn]: emit transaction failed: error_class=ArgumentError error="comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" location="/usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'" tag="kubernetes.var.log.pods.test_rails-postgresql-example-1-build_f833abef-c772-42d8-bb04-2016823ba47a.git-clone.0.log"

       

       

       

       

       

       

       

       

      Show
      1) Deploy CLF instance   apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata:   name: instance    namespace: openshift-logging  spec:   outputs:   - kafka:       brokers:       - tls: //server1.example.com:9093       - tls: //server2.example.com:9093       topic: openshift     name: kafka-nwapplog     type: kafka   pipelines:   - inputRefs:     - application     name: application-to-remote     outputRefs:     - kafka-nwapplog   2) Create a new project   $ oc create namespace test 3) Label the namespace   $ oc label namespace test "app.test=test" $ oc label namespace test "app=config-server"   4) Deploy an application: $ oc new -app rails-postgresql-example 5) Check collector logs allocated in the same node as the application was deployed: 2022-12-28 08:00:24 +0000 [warn]: emit transaction failed: error_class=ArgumentError error= "comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" location= "/usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'" tag= "kubernetes. var .log.pods.test_rails-postgresql-example-1-build_f833abef-c772-42d8-bb04-2016823ba47a.git-clone.0.log"   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `write'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1023:in `block in handle_stream_with_standard_format'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:931:in `write_guard'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1022:in `handle_stream_with_standard_format'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:925:in `execute_chunking'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:851:in `emit_buffered'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_relabel.rb:29:in `process'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:839:in `emit_sync'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:196:in `emit_events'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event_router.rb:115:in `emit_stream'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1089:in `block in handle_notify'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1125:in `with_io'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1058:in `handle_notify'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `block in on_notify'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `synchronize'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1001:in `on_notify'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:816:in `on_notify'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:401:in `block in setup_watcher'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:734:in `on_change'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'   2022-12-28 08:00:24 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2022-12-28 08:00:24 +0000 [warn]: emit transaction failed: error_class=ArgumentError error= "comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" location= "/usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer.rb:340:in `sort'" tag= "kubernetes. var .log.pods.test_rails-postgresql-example-1-build_f833abef-c772-42d8-bb04-2016823ba47a.git-clone.0.log"                
    • Log Collection - Sprint 235, Log Collection - Sprint 236, Log Collection - Sprint 237
    • Moderate

      Description of problem:

      -RHOL Version 5.5.5

      "emit transaction failed: error_class=ArgumentError error="comparison of Fluent::Plugin::Buffer::Metadata with Fluent::Plugin::Buffer::Metadata failed" is shown in collector pods when trying to send application logs to an external kafka.

      I experience this error when namespaces have specific labels (maybe it could be related with--> https://issues.redhat.com/browse/LOG-3341)

       

              jcantril@redhat.com Jeffrey Cantrill
              acandelp Adrian Candel
              Libby Anderson Libby Anderson
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: