[release-5.2] Fluentd logs emit transaction failed: error_class=NoMethodError while forwarding to external syslog server

Description of problem:
when forwarding logs to external syslog using clusterlogforwarder, seeing the emit transaction failed error in fluentd pods

Version-Release number of selected component (if applicable):
Cluster version is 4.6
cluster-logging - 4.6

Errors messages we see in the pod

~~~

2021-06-25T09:11:53.889199910+09:00 2021-06-25 09:11:53 +0900 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `strip' for #<Hash:0x00007f6080042d00>" location="/etc/fluent/plugin/filter_parse_json_field.rb:83:in `block in do_replace_json_log'" tag="kubernetes.var.log.containers.analytics-client-5b6df86bc6-rd4fr_apic_client-922f752cd575c1b518866672111523acdf61f55447238e0aaf74e880a365e611.log"
2021-06-25T09:11:53.889232126+09:00 2021-06-25 09:11:53 +0900 [warn]: suppressed same stacktrace

~~~

 

We checked the error and we can see the file /etc/fluent/plugin/filter_parse_json_field.rb in fluentd pod.

 

Tried to understand the file it looks like the error is basically pointing to the function "do_replace_json_log" from the file. Can you please take a look on the code and help us to understand that why the customer is getting this error.

 

FYI:- Below code is from the internal cluster.

~~~
module Fluent::Plugin
  class ParseJSONFieldFilter < Fluent::Plugin::Filter    Fluent::Plugin.register_filter('parse_json_field', self)    config_param :merge_json_log, :bool, default: true
    config_param :preserve_json_log, :bool, default: true
    config_param :replace_json_log, :bool, default: false
    config_param :json_fields, :array, default: ['MESSAGE', 'log']    def initialize
      super
    end    def configure(conf)
      super
    end    def filter_stream(tag, es)
      return es unless @merge_json_log || @replace_json_log
      new_es = Fluent::MultiEventStream.new
      if @merge_json_log
        es.each

{ |time, record|           record = do_merge_json_log(record)           new_es.add(time, record)         }

      elsif @replace_json_log
        es.each

{ |time, record|           record = do_replace_json_log(record)           new_es.add(time, record)         }

      end
      new_es    end    def do_merge_json_log(record)
      json_fields.each do |merge_json_log_key|
        if record.has_key?(merge_json_log_key)
          value = (record[merge_json_log_key] || "").strip
          if value.start_with?('{') && value.end_with?('}')
            begin
              record = JSON.parse(value).merge(record)
              unless @preserve_json_log
                record.delete(merge_json_log_key)
              end
            rescue JSON::ParserError
              log.debug "parse_json_field could not parse field {merge_json_log_key} as JSON: value {value}"
            end
          end
          break
        end
      end
      record
    end    def do_replace_json_log(record)
      json_fields.each do |merge_json_log_key|
        if record.has_key?(merge_json_log_key)
          value = (record[merge_json_log_key] || "").strip
          if value.start_with?('{') && value.end_with?('}')
            begin
              parsed_value = JSON.parse(value)
              record[merge_json_log_key] = parsed_value
            rescue JSON::ParserError
              log.debug "parse_json_field could not parse field {merge_json_log_key} as JSON: value {value}"
            end
          end
          break
        end
      end
      record
    end  end
end
~~~

Cluster log forwarding CRD

~~~

spec:
outputs:

• name: rsyslog-X
  syslog:
  appName: ocp
  facility: local0
  procID: fluentd
  rfc: RFC5424
  severity: informational
  type: syslog
  url: tcp://IP:PORT
• name: rsyslog-X
  syslog:
  appName: ocp
  facility: local0
  procID: fluentd
  rfc: RFC5424
  severity: informational
  type: syslog
  url: tcp://IP:PORT
• name: remote-fluentd-X
  type: fluentdForward
  url: tcp://IP:PORT
• name: remote-fluentd-X
  type: fluentdForward
  url: tcp://IP:PORT
  pipelines:
• inputRefs:
• application
  labels:
  syslog: ocp
  name: syslog-X
  outputRefs:
• rsyslog-X
• rsyslog-X
• inputRefs:
• audit
  name: td-agent-audit
  outputRefs:
• remote-fluentd-X
• remote-fluentd-X

~~~

Please let us know in case of any additional information required.