Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1499

[release-5.2] Error "com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x92" in Elasticsearch/Fluentd logs

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • Logging 5.2
    • Logging 5.0, Logging 5.1, openshift-4.6
    • Log Collection
    • False
    • False
    • NEW
    • NEW
    • Hide
      Before this update, some log entries had unrecognized UTF-8 bytes, which caused Elasticsearch to reject the messages and block the entire buffered payload. With this update, rejected payloads drop the invalid log entries and resubmit the remaining entries to resolve the issue.
      Show
      Before this update, some log entries had unrecognized UTF-8 bytes, which caused Elasticsearch to reject the messages and block the entire buffered payload. With this update, rejected payloads drop the invalid log entries and resubmit the remaining entries to resolve the issue.
    • Logging (Core) - Sprint 203, Logging (Core) - Sprint 204

      It's possible to see in the Elasticsearch error messages like:

      ~~~
      021-04-20T07:10:45.091466957Z [2021-04-20T07:10:45,089][WARN ][r.suppressed ] [elasticsearch-cdm-j69zbfpm-1] path: /_bulk, params: {}
      2021-04-20T07:10:45.091466957Z com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x92
      2021-04-20T07:10:45.091466957Z at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@21cdf37b; line: 1, column: 81]
      2021-04-20T07:10:45.091466957Z at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
      ~~~

      Similar error is possible to read on the fluentd side:

      ~~~
      fluentd-f29lv/fluentd/fluentd/logs/current.log:2021-04-19T18:20:29.886010127Z 2021-04-19 18:20:29 +0000 [warn]: [retry_clo_default_output_es] failed to flush the buffer. retry_time=1 next_retry_seconds=2021-04-19 18:20:30 +0000 chunk="5b83a8bc000a55be294f55337118ffa7" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [500] {\"error\":{\"root_cause\":[

      {\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@204fe9c9; line: 1, column: 81]\"}

      ],\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92
      n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@204fe9c9; line: 1, column: 81]\"},\"status\":500}"
      fluentd-f29lv/fluentd/fluentd/logs/current.log:2021-04-19T18:20:35.296756674Z 2021-04-19 18:20:35 +0000 [warn]: [retry_clo_default_output_es] failed to flush the buffer. retry_time=0 next_retry_seconds=2021-04-19 18:20:36 +0000 chunk="5b83a8bc000a55be294f55337118ffa7" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [500] {\"error\":{\"root_cause\":[

      {\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@4cfcdc0; line: 1, column: 81]\"}

      ],\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92
      n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@4cfcdc0; line: 1, column: 81]\"},\"status\":500}"

              jcantril@redhat.com Jeffrey Cantrill
              jcantril@redhat.com Jeffrey Cantrill
              Qiaoling Tang Qiaoling Tang
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: