Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1499

[release-5.2] Error "com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x92" in Elasticsearch/Fluentd logs

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Urgent
    • Resolution: Done
    • OpenShift 4.6, Logging 5.0, Logging 5.1
    • Logging 5.2
    • Core
    • Logging (Core) - Sprint 203, Logging (Core) - Sprint 204
    • Passed
    • NEW
    • NEW
    • Hide
      Before this update, some log entries had unrecognized UTF-8 bytes, which caused Elasticsearch to reject the messages and block the entire buffered payload. With this update, rejected payloads drop the invalid log entries and resubmit the remaining entries to resolve the issue.
      Show
      Before this update, some log entries had unrecognized UTF-8 bytes, which caused Elasticsearch to reject the messages and block the entire buffered payload. With this update, rejected payloads drop the invalid log entries and resubmit the remaining entries to resolve the issue.

    Description

      It's possible to see in the Elasticsearch error messages like:

      ~~~
      021-04-20T07:10:45.091466957Z [2021-04-20T07:10:45,089][WARN ][r.suppressed ] [elasticsearch-cdm-j69zbfpm-1] path: /_bulk, params: {}
      2021-04-20T07:10:45.091466957Z com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x92
      2021-04-20T07:10:45.091466957Z at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@21cdf37b; line: 1, column: 81]
      2021-04-20T07:10:45.091466957Z at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
      ~~~

      Similar error is possible to read on the fluentd side:

      ~~~
      fluentd-f29lv/fluentd/fluentd/logs/current.log:2021-04-19T18:20:29.886010127Z 2021-04-19 18:20:29 +0000 [warn]: [retry_clo_default_output_es] failed to flush the buffer. retry_time=1 next_retry_seconds=2021-04-19 18:20:30 +0000 chunk="5b83a8bc000a55be294f55337118ffa7" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [500] {\"error\":{\"root_cause\":[

      {\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@204fe9c9; line: 1, column: 81]\"}

      ],\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92
      n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@204fe9c9; line: 1, column: 81]\"},\"status\":500}"
      fluentd-f29lv/fluentd/fluentd/logs/current.log:2021-04-19T18:20:35.296756674Z 2021-04-19 18:20:35 +0000 [warn]: [retry_clo_default_output_es] failed to flush the buffer. retry_time=0 next_retry_seconds=2021-04-19 18:20:36 +0000 chunk="5b83a8bc000a55be294f55337118ffa7" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [500] {\"error\":{\"root_cause\":[

      {\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@4cfcdc0; line: 1, column: 81]\"}

      ],\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92
      n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@4cfcdc0; line: 1, column: 81]\"},\"status\":500}"

      Attachments

        Activity

          People

            jcantril@redhat.com Jeffrey Cantrill
            jcantril@redhat.com Jeffrey Cantrill
            Qiaoling Tang Qiaoling Tang
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: