Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: Logging 5.2
Affects Version/s: Logging 5.0, Logging 5.1, openshift-4.6
Component/s: Log Collection
Labels:
- devel_ack+

Blocked:
False
Ready:
False
Docs QE Status:
NEW
QE Status:
NEW
Release Note Text:

Hide
Before this update, some log entries had unrecognized UTF-8 bytes, which caused Elasticsearch to reject the messages and block the entire buffered payload. With this update, rejected payloads drop the invalid log entries and resubmit the remaining entries to resolve the issue.

Show
Before this update, some log entries had unrecognized UTF-8 bytes, which caused Elasticsearch to reject the messages and block the entire buffered payload. With this update, rejected payloads drop the invalid log entries and resubmit the remaining entries to resolve the issue.

Sprint:
Logging (Core) - Sprint 203, Logging (Core) - Sprint 204

SFDC Cases Links:
SFDC Cases Counter:

Description

It's possible to see in the Elasticsearch error messages like:

~~~
021-04-20T07:10:45.091466957Z [2021-04-20T07:10:45,089][WARN ][r.suppressed ] [elasticsearch-cdm-j69zbfpm-1] path: /_bulk, params: {}
2021-04-20T07:10:45.091466957Z com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x92
2021-04-20T07:10:45.091466957Z at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@21cdf37b; line: 1, column: 81]
2021-04-20T07:10:45.091466957Z at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
~~~

Similar error is possible to read on the fluentd side:

~~~
fluentd-f29lv/fluentd/fluentd/logs/current.log:2021-04-19T18:20:29.886010127Z 2021-04-19 18:20:29 +0000 [warn]: [retry_clo_default_output_es] failed to flush the buffer. retry_time=1 next_retry_seconds=2021-04-19 18:20:30 +0000 chunk="5b83a8bc000a55be294f55337118ffa7" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [500] {\"error\":{\"root_cause\":[

{\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@204fe9c9; line: 1, column: 81]\"}

],\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92
n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@204fe9c9; line: 1, column: 81]\"},\"status\":500}"
fluentd-f29lv/fluentd/fluentd/logs/current.log:2021-04-19T18:20:35.296756674Z 2021-04-19 18:20:35 +0000 [warn]: [retry_clo_default_output_es] failed to flush the buffer. retry_time=0 next_retry_seconds=2021-04-19 18:20:36 +0000 chunk="5b83a8bc000a55be294f55337118ffa7" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [500] {\"error\":{\"root_cause\":[

{\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@4cfcdc0; line: 1, column: 81]\"}

],\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92
n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@4cfcdc0; line: 1, column: 81]\"},\"status\":500}"