-
Bug
-
Resolution: Done
-
Normal
-
Logging 5.9.0
-
False
-
None
-
False
-
NEW
-
OBSDA-549 - Reliability and performance tuning for log collection
-
NEW
-
Release Note Not Required
-
-
-
Log Collection - Sprint 251
-
Low
Description of problem:
Can not forward log to Elasticsarch( E8 or the dafault ES) using compression=snapy. collector pods raise errors
2024-03-14T15:02:22.227434Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Retrying after response. reason=500 Internal Server Error: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Unexpected character ((CTRL-CHAR, code 153)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"}],"type":"json_parse_exception","reason":"Unexpected character ((CTRL-CHAR, code 153)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"},"status":500} internal_log_rate_limit=true 2024-03-14T15:02:23.236606Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding. 2024-03-14T15:02:24.345724Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::service::health: Endpoint is unhealthy. endpoint=https://elasticsearch:9200 2024-03-14T15:02:32.415142Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 35 times. 2024-03-14T15:02:32.415164Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Retrying after response. reason=500 Internal Server Error: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Invalid UTF-8 start byte 0xb0\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 4]"}],"type":"json_parse_exception","reason":"Invalid UTF-8 start byte 0xb0\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 4]"},"status":500} internal_log_rate_limit=true 2024-03-14T15:02:32.507753Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding.
Elasticsearch raise error:
[2024-03-14T15:04:47,594][WARN ][r.suppressed ] [elasticsearch-cdm-4p7876oe-1] path: /_bulk, params: {timeout=2147483648s}
com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xb0
at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 4]
.....
Can not forward logs to Elasticsearch( E8 or the dafault ES) when compression=zstd or snappy
oc logs collector-92s8r Creating the directory used for persisting Vector state /var/lib/vector Starting Vector process... 2024-03-14T15:09:42.593458Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Retrying after response. reason=500 Internal Server Error: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Unexpected character ('(' (code 40)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"}],"type":"json_parse_exception","reason":"Unexpected character ('(' (code 40)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"},"status":500} internal_log_rate_limit=true 2024-03-14T15:09:43.597521Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding. 2024-03-14T15:09:44.900750Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::service::health: Endpoint is unhealthy. endpoint=https://elasticsearch:9200 2024-03-14T15:09:52.959217Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 35 times. 2024-03-14T15:09:52.959241Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Retrying after response. reason=500 Internal Server Error: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Unexpected character ('(' (code 40)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"}],"type":"json_parse_exception","reason":"Unexpected character ('(' (code 40)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"},"status":500} internal_log_rate_limit=true 2024-03-14T15:09:53.052940Z WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding.
Steps to Reproduce:
- ...