Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5246

Can not forward logs to elasticsearch using snappy and zstd

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • OBSDA-549 - Reliability and performance tuning for log collection
    • NEW
    • Release Note Not Required
    • Log Collection - Sprint 251

      Description of problem:

      Can not forward log to Elasticsarch( E8 or the dafault ES) using compression=snapy. collector pods raise errors

      2024-03-14T15:02:22.227434Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Retrying after response. reason=500 Internal Server Error: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Unexpected character ((CTRL-CHAR, code 153)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"}],"type":"json_parse_exception","reason":"Unexpected character ((CTRL-CHAR, code 153)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"},"status":500} internal_log_rate_limit=true
      2024-03-14T15:02:23.236606Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding.
      2024-03-14T15:02:24.345724Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::service::health: Endpoint is unhealthy. endpoint=https://elasticsearch:9200
      2024-03-14T15:02:32.415142Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 35 times.
      2024-03-14T15:02:32.415164Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Retrying after response. reason=500 Internal Server Error: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Invalid UTF-8 start byte 0xb0\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 4]"}],"type":"json_parse_exception","reason":"Invalid UTF-8 start byte 0xb0\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 4]"},"status":500} internal_log_rate_limit=true
      2024-03-14T15:02:32.507753Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding.
      

      Elasticsearch raise error:

      [2024-03-14T15:04:47,594][WARN ][r.suppressed             ] [elasticsearch-cdm-4p7876oe-1] path: /_bulk, params: {timeout=2147483648s}
      com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xb0
       at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 4]
      .....
      

      Can not forward logs to Elasticsearch( E8 or the dafault ES) when compression=zstd or snappy

      oc logs collector-92s8r
      Creating the directory used for persisting Vector state /var/lib/vector
      Starting Vector process...
      2024-03-14T15:09:42.593458Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Retrying after response. reason=500 Internal Server Error: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Unexpected character ('(' (code 40)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"}],"type":"json_parse_exception","reason":"Unexpected character ('(' (code 40)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"},"status":500} internal_log_rate_limit=true
      2024-03-14T15:09:43.597521Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding.
      2024-03-14T15:09:44.900750Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::service::health: Endpoint is unhealthy. endpoint=https://elasticsearch:9200
      2024-03-14T15:09:52.959217Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 35 times.
      2024-03-14T15:09:52.959241Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Retrying after response. reason=500 Internal Server Error: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Unexpected character ('(' (code 40)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"}],"type":"json_parse_exception","reason":"Unexpected character ('(' (code 40)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.elasticsearch.transport.netty4.ByteBufStreamInput); line: 1, column: 2]"},"status":500} internal_log_rate_limit=true
      2024-03-14T15:09:53.052940Z  WARN sink{component_kind="sink" component_id=output_default component_type=elasticsearch}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding.
      
      

      Steps to Reproduce:

      1. ...

      Actual results:

      Expected results:

      Additional info:

            rh-ee-calee Calvin Lee
            rhn-support-anli Anping Li
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: