-
Bug
-
Resolution: Done
-
Normal
-
Logging 5.9.0
-
False
-
None
-
False
-
NEW
-
OBSDA-549 - Reliability and performance tuning for log collection
-
NEW
-
Release Note Not Required
-
-
-
Log Collection - Sprint 251
-
Informational
Description of problem:
When compression is zlib or zstd. the logs can not be be sent to lokistack.
$ oc logs collector-xx6bh Creating the directory used for persisting Vector state /var/lib/vector Starting Vector process... 2024-03-08T01:39:31.590056Z ERROR sink{component_kind="sink" component_id=output_default_mylogsqa component_type=loki}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Server responded with an error: 400 Bad Request internal_log_rate_limit=true 2024-03-08T01:39:31.590091Z ERROR sink{component_kind="sink" component_id=output_default_mylogsqa component_type=loki}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(ServerError { code: 400 }) request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true 2024-03-08T01:39:31.590107Z ERROR sink{component_kind="sink" component_id=output_default_mylogsqa component_type=loki}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=18 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true 2024-03-08T01:39:32.545786Z ERROR sink{component_kind="sink" component_id=output_default_mylogsqa component_type=loki}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being suppressed to avoid flooding. 2024-03-08T01:39:32.545817Z ERROR sink{component_kind="sink" component_id=output_default_mylogsqa component_type=loki}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.
How reproducible:
Always
Steps to Reproduce:
- Forward logs to lokistack using compression='zstd' or compression='zlib'
#CLF to lokistack cat <<EOF|oc create -f - apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: inputs: - application: namespaces: - project-qa-1 name: myLogsQA outputs: - name: default tuning: compression: 'zlib' type: loki pipelines: - inputRefs: - myLogsQA name: app-to-default outputRefs: - default EOF
- ...
