Loading...

Type: Bug
Resolution: Done-Errata
Priority: Major
Fix Version/s: Logging 6.0.2
Affects Version/s: Logging 5.8.z, Logging 5.9.z
Component/s: Log Collection
Labels:
- NeedTestCases
- devel_ack+

Blocked:
False
Blocked Reason:
None
Ready:
False
Docs QE Status:
NEW
QE Status:
NEW
Release Note Text:
Before this change the collector would discard audit log messages that exceeded the configured threshold. This modifies the audit configuration thresholds for the maximum line size as well as the number of bytes read during a read cycle.
Release Note Type:
Bug Fix
Intelligence Requested:
Market:

Sprint:
Log Collection - Sprint 261, Log Collection - Sprint 262
Severity:
Moderate

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

In the Vector collector is observed line errors where rejected to be read as exceeding the `max_line_size`:

WARN file_source::buffer: Internal log [Found line that exceeds max_line_bytes; discarding.] is being suppressed to avoid flooding.
WARN file_source::buffer: Internal log [Found line that exceeds max_line_bytes; discarding.] has been suppressed 1 times.
WARN file_source::buffer: Found line that exceeds max_line_bytes; discarding. Internal_log_rate_limit=true

It should be expected:

that the log line throwing the error in Vector indicating the source file from where this line exceeding the `max_line_bytes` is coming from
2. It could be also desired an alert that highlight this

Version-Release number of selected component (if applicable):

Vector collector
All Logging versions from 5 where Vector GA

How reproducible:

Always

Steps to Reproduce:

Enable Audit logs with AllRequestBodies as detailed in https://docs.openshift.com/container-platform/4.17/security/audit-log-policy-config.html#about-audit-log-profiles_audit-log-policy-config
Enable to collect the audit logs for the collector
Review the logs from the collector checking errors with max_line_bytes
Review for example the logs from openshift-apiserver, for downloading
Run an script as below against the openshift-apiserver.log redirecting the output to a file. In this file, it will be the lines exceeding the max_line_bytes of 102400

$ cat /tmp/find_lines.sh
#!/bin/bash
file="/tmp/openshift-apiserver.log"
while read line
do
        a=$(echo -n "$line" | wc -c)
        if [ "$a" -gt 102400 ]; then
          echo "${line}"
        fi
done <"$file"

$ /bin/bash /tmp/find_lines.sh > /tmp/openshift-apiserver_lines_exceeding

Actual results:

The error from Vector is as below where not possible to identify the source file from where the lines are discarded. The only visible is the log_type:

$ for pod in $(oc get pods -l component=collector -o name); do oc logs $pod -c collector ; done |grep -c max_line_bytes
608
 $ for pod in $(oc get pods -l component=collector -o name); do oc logs $pod -c collector ; done |grep  max_line_bytes |head -1
2024-10-22T20:27:39.508209Z  WARN file_source::buffer: Found line that exceeds max_line_bytes; discarding. internal_log_rate_limit=true

// An entire error log
2024-10-22T20:40:20.402407Z  WARN sink{component_kind="sink" component_id=output_default_loki_audit component_type=loki}: vector::sinks::util::retries: Internal log [Retrying after error.] is being suppressed to avoid flooding.
2024-10-22T20:40:30.096911Z  WARN file_source::buffer: Internal log [Found line that exceeds max_line_bytes; discarding.] has been suppressed 273 times.

// Bytes of the first and last line exceeding the max_line_bytes

$ head -1 /tmp/lines |wc -c
248090
$ tail  -1 /tmp/lines |wc -c
1139398

// A couple of lines exceeding the size is when listing the `images` and the `imagestreams`

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"3fa54595-ce18-4323-ac32-2c2c236673ea","stage":"ResponseComplete","requestURI":"/apis/image.openshift.io/v1/images?limit=500u0026resourceVersion=0","verb":"list","user":{"username":"system:apiserver","uid":"fa915fa4-1481-4d47-8b66-b174cbb89fcb","groups"...

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"b54ed4cd-56a7-4021-a521-1d6e
a0010047","stage":"ResponseComplete","requestURI":"/apis/image.openshift.io/v1/imagestreams?limit=500u0026resour
ceVersion=0","verb":"list","user":{"username":"system:apiserver","uid":"fa915fa4-1481-4d47-8b66-b174cbb89fcb","g
roups":["system:masters"]},"sourceIPs":["::1"],"userAgent":"openshift-apiserver/v0.0.0 (linux/amd64) kubernetes/
$Format","objectRef":{"resource":"imagestreams","apiGroup":"image.openshift.io","apiVersion":"v1"},"responseStat