Loading...

Type: Bug
Resolution: Done-Errata
Priority: Normal
Fix Version/s: Logging 5.9.2
Affects Version/s: Logging 5.9.0
Component/s: Log Collection
Labels:

Blocked:
False
Blocked Reason:
None
Ready:
False
Docs QE Status:
NEW
QE Status:
NEW
Release Note Text:
With certain filtering enabled, rotated infrastructure logs were being sent to the app index. This change ensures rotated log files are correctly excluded.
Release Note Type:
Bug Fix
Intelligence Requested:
Market:

Sprint:
Log Collection - Sprint 250, Log Collection - Sprint 252, Log Collection - Sprint 253
Severity:
Important

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

Vector rules for infrastructure collects log files ending in *.log. However, as documented in https://access.redhat.com/solutions/4924281, crio rotates logs with the configuration:

the rotation is being done by default at ~ 10M (containerLogMaxSize) maintaining 5 files (containerLogMaxFiles)

When a log file is rotated, it crio appends the rotation date, and now it does not match the infrastructure but the application tenant

Version-Release number of selected component (if applicable):

5.9.0

How reproducible:

Always

Steps to Reproduce:

Go to the console
Select the application tenant in the filter
Use a LogQl like "{ log_type="application", kubernetes_namespace_name=~"openshift-.*" } | json". In any cluster running for some time it would show up records like the one I show in additional info.

Actual results:

Log collected to infrastructure and application several times.

Expected results:

Only one record collected exactly to infrastructure.

Additional info:

{"@timestamp":"2024-02-29T03:41:15.873213094Z","{*}file{*}":"{color:#FF0000}/var/log/pods/openshift-etcd_etcd-dlms2.sa-iberia.lab.eng.brq2.redhat.com_8c874e477ac9490f2308da857a0dfae4/etcd/3.log.{*}20240229-043220{*}{color}","hostname":"dlms2.sa-iberia.lab.eng.brq2.redhat.com","kubernetes":\{"annotations":{"kubectl.kubernetes.io/default-container":"etcd","kubernetes.io/config.hash":"8c874e477ac9490f2308da857a0dfae4","kubernetes.io/config.mirror":"8c874e477ac9490f2308da857a0dfae4","kubernetes.io/config.seen":"2024-02-28T22:27:53.882006079Z","kubernetes.io/config.source":"file","target.workload.openshift.io/management":"{\"effect\": \"PreferredDuringScheduling\"}"},"container_id":"cri-o://e259a052724b4920a9942388ed39327c08691bf875e48919feb52fd350572e71","container_image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c74c57f91f0f7ed26bb62f58c7b84c55750e51947fd6cc5711fa18f30b9f68c","container_image_id":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c74c57f91f0f7ed26bb62f58c7b84c55750e51947fd6cc5711fa18f30b9f68c","container_name":"etcd","labels":\{"app":"etcd","etcd":"true","k8s-app":"etcd","revision":"11"},"namespace_id":"78f1b483-db90-47a5-a924-0e63da75da89","namespace_labels":\{"kubernetes_io_metadata_name":"openshift-etcd","openshift_io_run-level":"0","pod-security_kubernetes_io_audit":"privileged","pod-security_kubernetes_io_enforce":"privileged","pod-security_kubernetes_io_warn":"privileged"},"{*}namespace_name{*}":"{*}{color:#FF0000}openshift-etcd{color}{*}","pod_id":"afa0e018-6975-43fd-b16d-d4d250d08faa","pod_ip":"10.37.233.4","pod_name":"etcd-dlms2.sa-iberia.lab.eng.brq2.redhat.com","pod_owner":"Node/dlms2.sa-iberia.lab.eng.brq2.redhat.com"},"level":"warn","{*}log_type{*}":"{*}{color:#FF0000}application{color}{*}","message":"\{\"level\":\"warn\",\"ts\":\"2024-02-29T03:41:15.873153Z\",\"caller\":\"rafthttp/peer.go:267\",\"msg\":\"dropped internal Raft message since sending buffer is full (overloaded network)\",\"message-type\":\"MsgHeartbeat\",\"local-member-id\":\"5b394353c83fce62\",\"from\":\"5b394353c83fce62\",\"remote-peer-id\":\"87bbff8f57e6c290\",\"remote-peer-name\":\"pipeline\",\"remote-peer-active\":false}","openshift":\{"cluster_id":"edf2e1a1-aa36-4464-ba23-b83e29cb4aef","sequence":1709205430408556822}}