-
Bug
-
Resolution: Unresolved
-
Normal
-
Logging 5.9.0
-
False
-
None
-
False
-
NEW
-
NEW
-
-
-
Log Collection - Sprint 250, Log Collection - Sprint 252, Log Collection - Sprint 253
-
Important
Description of problem:
Vector rules for infrastructure collects log files ending in *.log. However, as documented in https://access.redhat.com/solutions/4924281, crio rotates logs with the configuration:
the rotation is being done by default at ~ 10M (containerLogMaxSize) maintaining 5 files (containerLogMaxFiles)
When a log file is rotated, it crio appends the rotation date, and now it does not match the infrastructure but the application tenant
Version-Release number of selected component (if applicable):
5.9.0
How reproducible:
Always
Steps to Reproduce:
- Go to the console
- Select the application tenant in the filter
- Use a LogQl like "{ log_type="application", kubernetes_namespace_name=~"openshift-.*" } | json". In any cluster running for some time it would show up records like the one I show in additional info.
Actual results:
Log collected to infrastructure and application several times.
Expected results:
Only one record collected exactly to infrastructure.
Additional info:
{"@timestamp":"2024-02-29T03:41:15.873213094Z","{*}file{*}":"{color:#FF0000}/var/log/pods/openshift-etcd_etcd-dlms2.sa-iberia.lab.eng.brq2.redhat.com_8c874e477ac9490f2308da857a0dfae4/etcd/3.log.{*}20240229-043220{*}{color}","hostname":"dlms2.sa-iberia.lab.eng.brq2.redhat.com","kubernetes":\{"annotations":{"kubectl.kubernetes.io/default-container":"etcd","kubernetes.io/config.hash":"8c874e477ac9490f2308da857a0dfae4","kubernetes.io/config.mirror":"8c874e477ac9490f2308da857a0dfae4","kubernetes.io/config.seen":"2024-02-28T22:27:53.882006079Z","kubernetes.io/config.source":"file","target.workload.openshift.io/management":"{\"effect\": \"PreferredDuringScheduling\"}"},"container_id":"cri-o://e259a052724b4920a9942388ed39327c08691bf875e48919feb52fd350572e71","container_image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c74c57f91f0f7ed26bb62f58c7b84c55750e51947fd6cc5711fa18f30b9f68c","container_image_id":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c74c57f91f0f7ed26bb62f58c7b84c55750e51947fd6cc5711fa18f30b9f68c","container_name":"etcd","labels":\{"app":"etcd","etcd":"true","k8s-app":"etcd","revision":"11"},"namespace_id":"78f1b483-db90-47a5-a924-0e63da75da89","namespace_labels":\{"kubernetes_io_metadata_name":"openshift-etcd","openshift_io_run-level":"0","pod-security_kubernetes_io_audit":"privileged","pod-security_kubernetes_io_enforce":"privileged","pod-security_kubernetes_io_warn":"privileged"},"{*}namespace_name{*}":"{*}{color:#FF0000}openshift-etcd{color}{*}","pod_id":"afa0e018-6975-43fd-b16d-d4d250d08faa","pod_ip":"10.37.233.4","pod_name":"etcd-dlms2.sa-iberia.lab.eng.brq2.redhat.com","pod_owner":"Node/dlms2.sa-iberia.lab.eng.brq2.redhat.com"},"level":"warn","{*}log_type{*}":"{*}{color:#FF0000}application{color}{*}","message":"\{\"level\":\"warn\",\"ts\":\"2024-02-29T03:41:15.873153Z\",\"caller\":\"rafthttp/peer.go:267\",\"msg\":\"dropped internal Raft message since sending buffer is full (overloaded network)\",\"message-type\":\"MsgHeartbeat\",\"local-member-id\":\"5b394353c83fce62\",\"from\":\"5b394353c83fce62\",\"remote-peer-id\":\"87bbff8f57e6c290\",\"remote-peer-name\":\"pipeline\",\"remote-peer-active\":false}","openshift":\{"cluster_id":"edf2e1a1-aa36-4464-ba23-b83e29cb4aef","sequence":1709205430408556822}}
- links to