-
Bug
-
Resolution: Done-Errata
-
Normal
-
Logging 6.0.0
-
False
-
None
-
False
-
NEW
-
NEW
-
-
Bug Fix
-
-
-
Log Collection - Sprint 262
Description of problem:
"obsclf.spec.outputs.syslog.enrichment: KubernetesMinimal" (as "clf.spec.outputs.syslog.addLogSource: true" in Logging5.x) is the option to add namespace_name=, pod_name= and container_name= info when forwarding logs.
Since those info are related only to container, they should be added only to container logs.
But vector attempts to add those info even to journal and audit logs as follows.
$ cat infra-node.worker0.example.com.log | grep RUNTIME_SCOPE | tail -n 1 | jq .
{
"timegenerated": "Oct 11 11:15:05",
"timereported": "Oct 11 02:17:50",
"hostname": "worker0.example.com",
"syslogfacility-text": "local0",
"syslogpriority-text": "info",
"syslogtag": ":",
"programname": "",
"app-name": "-",
"procid": "-",
"msgid": "-",
"msg": " namespace_name=, container_name=, pod_name=, message={\"@timestamp\":..."
}
Version-Release number of selected component (if applicable):
cluster-logging.v6.0.0
How reproducible:
Always
Steps to Reproduce:
Step1. Prepare rsyslog server outside of the cluster.
We use the following conf in rsyslog server to handle logs forwarded from vector.
$ cat /etc/rsyslog.d/remotelog.conf
template(name="OpenShiftLogging_InfraNodeLogFile" type="list") {
constant(value="/var/log/remote/infra-node.")
property(name="hostname")
constant(value=".log")
}
template(name="OpenShiftLogging_InfraContainerLogFile" type="list") {
constant(value="/var/log/remote/infra-container.")
property(name="hostname")
constant(value=".log")
}
template(name="OpenShiftLogging_AppLogFile" type="list") {
constant(value="/var/log/remote/app.")
property(name="hostname")
constant(value=".log")
}
template(name="OpenShiftLogging_AuditLogFile" type="list") {
constant(value="/var/log/remote/audit.")
property(name="hostname")
constant(value=".log")
}
template(name="DebugJsonFormat" type="list") {
constant(value="{")
property(name="timegenerated" format="jsonf")
constant(value=",")
property(name="timereported" format="jsonf")
constant(value=",")
property(name="hostname" format="jsonf")
constant(value=",")
property(name="syslogfacility-text" format="jsonf")
constant(value=",")
property(name="syslogpriority-text" format="jsonf")
constant(value=",")
property(name="syslogtag" format="jsonf")
constant(value=",")
property(name="programname" format="jsonf")
constant(value=",")
property(name="app-name" format="jsonf")
constant(value=",")
property(name="procid" format="jsonf")
constant(value=",")
property(name="msgid" format="jsonf")
constant(value=",")
property(name="msg" format="jsonf")
constant(value="}\n")
}
# Adding this ruleset to process remote messages
ruleset(name="OpenShiftLogging_Ruleset") {
if ($syslogfacility-text == "local0") then {
if ($syslogtag startswith "openshift-logging-infra") then {
*.* action(type="omfile" DynaFile="OpenShiftLogging_InfraContainerLogFile" Template="DebugJsonFormat")
} else {
*.* action(type="omfile" DynaFile="OpenShiftLogging_InfraNodeLogFile" Template="DebugJsonFormat")
}
} else if ($syslogfacility-text == "local1") then {
*.* action(type="omfile" DynaFile="OpenShiftLogging_AppLogFile" Template="DebugJsonFormat")
} else if ($syslogfacility-text == "local2") then {
*.* action(type="omfile" DynaFile="OpenShiftLogging_AuditLogFile" Template="DebugJsonFormat")
}
}
# Load the imptcp module to provide the ability to receive messages over plain TCP
module(load="imtcp")
input(type="imtcp" port="514" ruleset="OpenShiftLogging_Ruleset")
Step2. Deploy vector from Cluster Logging Operator. [1]
(You don't need to deploy Loki. You just need to deploy vector)
Step3. Apply the following yaml to forward infrastructure logs(which include journal logs) to rsyslog server.
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: collector
namespace: openshift-logging
spec:
managementState: Managed
serviceAccount:
name: collector
collector:
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 100m
memory: 64Mi
outputs:
- name: external-rsyslog-infra
type: syslog
syslog:
url: 'tcp://<your rsyslog server>:514'
enrichment: KubernetesMinimal
rfc: RFC3164
facility: local0
severity: informational
tag: openshift-logging-infra
pipelines:
- name: pipeline-infra
outputRefs:
- external-rsyslog-infra
inputRefs:
- infrastructure
Step4. Log in to rsyslog server then check journal logs(/var/log/remote/infra-node.*.log) forwarded from vector.
[1] https://docs.openshift.com/container-platform/4.15/observability/logging/logging-6.0/log6x-clf.html
Actual results:
Unnecessary LogSource template strings (namespace_name=, container_name=, pod_name=) are included in "msg" field of non-container logs.
Expected results:
Unnecessary LogSource template strings should not be included in "msg" field of non-container logs.
Additional information:
Just like when sending journal logs, unnecessary LogSource template strings are included in "msg" field when sending audit logs.
$ sudo tail -n 1 audit.master0.example.com.log | jq .
{
"timegenerated": "Oct 11 10:20:03",
"timereported": "Oct 11 01:22:48",
"hostname": "master0.example.com",
"syslogfacility-text": "local2",
"syslogpriority-text": "info",
"syslogtag": ":",
"programname": "",
"app-name": "-",
"procid": "-",
"msgid": "-",
"msg": " namespace_name=, container_name=, pod_name=, message={\"@timestamp\":..."
}
- links to
-
RHBA-2024:140332
Logging for Red Hat OpenShift - 6.0.2