-
Bug
-
Resolution: Done
-
Major
-
None
-
Incidents & Support
-
False
-
-
False
-
NEW
-
NEW
-
-
Bug Fix
-
Moderate
Description of problem:
CLF sends the broken syslog format to syslog server under the following conditions:
- The logging-eventrouter is deployed to collect cluster events.
- An event is generated with a message that contains a newline character (\n).
The original event recorded in eventrouter pod's log.
{"verb":"ADDED","event":{"metadata":{"name":"machine-config-operator.18759f5645979b76.755fb693","namespace":"openshift-machine-config-operator","uid":"0c70b30c-5a2a-4b2c-b70e-f594f8f28bb1","resourceVersion":"91604783","creationTimestamp":"2025-11-07T04:33:42Z","managedFields":[{"manager":"machine-config-operator","operation":"Update","apiVersion":"v1","time":"2025-11-07T04:33:42Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{}},"f:type":{}}}]},"involvedObject":{"kind":"Deployment","namespace":"openshift-machine-config-operator","name":"machine-config-operator","apiVersion":"apps/v1"},"reason":"ConfigMapUpdated","message":"Updated ConfigMap/kube-rbac-proxy -n openshift-machine-config-operator:\ncause by changes in data.config-file.yaml","source":{"component":"machine-config-operator"},"firstTimestamp":"2025-11-07T04:33:42Z","lastTimestamp":"2025-11-07T04:33:42Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
The resulting syslog output is as follows.
The second line looks like a new record, but this is wrong.
The fields such as 'hostname', 'syslogtag', and others contain invalid values like "cause" and "by".
{"timegenerated":"Nov 7 04:33:42","timereported":"Nov 7 04:33:42","hostname":"infra0","syslogfacility-text":"local0","syslogpriority-text":"info","syslogtag":"application","programname":"application","app-name":"application","procid":"-","msgid":"-","msg":"namespace_name=logging-eventrouter, container_name=kube-eventrouter, pod_name=eventrouter-7747456cfd-ncls4, message=Updated ConfigMap\/kube-rbac-proxy -n openshift-machine-config-operator:"}
{"timegenerated":"Nov 7 04:33:42","timereported":"Nov 7 04:33:42","hostname":"cause","syslogfacility-text":"user","syslogpriority-text":"notice","syslogtag":"by","programname":"by","app-name":"by","procid":"-","msgid":"-","msg":" changes in data.config-file.yaml"}
Version-Release number of selected component (if applicable):
- OpenShift v4.20.0
- cluster-logging.v6.2.6
How reproducible:
Always
Steps to Reproduce:
Step1. Deploy Logging v6.2.6 and eventrouter on openshift-logging.
Step2. Define ClusterLogForwarder as to forward eventrouter's logs to syslog server.
Please modify <your syslog server> of the following example to an url of your syslog server.
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: collector
namespace: openshift-logging
spec:
managementState: Managed
outputs:
- name: external-rsyslog
syslog:
appName: '{.systemd.u.SYSLOG_IDENTIFIER||.log_type||"-"}'
enrichment: KubernetesMinimal
facility: local0
msgId: '{.systemd.u.MESSAGE_ID||"-"}'
payloadKey: '{.message}'
procId: '{.systemd.t.PID||"-"}'
rfc: RFC5424
severity: informational
tuning:
deliveryMode: AtLeastOnce
url: <your syslog server>
type: syslog
pipelines:
- inputRefs:
- infrastructure
name: infra
outputRefs:
- external-rsyslog
serviceAccount:
name: collector
Step3. Wait for a while, or cause an event which has "\n" manually by the following command.
$ oc edit -n openshift-machine-config-operator configmap kube-rbac-proxy
...
apiVersion: v1
data:
config-file.yaml: |-
test: "2" <=== Add this
authorization:
resourceAttributes:
apiVersion: v1
resource: namespace
subresource: metrics
namespace: openshift-machine-config-operator
Step4. Check if unexpected syslog record splitting occurred.
Actual results:
- Syslog format corruption occurs
- Unexpected syslog record splitting occurred.
- In the second record, metadata fields(hostname, syslogtag, etc.) are wrong.
{"timegenerated":"Nov 7 04:33:42","timereported":"Nov 7 04:33:42","hostname":"infra0","syslogfacility-text":"local0","syslogpriority-text":"info","syslogtag":"application","programname":"application","app-name":"application","procid":"-","msgid":"-","msg":"namespace_name=logging-eventrouter, container_name=kube-eventrouter, pod_name=eventrouter-7747456cfd-ncls4, message=Updated ConfigMap\/kube-rbac-proxy -n openshift-machine-config-operator:"} \{"timegenerated":"Nov 7 04:33:42","timereported":"Nov 7 04:33:42","hostname":"cause","syslogfacility-text":"user","syslogpriority-text":"notice","syslogtag":"by","programname":"by","app-name":"by","procid":"-","msgid":"-","msg":" changes in data.config-file.yaml"}
Expected results:
- The syslog format doesn't get broken.
- The event record isn't splitted with "\n".
- It should be outputted as one record as follows.
{"timegenerated":"Nov 7 04:33:42","timereported":"Nov 7 04:33:42","hostname":"infra0","syslogfacility-text":"local0","syslogpriority-text":"info","syslogtag":"application","programname":"application","app-name":"application","procid":"-","msgid":"-","msg":"namespace_name=logging-eventrouter, container_name=kube-eventrouter, pod_name=eventrouter-7747456cfd-ncls4, message=Updated ConfigMap/kube-rbac-proxy -n openshift-machine-config-operator:\ncause by changes in data.config-file.yaml"}
Additional info:
There are the following two points which ClusterLogForward should consider.
- Point1: Rsyslog recognizes a changing line character code (LF) as a delimiter, so each syslog record shouldn't include it in its message.
- Point2: ClusterLogForwarder calls parse_json for eventrouter's logs. It seems that it converts "\n" to LF.
$ oc exec -it -n openshift-logging ds/fluentd – cat /etc/vector/vector.toml | grep eventrouter- -A 1
if starts_with(pod_name, "eventrouter-")
{ parsed, err = parse_json(.message)
Point2 is the reason of why this issue can happen only with eventrouter's logs.
We got a tcpdump in syslog server.
It shows that "\n" was converted before syslog server receives logs. The second line "cause by changes in data.config-file.yaml" is invalid as syslog record, so metadata fields(hostname, syslogtag, etc.) become wrong.
02:42:34.797321 eth0 In IP infra0.35406 > syslog.514: Flags [P.], seq 21069:21384, ack 1, win 510, options [nop,nop,TS val 2859807676 ecr 2556327698], length 315 ... <134>1 2025-11-07T02:42:34.793Z infra0 application - - - namespace_name=logging-eventrouter, container_name=kube-eventrouter, pod_name=eventrouter-b7c84c458-4vjw7, message=Updated ConfigMap/kube-rbac-proxy -n openshift-machine-config-operator: cause by changes in data.config-file.yaml
