-
Bug
-
Resolution: Duplicate
-
Undefined
-
None
-
Logging 5.8.19, Logging 5.9.13
-
None
-
False
-
-
False
-
NEW
-
NEW
-
Bug Fix
-
-
-
Important
Description of problem:
Vector duplicates the fields inside a log message when log forwarding to syslog producing an excessive load.
Take as example the original log as produced by the application:
{"log_type":"openshift_audit","event type":"Sign on Success","userName":"XXXXX","event":"LEEF:1.0|redhat|openshift|4.12|authenticate devTime=2025-04-03T07:46:30.621410481-05:00[America/Chicago] devTimeFormat=yyyy-MM-dd HH:mm:ssZ requestClientApplication=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 requestMethod=GET sourceServiceName=xxx.xxx.xx.xxx src=xxx.xxx.xx.xxx srcPort=8080 dst=xxx.xxx.xx.xxx dstPort=35302 proto=HTTP/1.1 apiUrl=https://server.example.com/resource-server/actuator/health"}
Let's see how it's received on the syslog server where the fields above are duplicated appearing inside the "message" and also out of the message field as it could be the "sourceServiceName":
Apr 7 15:05:06 server.example.com vector {"@timestamp":"2025-04-07T15:05:06.245293157Z","event":"LEEF:1.0|redhat|openshift|4.12|authenticate devTime=2025-04-03T07:46:30.621410481-05:00[America/Chicago] devTimeFormat=yyyy-MM-dd HH:mm:ssZ requestClientApplication=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 requestMethod=GET sourceServiceName=xxx.xxx.xx.xxx src=xxx.xxx.xx.xxx srcPort=8080 dst=xxx.xxx.xx.xxx dstPort=35302 proto=HTTP/1.1 apiUrl=https://server.example.com/resource-server/actuator/health","event type":"Sign on Success","file":"/var/log/pods/syslogtest_hello-node-8dd54cb99-5hsbs_7076289c-ca8b-4e99-a7fc-fffe9f1e295e/agnhost/0.log","hostname":"server.example.com","kubernetes":{"annotations":{"k8s.ovn.org/pod-networks":"{\"default\":{\"ip_addresses\":[\"10.128.2.92/23\"],\"mac_address\":\"0a:58:0a:80:02:5c\",\"gateway_ips\":[\"10.128.2.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.128.2.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.128.2.1\"},{\"dest\":\"169.254.169.5/32\",\"nextHop\":\"10.128.2.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.128.2.1\"}],\"ip_address\":\"10.128.2.92/23\",\"gateway_ip\":\"10.128.2.1\"}}","k8s.v1.cni.cncf.io/network-status":"[{\n \"name\": \"ovn-kubernetes\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.128.2.92\"\n ],\n \"mac\": \"0a:58:0a:80:02:5c\",\n \"default\": true,\n \"dns\": {}\n}]","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_id":"cri-o://6cfd642e5d466988ba6952844f539cd35ec1305908389fc1258b1550bae8ccb6","container_image":"registry.k8s.io/e2e-test-images/agnhost:2.43","container_name":"agnhost","labels":{"app":"hello-node","pod-template-hash":"8dd54cb99"},"namespace_id":"5b84ab56-c441-4942-b241-5e7a75799774","namespace_labels":{"kubernetes_io_metadata_name":"syslogtest","pod-security_kubernetes_io_audit":"restricted","pod-security_kubernetes_io_audit-version":"v1.24","pod-security_kubernetes_io_warn":"restricted","pod-security_kubernetes_io_warn-version":"v1.24"},"namespace_name":"syslogtest","pod_id":"7076289c-ca8b-4e99-a7fc-fffe9f1e295e","pod_ip":"10.128.2.92","pod_name":"hello-node-8dd54cb99-5hsbs","pod_owner":"ReplicaSet/hello-node-8dd54cb99"},"level":"default","log_type":"openshift_audit","message":"{\"log_type\":\"openshift_audit\",\"event type\":\"Sign on Success\",\"userName\":\"XXXXXXX\",\"event\":\"LEEF:1.0|redhat|openshift|4.12|authenticate devTime=2025-04-03T07:46:30.621410481-05:00[America/Chicago] devTimeFormat=yyyy-MM-dd HH:mm:ssZ requestClientApplication=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 requestMethod=GET sourceServiceName=xxx.xxx.xx.xxx src=xxx.xxx.xx.xxx srcPort=8080 dst=xxx.xxx.xx.xxx dstPort=35302 proto=HTTP/1.1 apiUrl=https://server.example.com/resource-server/actuator/health\"}","openshift":{"cluster_id":"ec905b28-0bd2-4ab7-bcdc-201125e35249","sequence":3273},"userName":"XXXXXXX"}
Version-Release number of selected component (if applicable):
Logging 5.8.19 and Logging 5.9.12
Vector
How reproducible:
Always
Steps to Reproduce:
Steps to Reproduce:
- Deploy a syslog server
- Deploy the Cluster Logging Operator with Vector as collector type
- Configure clusterLogForwarder for log forwarding to a syslog server with the next configuration:
apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - name: logs type: syslog url: tcp://rsyslog-server.rsyslog-pj.svc:6514 pipelines: - inputRefs: - application - audit name: syslog-pl outputRefs: - logs
Actual results:
Fields inside the original log message are observed as duplicated in the log received by the syslog server
Expected results:
The content of the log message is not observed as duplicated in the log received by the syslog server
Additional info:
- duplicates
-
-
- Closed
-
