Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: Logging 5.9.16
Affects Version/s: Logging 5.9.12, Logging 5.8.19
Component/s: Log Collection
Labels:
- devel_ack+

Activity Type:
Future Sustainability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Docs QE Status:
NEW
QE Status:
NEW
Release Note Text:

Hide
Before this update, Vector could overwrite the log_type when writing to syslog if .message also included a log_type field. With this update, the .message field is replaced if it contains a valid JSON string, allowing structured data to be handled correctly and preserving nested fields during syslog encoding.

Show
Before this update, Vector could overwrite the log_type when writing to syslog if .message also included a log_type field. With this update, the .message field is replaced if it contains a valid JSON string, allowing structured data to be handled correctly and preserving nested fields during syslog encoding.
Release Note Type:
Bug Fix
Intelligence Requested:
Market:

Sprint:
Log Collection - Sprint 269, Log Collection - Sprint 270, Log Collection - Sprint 271, Log Collection - Sprint 273, Logging - Sprint 274, Logging - Sprint 275, Logging - Sprint 276
Severity:
Important

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem:

When the application is generation a log that contains inside the message a field like "log_type":"openshift_audit", the log message is labelled by Vector as audit log when log forwarding to syslog:

Original log:

{"log_type":"openshift_audit","event type":"Sign on Success","userName":"XXXXX","event":"LEEF:1.0|redhat|openshift|4.12|authenticate devTime=2025-04-03T07:46:30.621410481-05:00[America/Chicago] devTimeFormat=yyyy-MM-dd HH:mm:ssZ requestClientApplication=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 requestMethod=GET sourceServiceName=xxx.xxx.xx.xxx src=xxx.xxx.xx.xxx srcPort=8080 dst=xxx.xxx.xx.xxx dstPort=35302 proto=HTTP/1.1 apiUrl=https://server.example.com/resource-server/actuator/health"}

As Vector sends the message where it's visible that labelled as "log_type":"openshift_audit":

Apr  7 14:45:24 server.example.com vector {"@timestamp":"2025-04-07T14:45:24.699132308Z","event":"LEEF:1.0|redhat|openshift|4.12|Apr  7 15:05:06 server.example.com vector {"@timestamp":"2025-04-07T15:05:06.245293157Z","event":"LEEF:1.0|redhat|openshift|4.12|authenticate devTime=2025-04-03T07:46:30.621410481-05:00[America/Chicago] devTimeFormat=yyyy-MM-dd HH:mm:ssZ requestClientApplication=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 requestMethod=GET sourceServiceName=xxx.xxx.xx.xxx src=xxx.xxx.xx.xxx srcPort=8080 dst=xxx.xxx.xx.xxx dstPort=35302 proto=HTTP/1.1 apiUrl=https://server.example.com/resource-server/actuator/health","event type":"Sign on Success","file":"/var/log/pods/syslogtest_hello-node-8dd54cb99-5hsbs_7076289c-ca8b-4e99-a7fc-fffe9f1e295e/agnhost/0.log","hostname":"server.example.com","kubernetes":{"annotations":{"k8s.ovn.org/pod-networks":"{\"default\":{\"ip_addresses\":[\"10.128.2.92/23\"],\"mac_address\":\"0a:58:0a:80:02:5c\",\"gateway_ips\":[\"10.128.2.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.128.2.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.128.2.1\"},{\"dest\":\"169.254.169.5/32\",\"nextHop\":\"10.128.2.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.128.2.1\"}],\"ip_address\":\"10.128.2.92/23\",\"gateway_ip\":\"10.128.2.1\"}}","k8s.v1.cni.cncf.io/network-status":"[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.128.2.92\"\n    ],\n    \"mac\": \"0a:58:0a:80:02:5c\",\n    \"default\": true,\n    \"dns\": {}\n}]","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_id":"cri-o://6cfd642e5d466988ba6952844f539cd35ec1305908389fc1258b1550bae8ccb6","container_image":"registry.k8s.io/e2e-test-images/agnhost:2.43","container_name":"agnhost","labels":{"app":"hello-node","pod-template-hash":"8dd54cb99"},"namespace_id":"5b84ab56-c441-4942-b241-5e7a75799774","namespace_labels":{"kubernetes_io_metadata_name":"syslogtest","pod-security_kubernetes_io_audit":"restricted","pod-security_kubernetes_io_audit-version":"v1.24","pod-security_kubernetes_io_warn":"restricted","pod-security_kubernetes_io_warn-version":"v1.24"},"namespace_name":"syslogtest","pod_id":"7076289c-ca8b-4e99-a7fc-fffe9f1e295e","pod_ip":"10.128.2.92","pod_name":"hello-node-8dd54cb99-5hsbs","pod_owner":"ReplicaSet/hello-node-8dd54cb99"},"level":"default","log_type":"openshift_audit","message":"{\"log_type\":\"openshift_audit\",\"event type\":\"Sign on Success\",\"userName\":\"XXXXXXX\",\"event\":\"LEEF:1.0|redhat|openshift|4.12|authenticate devTime=2025-04-03T07:46:30.621410481-05:00[America/Chicago] devTimeFormat=yyyy-MM-dd HH:mm:ssZ requestClientApplication=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 requestMethod=GET sourceServiceName=xxx.xxx.xx.xxx src=xxx.xxx.xx.xxx srcPort=8080 dst=xxx.xxx.xx.xxx dstPort=35302 proto=HTTP/1.1 apiUrl=https://server.example.com/resource-server/actuator/health\"}","openshift":{"cluster_id":"ec905b28-0bd2-4ab7-bcdc-201125e35249","sequence":3273},"userName":"XXXXXXX"}

Small note: Fluentd was detecting the previous message sent by an application correctly as "application":

Apr  7 14:45:25 server.example.com fluentd {"@timestamp":"2025-04-07T14:45:24.699132308+00:00","message":{"log_type":"openshift_audit","event type":"Sign on Success","userName":"XXXXXXX","event":"LEEF:1.0|redhat|openshift|4.12|authenticate devTime=2025-04-03T07:46:30.621410481-05:00[America/Chicago] devTimeFormat=yyyy-MM-dd HH:mm:ssZ requestClientApplication=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 requestMethod=GET sourceServiceName=xxx.xxx.xx.xxx src=xxx.xxx.xx.xxx srcPort=8080 dst=xxx.xxx.xx.xxx dstPort=35302 proto=HTTP/1.1 apiUrl=https://server.example.com/resource-server/actuator/health"},"docker":{"container_id":"6cfd642e5d466988ba6952844f539cd35ec1305908389fc1258b1550bae8ccb6"},"kubernetes":{"container_name":"agnhost","namespace_name":"syslogtest","pod_name":"hello-node-8dd54cb99-5hsbs","container_image":"registry.k8s.io/e2e-test-images/agnhost:2.43","container_image_id":"registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e","pod_id":"7076289c-ca8b-4e99-a7fc-fffe9f1e295e","pod_ip":"10.128.2.92","host":"server.example.com","labels":{"app":"hello-node","pod-template-hash":"8dd54cb99"},"master_url":"https://kubernetes.default.svc","namespace_id":"5b84ab56-c441-4942-b241-5e7a75799774","namespace_labels":{"kubernetes_io_metadata_name":"syslogtest","pod-security_kubernetes_io_audit":"restricted","pod-security_kubernetes_io_audit-version":"v1.24","pod-security_kubernetes_io_warn":"restricted","pod-security_kubernetes_io_warn-version":"v1.24"},"flat_labels":["app=hello-node","pod-template-hash=8dd54cb99"]},"level":"unknown","hostname":"server.example.com","pipeline_metadata":{"collector":{"ipaddr4":"10.37.205.135","inputname":"fluent-plugin-systemd","name":"fluentd","received_at":"2025-04-07T14:45:24.702567+00:00","version":"1.16.2 1.6.0"}},"openshift":{"sequence":131954,"cluster_id":"ec905b28-0bd2-4ab7-bcdc-201125e35249"},"viaq_msg_id":"NjI1NjU3OTItOGNmYi00ZDJmLWJiY2EtNmExZWI5MjY0NTc5","log_type":"application"}

Version-Release number of selected component (if applicable):

Logging 5.8.19 and Logging 5.9.12
Vector

How reproducible:

Always

Steps to Reproduce:

Deploy a syslog server
Deploy the Cluster Logging Operator with Vector as collector type

Configure clusterLogForwarder for log forwarding to a syslog server with the next configuration:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
  - name: logs
    type: syslog
    url: tcp://rsyslog-server.rsyslog-pj.svc:6514
  pipelines:
  - inputRefs:
    - application
    - audit
    name: syslog-pl
    outputRefs:
    - logs

Actual results:

The log being received in the syslog server is labelled as application log:

[...] "level":"default","log_type":"openshift_audit","message":"{\"log_type\":\"openshift_audit\",\"event type\":\"Sign on Success [...]

Expected results:

The log is labelled as application log, even, when the original log message as produced by the application contains "log_level: openshift_audit". The log level shouldn't be overriden automatically depending on the content of the message at least that one explicit configuration done by the Openshift Admin should allow it (something not possible now).

Additional info:

depends on

LOG-7183 [release-6.0] Syslog: merging data from 'message' to the root of log makes log event inconsistent with ViaQ data model

Closed

LOG-7184 [release-6.1] Syslog: merging data from 'message' to the root of log make log event inconsistent with ViaQ data model

Closed

LOG-7185 [release-6.2] Syslog: merging data from 'message' to the root of log make log event inconsistent with ViaQ data model

Closed

LOG-7189 Syslog: merging data from 'message' to the root of log make log event inconsistent with ViaQ data model

Closed

is depended on by

LOG-7008 Vector sends duplicated the same log when log forwarding to syslog

Closed

LOG-7010 Vector duplicates the fields inside a log message when log forwarding to syslog

Closed

links to

openshift/cluster-logging-operator#3034: [master]LOG-7189:avoid merge data from message to the log event

Vector overwrites the log_type when forwarding messages to syslog with a log_type field in the message payload

mentioned on

Merge request - Pin vector to previous commit

Merge request - Updated US source to: 7a5fc40 LOG-7131: update documentation Tuning Delivery mode default settings

(1 is depended on by, 2 links to, 2 mentioned on)

Details

Description

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates