-
Bug
-
Resolution: Done
-
Undefined
-
Logging 6.1.z
-
Incidents & Support
-
False
-
-
False
-
NEW
-
VERIFIED
-
Release Note Not Required
-
-
-
Log Collection - Sprint 268, Log Collection - Sprint 269, Log Collection - Sprint 270, Log Collection - Sprint 271, Log Collection - Sprint 272
-
Important
Description of problem:
When configured in Logging 6.1.3 to log forward to syslog with the most simple configuration:
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: collector
namespace: openshift-logging
spec:
serviceAccount:
name: collector
outputs:
- name: systemlog
syslog:
appName: test
facility: local0
rfc: RFC5424
severity: informational
url: tcp://rsyslog-server.rsyslog-pj.svc:6514
type: syslog
- name: applog
syslog:
appName: test
facility: local0
rfc: RFC5424
severity: informational
url: tcp://rsyslog-server.rsyslog-pj.svc:6514
type: syslog
pipelines:
- inputRefs:
- infrastructure
name: oc-syslog
outputRefs:
- systemlog
- inputRefs:
- application
name: oc-applog
outputRefs:
- applog
The syslog configuration is also basic:
$ModLoad imtcp $ModLoad imudp $InputTCPServerRun 6514 $UDPServerRun 6514 # Increase the amount of open files rsyslog is allowed, which includes open tcp sockets # This is important if there are many clients. # http://www.rsyslog.com/doc/rsconf1_maxopenfiles.html $MaxOpenFiles 2048 *.* /var/log/messages
Then, it's observed logs in the syslog server like:
Mar 12 22:23:10 example.com test[3f40848b-98af-4c88-9c92-3bb7f270f100] {"@timestamp":"2025-03-12T22:23:10.639887194Z","app_name":"test","hostname":"exampe.com","kubernetes":{"annotations":{"debug.openshift.io/source-container":"container-00","debug.openshift.io/source-resource":"/v1, Resource=nodes/example.com","openshift.io/scc":"privileged"},"container_id":"cri-o://87e3113a247078b626344eda64526a2d5fa3ad97f255a959f0b91a6e2f2b87a8","container_image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f97ae8ff716837a5147b7c8c5aae366417479bbcedf457700d3cee822057d9f","container_image_id":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f97ae8ff716837a5147b7c8c5aae366417479bbcedf457700d3cee822057d9f","container_iostream":"stdout","container_name":"container-00","namespace_id":"3f5c2da7-df58-4a73-a3e0-7d8235069306","namespace_labels":{"kubernetes_io_metadata_name":"openshift-debug-6hsw8","pod-security_kubernetes_io_audit":"privileged","pod-security_kubernetes_io_enforce":"privileged","pod-security_kubernetes_io_warn":"privileged","security_openshift_io_scc_podSecurityLabelSync":"false"},"namespace_name":"openshift-debug-6hsw8","pod_id":"3f40848b-98af-4c88-9c92-3bb7f270f100","pod_ip":"10.37.205.36","pod_name":"example.com-0-l2zfs-debug-x8fhj"},"level":"warn","log_source":"container","log_type":"infrastructure","message":"Y,��[W�\u0015<134>1 2025-03-12T22:21:08.291Z example.com test a4335afe-cc6b-4569-9c17-a7efb5113b6c container - {\"@timestamp\":\"2025-03-12T22:21:08.291092261Z\",\"app_name\":\"test\",\"hostname\":\"example.com\",\"kubernetes\":{\"annotations\":{\"flows.netobserv.io/config-digest\":\"1lheivj9unwvc\",\"k8s.ovn.org/pod-networks\":\"{\\\"default\\\":{\\\"ip_addresses\\\":[\\\"10.128.3.79/23\\\"],\\\"mac_address\\\":\\\"0a:58:0a:80:03:4f\\\",\\\"gateway_ips\\\":[\\\"10.128.2.1\\\"],\\\"routes\\\":[{\\\"dest\\\":\\\"10.128.0.0/14\\\",\\\"nextHop\\\":\\\"10.128.2.1\\\"},{\\\"dest\\\":\\\"172.30.0.0/16\\\",\\\"nextHop\\\":\\\"10.128.2.1\\\"},{\\\"dest\\\":\\\"169.254.0.5/32\\\",\\\"nextHop\\\":\\\"10.128.2.1\\\"},{\\\"dest\\\":\\\"100.64.0.0/16\\\",\\\"nextHop\\\":\\\"10.128.2.1\\\"}],\\\"ip_address\\\":\\\"10.128.3.79/23\\\",\\\"gateway_ip\\\":\\\"10.128.2.1\\\",\\\"role\\\":\\\"primary\\\"}}\",\"k8s.v1.cni.cncf.io/network-status\":\"[{\\n \\\"name\\\": \\\"ovn-kubernetes\\\",\\n \\\"interface\\\": \\\"eth0\\\",\\n \\\"ips\\\": [\\n \\\"10.128.3.79\\\"\\n ],\\n \\\"mac\\\": \\\"0a:58:0a:80:03:4f\\\",\\n \\\"default\\\": true,\\n \\\"dns\\\": {}\\n}]\",\"openshift.io/scc\":\"hostnetwork\",\"prometheus.io/scrape\":\"true\",\"prometheus.io/scrape_port\":\"9401\"},\"container_id\":\"cri-o://668606c2d36e1b5d85ed8f66467a088f4ede0b003436aa080b678553d32a9f56\",\"container_image\":\"registry.redhat.io/network-observability/network-observability-flowlogs-pipeline-rhel9@sha256:47540b6fd2cb1a0d89cb66859bc10b72e0266d828c6096ff0e9d83f4aa406579\",\"container_image_id\":\"registry.redhat.io/network-observability/network-observability-flowlogs-pipeline-rhel9@sha256:47540b6fd2cb1a0d89cb66859bc10b72e0266d828c6096ff0e9d83f4aa406579\",\"container_iostream\":\"stderr\",\"container_name\":\"flowlogs-pipeline\",\"labels\":{\"app\":\"flowlogs-pipeline\",\"controller-revision-hash\":\"74f7896955\",\"pod-template-generation\":\"2\",\"version\":\"47540b6fd2cb1a0d89cb66859bc10b72e0266d828c6096ff0e9d83f4aa40657\"},\"namespace_id\":\"6a6f87da-de73-4553-8b0e-c58251daa8e5\",\"namespace_labels\":{\"kubernetes_io_metadata_name\":\"netobserv\",\"olm_operatorgroup_uid_dbef20de-5e01-4784-83fc-9612ff232570\":\"\",\"openshift_io_cluster-monitoring\":\"true\",\"pod-security_kubernetes_io_audit\":\"privileged\",\"pod-security_kubernetes_io_audit-version\":\"latest\",\"pod-security_kubernetes_io_warn\":\"privileged\",\"pod-security_kubernetes_io_warn-version\":\"latest\"},\"namespace_name\":\"netobserv\",\"pod_id\":\"a4335afe-cc6b-4569-9c17-a7efb5113b6c\",\"pod_ip\":\"10.128.3.79\",\"pod_name\":\"flowlogs-pipeline-vjtwd\",\"pod_owner\":\"DaemonSet/flowlogs-pipeline\"},\"level\":\"warn\",\"log_source\":\"container\",\"log_type\":\"application\",\"message\":\"time=2025-03-12T22:21:08Z level=info component=client error=Post \\\"http://loki.netobserv.svc:3100/loki/api/v1/push\\\": dial tcp: lookup loki.netobserv.svc on 172.30.0.10:53: no such host fields.level=warn fields.msg=error sending batch, will retry host=loki.netobserv.svc:3100 module=export/loki status=-1\",\"msg_id\":\"container\",\"openshift\":{\"cluster_id\":\"8ae1d57c-a4b2-4991-968d-cd8f2728cdac\",\"sequence\":1741818068831693989},\"proc_id\":\"a4335afe-cc6b-4569-9c17-a7efb5113b6c\"}","msg_id":"container","openshift":{"cluster_id":"8ae1d57c-a4b2-4991-968d-cd8f2728cdac","sequence":1741818202866334261},"proc_id":"3f40848b-98af-4c88-9c92-3bb7f270f100"}
If it's reviewed this log line, it contains what they should be really two log lines. The second log line starts in the "<134>. It can checked that the "message" field from the first log line is not complete and some not UTF-8 characters are present. The exact part where the first log line should be finishing and starting the first is:
""message":"Y,��[W�\u0015<134>
Version-Release number of selected component (if applicable):
$ oc get csv |grep logging cluster-logging.v6.1.2 Red Hat OpenShift Logging 6.1.2 cluster-logging.v6.1.1 Succeeded
How reproducible:
Always
Steps to Reproduce:
- Deploy a syslog server
- Deploy Logging 6.1.z latest
- Configure clusterLogForwarder as:
apiVersion: observability.openshift.io/v1 kind: ClusterLogForwarder metadata: name: collector namespace: openshift-logging spec: serviceAccount: name: collector outputs: - name: systemlog syslog: appName: test facility: local0 rfc: RFC5424 severity: informational url: tcp://syslog.svc:6514 type: syslog - name: applog syslog: appName: test facility: local0 rfc: RFC5424 severity: informational url: tcp://syslog.svc:6514 type: syslog pipelines: - inputRefs: - infrastructure name: oc-syslog outputRefs: - systemlog - inputRefs: - application name: oc-applog outputRefs: - applog
Actual results:
It's observed log lines joined when it should be expected to have them in indiviual log lines. Examples will be attached to the bug.
Expected results:
It should be expected to have the different log lines as individual log entries and not joined in the same.
Additional info:
- depends on
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
- links to
(1 links to)
