Loading...

Type: Bug
Resolution: Done
Priority: Undefined
Fix Version/s: Logging 6.2.3
Affects Version/s: Logging 6.1.z
Component/s: Log Collection
Labels:

Activity Type:
Incidents & Support
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Docs QE Status:
NEW
QE Status:
VERIFIED
Release Note Type:
Release Note Not Required
Intelligence Requested:
Market:

Sprint:
Log Collection - Sprint 268, Log Collection - Sprint 269, Log Collection - Sprint 270, Log Collection - Sprint 271, Log Collection - Sprint 272
Severity:
Important

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem:

When configured in Logging 6.1.3 to log forward to syslog with the most simple configuration:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: collector
  namespace: openshift-logging
spec:
  serviceAccount:
    name: collector
  outputs:
  - name: systemlog
    syslog:
      appName: test
      facility: local0
      rfc: RFC5424
      severity: informational
      url: tcp://rsyslog-server.rsyslog-pj.svc:6514 
    type: syslog
  - name: applog
    syslog:
      appName: test
      facility: local0
      rfc: RFC5424
      severity: informational
      url: tcp://rsyslog-server.rsyslog-pj.svc:6514
    type: syslog
  pipelines:
  - inputRefs:
    - infrastructure
    name: oc-syslog
    outputRefs:
    - systemlog
  - inputRefs:
    - application
    name: oc-applog
    outputRefs:
    - applog

The syslog configuration is also basic:

$ModLoad imtcp
$ModLoad imudp

$InputTCPServerRun 6514
$UDPServerRun 6514

# Increase the amount of open files rsyslog is allowed, which includes open tcp sockets
# This is important if there are many clients.
# http://www.rsyslog.com/doc/rsconf1_maxopenfiles.html
$MaxOpenFiles 2048
*.*                                                  /var/log/messages

Then, it's observed logs in the syslog server like:

Mar 12 22:23:10 example.com test[3f40848b-98af-4c88-9c92-3bb7f270f100] {"@timestamp":"2025-03-12T22:23:10.639887194Z","app_name":"test","hostname":"exampe.com","kubernetes":{"annotations":{"debug.openshift.io/source-container":"container-00","debug.openshift.io/source-resource":"/v1, Resource=nodes/example.com","openshift.io/scc":"privileged"},"container_id":"cri-o://87e3113a247078b626344eda64526a2d5fa3ad97f255a959f0b91a6e2f2b87a8","container_image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f97ae8ff716837a5147b7c8c5aae366417479bbcedf457700d3cee822057d9f","container_image_id":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f97ae8ff716837a5147b7c8c5aae366417479bbcedf457700d3cee822057d9f","container_iostream":"stdout","container_name":"container-00","namespace_id":"3f5c2da7-df58-4a73-a3e0-7d8235069306","namespace_labels":{"kubernetes_io_metadata_name":"openshift-debug-6hsw8","pod-security_kubernetes_io_audit":"privileged","pod-security_kubernetes_io_enforce":"privileged","pod-security_kubernetes_io_warn":"privileged","security_openshift_io_scc_podSecurityLabelSync":"false"},"namespace_name":"openshift-debug-6hsw8","pod_id":"3f40848b-98af-4c88-9c92-3bb7f270f100","pod_ip":"10.37.205.36","pod_name":"example.com-0-l2zfs-debug-x8fhj"},"level":"warn","log_source":"container","log_type":"infrastructure","message":"Y,��[W�\u0015<134>1 2025-03-12T22:21:08.291Z example.com test a4335afe-cc6b-4569-9c17-a7efb5113b6c container - {\"@timestamp\":\"2025-03-12T22:21:08.291092261Z\",\"app_name\":\"test\",\"hostname\":\"example.com\",\"kubernetes\":{\"annotations\":{\"flows.netobserv.io/config-digest\":\"1lheivj9unwvc\",\"k8s.ovn.org/pod-networks\":\"{\\\"default\\\":{\\\"ip_addresses\\\":[\\\"10.128.3.79/23\\\"],\\\"mac_address\\\":\\\"0a:58:0a:80:03:4f\\\",\\\"gateway_ips\\\":[\\\"10.128.2.1\\\"],\\\"routes\\\":[{\\\"dest\\\":\\\"10.128.0.0/14\\\",\\\"nextHop\\\":\\\"10.128.2.1\\\"},{\\\"dest\\\":\\\"172.30.0.0/16\\\",\\\"nextHop\\\":\\\"10.128.2.1\\\"},{\\\"dest\\\":\\\"169.254.0.5/32\\\",\\\"nextHop\\\":\\\"10.128.2.1\\\"},{\\\"dest\\\":\\\"100.64.0.0/16\\\",\\\"nextHop\\\":\\\"10.128.2.1\\\"}],\\\"ip_address\\\":\\\"10.128.3.79/23\\\",\\\"gateway_ip\\\":\\\"10.128.2.1\\\",\\\"role\\\":\\\"primary\\\"}}\",\"k8s.v1.cni.cncf.io/network-status\":\"[{\\n    \\\"name\\\": \\\"ovn-kubernetes\\\",\\n    \\\"interface\\\": \\\"eth0\\\",\\n    \\\"ips\\\": [\\n        \\\"10.128.3.79\\\"\\n    ],\\n    \\\"mac\\\": \\\"0a:58:0a:80:03:4f\\\",\\n    \\\"default\\\": true,\\n    \\\"dns\\\": {}\\n}]\",\"openshift.io/scc\":\"hostnetwork\",\"prometheus.io/scrape\":\"true\",\"prometheus.io/scrape_port\":\"9401\"},\"container_id\":\"cri-o://668606c2d36e1b5d85ed8f66467a088f4ede0b003436aa080b678553d32a9f56\",\"container_image\":\"registry.redhat.io/network-observability/network-observability-flowlogs-pipeline-rhel9@sha256:47540b6fd2cb1a0d89cb66859bc10b72e0266d828c6096ff0e9d83f4aa406579\",\"container_image_id\":\"registry.redhat.io/network-observability/network-observability-flowlogs-pipeline-rhel9@sha256:47540b6fd2cb1a0d89cb66859bc10b72e0266d828c6096ff0e9d83f4aa406579\",\"container_iostream\":\"stderr\",\"container_name\":\"flowlogs-pipeline\",\"labels\":{\"app\":\"flowlogs-pipeline\",\"controller-revision-hash\":\"74f7896955\",\"pod-template-generation\":\"2\",\"version\":\"47540b6fd2cb1a0d89cb66859bc10b72e0266d828c6096ff0e9d83f4aa40657\"},\"namespace_id\":\"6a6f87da-de73-4553-8b0e-c58251daa8e5\",\"namespace_labels\":{\"kubernetes_io_metadata_name\":\"netobserv\",\"olm_operatorgroup_uid_dbef20de-5e01-4784-83fc-9612ff232570\":\"\",\"openshift_io_cluster-monitoring\":\"true\",\"pod-security_kubernetes_io_audit\":\"privileged\",\"pod-security_kubernetes_io_audit-version\":\"latest\",\"pod-security_kubernetes_io_warn\":\"privileged\",\"pod-security_kubernetes_io_warn-version\":\"latest\"},\"namespace_name\":\"netobserv\",\"pod_id\":\"a4335afe-cc6b-4569-9c17-a7efb5113b6c\",\"pod_ip\":\"10.128.3.79\",\"pod_name\":\"flowlogs-pipeline-vjtwd\",\"pod_owner\":\"DaemonSet/flowlogs-pipeline\"},\"level\":\"warn\",\"log_source\":\"container\",\"log_type\":\"application\",\"message\":\"time=2025-03-12T22:21:08Z level=info component=client error=Post \\\"http://loki.netobserv.svc:3100/loki/api/v1/push\\\": dial tcp: lookup loki.netobserv.svc on 172.30.0.10:53: no such host fields.level=warn fields.msg=error sending batch, will retry host=loki.netobserv.svc:3100 module=export/loki status=-1\",\"msg_id\":\"container\",\"openshift\":{\"cluster_id\":\"8ae1d57c-a4b2-4991-968d-cd8f2728cdac\",\"sequence\":1741818068831693989},\"proc_id\":\"a4335afe-cc6b-4569-9c17-a7efb5113b6c\"}","msg_id":"container","openshift":{"cluster_id":"8ae1d57c-a4b2-4991-968d-cd8f2728cdac","sequence":1741818202866334261},"proc_id":"3f40848b-98af-4c88-9c92-3bb7f270f100"}

If it's reviewed this log line, it contains what they should be really two log lines. The second log line starts in the "<134>. It can checked that the "message" field from the first log line is not complete and some not UTF-8 characters are present. The exact part where the first log line should be finishing and starting the first is:

 ""message":"Y,��[W�\u0015<134>

Version-Release number of selected component (if applicable):

$ oc get csv |grep logging
cluster-logging.v6.1.2                  Red Hat OpenShift Logging                        6.1.2      cluster-logging.v6.1.1                  Succeeded

How reproducible:

Always

Steps to Reproduce:

Deploy a syslog server
Deploy Logging 6.1.z latest

Configure clusterLogForwarder as:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: collector
  namespace: openshift-logging
spec:
  serviceAccount:
    name: collector
  outputs:
  - name: systemlog
    syslog:
      appName: test
      facility: local0
      rfc: RFC5424
      severity: informational
      url: tcp://syslog.svc:6514 
    type: syslog
  - name: applog
    syslog:
      appName: test
      facility: local0
      rfc: RFC5424
      severity: informational
      url: tcp://syslog.svc:6514
    type: syslog
  pipelines:
  - inputRefs:
    - infrastructure
    name: oc-syslog
    outputRefs:
    - systemlog
  - inputRefs:
    - application
    name: oc-applog
    outputRefs:
    - applog

Actual results:

It's observed log lines joined when it should be expected to have them in indiviual log lines. Examples will be attached to the bug.

Expected results:

It should be expected to have the different log lines as individual log entries and not joined in the same.

Additional info:

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

log-6863.sample.tar.gz
16 kB
2025/03/13 12:22 PM
vector.toml
21 kB
2025/03/18 8:04 AM

depends on

LOG-7183 [release-6.0] Syslog: merging data from 'message' to the root of log makes log event inconsistent with ViaQ data model

Closed

LOG-7184 [release-6.1] Syslog: merging data from 'message' to the root of log make log event inconsistent with ViaQ data model

Closed

LOG-7185 [release-6.2] Syslog: merging data from 'message' to the root of log make log event inconsistent with ViaQ data model

Closed

LOG-7189 Syslog: merging data from 'message' to the root of log make log event inconsistent with ViaQ data model

Closed

links to

[KCS] Several log lines sent as one to the syslog server by the collector in RHOL 6

openshift/cluster-logging-operator#3034: [master]LOG-7189:avoid merge data from message to the log event

(1 links to)

Details

Description

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Attachments

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates