-
Bug
-
Resolution: Done
-
Major
-
Logging 5.7.0
-
False
-
None
-
False
-
NEW
-
VERIFIED
-
-
Bug Fix
-
-
Description of problem:
Syslog forwarding's output for journal log is different between Vector and Fluentd.
When configuring syslog forwarding with vector:
tag is set to a value specified in ClusterLogForwarding object.
namespace_name, container_name and pod_name are added with an empth value.
2023-05-16T08:28:36+00:00 <xxxxxx> openshift-logging-infra: namespace_name=, container_name=, pod_name=, message=E0516 08:28:36.784935 2352 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"ba1dc4ed-bd65-4382-a519-b7e1ef38b333\" found, but failed to rmdir() subpath at path /var/lib/kubelet/pods/ba1dc4ed-bd65-4382-a519-b7e1ef38b333/volume-subpaths/etc/tuned/4: not a directory" numErrs=1
When configuring syslog forwarding with Fluentd:
tag isn't modified. Fluentd keeps the original host process or host service name which outputs the messagee on the host.
namespace_name, container_name and pod_name aren't added.
2023-05-16T08:42:42+00:00 <xxxxx> kubenswrapper[2352]: E0516 06:53:14.784003 2352 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"ba1dc4ed-bd65-4382-a519-b7e1ef38b333\" found, but failed to rmdir() subpath at path /var/lib/kubelet/pods/ba1dc4ed-bd65-4382-a519-b7e1ef38b333/volume-subpaths/etc/tuned/4: not a directory" numErrs=1
It seems that vector handles journal log as container log when configuring syslog forwarding.
It should be fixed.
Version-Release number of selected component (if applicable):
OpenShift v4.13.0-rc.8
ClusterLogging Operator v5.7.0
How reproducible:
Always
Steps to Reproduce:
Step1. Deploy syslog service on OCP
Step2. Install OpenShift Logging 5.7.0.
Step3. Apply the following ClusterLogging object as to deploy collector pods with a vector image.
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance namespace: openshift-logging spec: collection: type: vector managementState: Managed
Step4. Apply the following ClusterLogForwarder object as to enable syslog forwarding.
apiVersion: "logging.openshift.io/v1" kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - name: external-rsyslog-infra type: syslog url: '<SYSLOG-URL>' syslog: rfc: RFC3164 facility: local0 severity: informational addLogSource: true payloadKey: message tag: openshift-logging-infra pipelines: - inputRefs: - infrastructure name: infra outputRefs: - external-rsyslog-infra
Step5. Check if /tmp/openshift-journal.log is created or not. (/tmp/openshift-journal.log would be created if your log collector keeps original tags)
$oc exec -it -n syslog deployment/<rsyslog-server> – bash -c "ls -l /tmp/openshift*.log"
Actual results:
Only /tmp/openshift-infra-container.log was created on the syslog pod.
$ oc exec -it -n syslog deployment/rsyslog-server – bash -c "ls -l /tmp/openshift*.log" rw-rr-. 1 1000760000 root 30792560 May 16 08:46 /tmp/openshift-infra-container.log Journal log was outputted into /tmp/openshift-infra-container.log with the following format. 2023-05-16T08:28:36+00:00 <xxxxx> openshift-logging-infra: namespace_name=, container_name=, pod_name=, message=E0516 08:28:36.784935 2352 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"ba1dc4ed-bd65-4382-a519-b7e1ef38b333\" found, but failed to rmdir() subpath at path /var/lib/kubelet/pods/ba1dc4ed-bd65-4382-a519-b7e1ef38b333/volume-subpaths/etc/tuned/4: not a directory" numErrs=1
Expected results:
/tmp/openshift-journal.log is also created as well as when we tested with fluentd.
$ oc exec -it -n syslog deployment/rsyslog-server – bash -c "ls -l /tmp/openshift*.log"
rw-rr-. 1 1000760000 root 30792560 May 16 08:46 /tmp/openshift-infra-container.log
rw-rr-. 1 1000760000 root 1643814 May 16 08:46 /tmp/openshift-journal.log
