-
Bug
-
Resolution: Done
-
Undefined
-
Logging 5.2.z
-
False
-
False
-
NEW
-
NEW
-
Before this update, the release did not include a filter change which caused fluentd to crash. With this update, the missing filter has been corrected.
-
-
Logging (Core) - Sprint 212, Logging (Core) - Sprint 213
After the upgrade of the CLO to 5.2.4-17 all fluentd pods are crashing.
$ oc logs -f fluentd-ldd9b fluentd Setting each total_size_limit for 4 buffers to 4811272243 bytes Setting queued_chunks_limit_size for each buffer to 573 Setting chunk_limit_size for each buffer to 8388608 2021-12-28 15:32:41 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp 2021-12-28 15:32:42 +0000 [error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="Unknown formatter plugin 'single_json_value'. Run 'gem search -rd fluent-plugin' to find plugins"
The ClusterLogForwarder is specified like below
apiVersion: v1
items:
- apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
annotations:
name: instance
namespace: openshift-logging
spec:
outputDefaults:
elasticsearch:
structuredTypeKey: kubernetes.labels.logFormat
structuredTypeName: json
outputs:
- name: rsyslog
syslog:
facility: local0
payloadKey: message
rfc: RFC3164
severity: informational
type: syslog
url: 'tls://example.com:5514'
pipelines:
- inputRefs:
- audit
- application
- infrastructure
name: syslog
outputRefs:
- rsyslog
- default
parse: json
kind: List
metadata:
resourceVersion: ""
selfLink: ""
[LOG-2104] fluentd crashing after upgrade to openshift-logging 5.2.4-17
Anping Li
added a comment - Add New Test Cases: https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-47834
Anping Li
added a comment - Add New Test Cases: https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-47834 Anping Li
added a comment - Verified on cluster-logging.5.2.5-21, openshift-logging/fluentd-rhel8/images/v1.7.4-71
Anping Li
added a comment - Verified on cluster-logging.5.2.5-21, openshift-logging/fluentd-rhel8/images/v1.7.4-71 vimalkum@redhat.com are there changes need for any other release?
vimalkum@redhat.com please update the status, set target version, etc as required. I added to the current sprint. Please clone if appropriate for older releases to ensure we have the needed paper trail
A change was missing in cpaas Dockerfiles for 5.2, have added the missing statement
I have case and same error
[warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
in collector pods of CLO 5.4. Does this issue applies for v5.4 as well?
I also have countless
[warn]: dump an error event: error_class=Fluent::Plugin::ConcatFilter::TimeoutError error="Timeout flush:
errors for which record }}value starts with {{{{{}@timestamp. Is there any correlation between these errors and this issue?}}
EDIT NOTE: Just realized that the collector pods having too many "Timeout flush" errors also has "unreadable file" errors. It seems the unreadable files are causing ruby processing to fail in fluentd.