Goals

• Assemble log messages again that are part of a stacktrace to store it as one single log record instead of multiple.
• Look at Java stacktraces at a minimum as this seems to be the most common language used for workloads deployed on OpenShift.

Motivation

Log messages as part of a stacktrace are often split into multiple lines where each line represents a new log record. A log collector takes every log record and stores it as an individual entry into a log storage. In this case, a user has no way to look at the single stacktrace or manual assemble it inside the log store and the context is completely lost which makes it harder to investigate the issue. Therefore, a log solution needs to store all lines as part of a stacktraces.

Alternatives

Users can log stacktraces as JSON which would force the entire log to be a single log record instead of multiple per log line.

Additional Notes

• As in Christian's comment below[1], there is a Fluentd plugin from Google to detect stacktraces from different language[2].
• Christian was successfully able to use this with a minimal Fluentd configuration (see[3]).
• Found another example from someone who modified our Fluentd to support multiline. See https://gist.github.com/jgrumboe/e56d0ceb51026a5f39c576181a3bda23.
• Splunk Connect for K8s multiline configuration[4]

[1] https://issues.redhat.com/browse/LOG-843?focusedCommentId=14376072&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14376072
[2] https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions
[3] https://github.com/sichvoge/fluentd-detect-exception-example/blob/master/fluentd.conf
[4] https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/manifests/splunk-kubernetes-logging/configMap.yaml#L148