Goal

• Document exact API and behavior to achieve the goals of LOG-2827 - control output encoding and fields.

Acceptance criteria

• Enhancement proposal document describing new API and behavior.

Initial Design Notes

New fields for ClusterLogForwarder.output:

output.encoding: (enum, optional, default "json")

• json: wrap log text in JSON envelope - the current behavior
• raw: forward original text as logged by application - which may be JSON
  • Process CRIO format to restore original text and line breaks.

output.fields: (list, optional, default [all]) select meta-data fields to include in envelope

• any.field.name: include the named field, using the Viaq dotted field name.
• -any.field.name: exclude the named field, using the Viaq dotted field name.
• Note: output.fields also apply in raw mode if the raw content is JSON, if not it is ignored.

• Special group names:
  • minimal: minimal source-id fields (TBD). Bounded size, no labels or other unbounded maps/lists.
  • common: commonly-used fields (TBD) including labels and other unbounded structures. Includes minimal
  • all: all available fields, everything we do today and more. Includes common

Notes:

• output.fields: [] is the same as omitting output.fields - use default value {https://issues.redhat.com/browse/LOG-2920{[all]}}.
• The message or structured fields are always included. This option affects only additional meta-data.
• For no meta-data at all use output.encoding:raw - original message only.
• Default may change default in future to a slimmer option: [common] or [minimal]

Examples

Send original log text only. Log text must contain sufficient metadata, for example netflow events, k8s events.

outputs:
  name: rawOutput
  encoding:raw

Send the minimal fields, plus one POD label value.

outputs:
  name: customizedOutput
  fields: [minimal, kubernetes.labels.foo]

Open Questions

• Logging console, LokiStack: should work correctly with mininmal and at least not error with none.
• Elasticsearch? Requires all?
• Syslog - has its own encoding, how does addLogSource fit with the above rules? == minimal?h1. Goal
• Document exact API and behavior to achieve the goals of LOG-2827 - control output encoding and fields.

Special-case JSON logs

We currently have multiple ways of handling JSON logs:

1. Defualt: viaq envelope, field "message" = JSON quoted string,
2. Structured (configuredin CLF): viaq envelope, field "structured" = JSON object
3. Special  cases:
   1. API-server audit events: some viaq fields are embedded directly in the event
   2. eventrouter events: no metadata, LOG-3987 proposes to make them consistent with API events

Since API server and k8s events are well known formats, and many customers already parse them, we can't easily drop the special-case formats with embedded metadata.

Should we provide configurable options for all JSON logs to be formatted either in envelope or embedded form? Or allow all JSON logs in "normal" envelope form, but keep the special cases for events for compatibility?

The embedded form could cause name clashes, so it is risky to extend it to all JSON events.