Story

As a receiver of JSON logs, I want the JSON to appear in "normal" form, in particular I want kubernetes.labels to be an object field with the original, unmodified k8s label names and values.

Currently the forwarder "flattens" labels into an array field called kubernetes.flat_labels, to work around elasticsearch indexing problems.

Make flattening optional:

• New Output field: format  (string, optional, default "flat")

• • "flat": do the flattening as we do today (required with our managed ES)
  • "normal": no flattening or other transformations.

The default behavior is the same as today, so applications that expect the existing "flat" format will not be broken.

Applications that want normal format can configure it on the outputs.

At some point in the future we may deprecate the "flat" format and/or change the default to "normal".

Acceptance Criteria

• Log records are flattened by default, and with format: flat
• Log records are not flattened with format: normal
• No other flattening, de-dotting or other such transformations are performed with format: normal

Notesmmulti

In normal format we do not forward the flat_labels field, only the labels field.

Part of this story is to review the existing behavior to check for any other mangling of fields that is happening. This story should remove all mangling of the record.