Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1148

Stop mangling JSON records for non-elasticsearch outputs

    XMLWordPrintable

Details

    • Story
    • Status: Closed (View Workflow)
    • Urgent
    • Resolution: Done
    • None
    • Logging 5.1
    • Core
    • None
    • Logging (Core) - Sprint 199, Logging (Core) - Sprint 200
    • 2
    • Passed
    • NEW

    Description

      Story

      As a receiver of JSON logs, I want the JSON to appear in "normal" form, in particular I want kubernetes.labels to be an object field with the original, unmodified k8s label names and values.

      Currently the forwarder "flattens" labels into an array field called kubernetes.flat_labels, to work around elasticsearch indexing problems.

      Make flattening optional:

      • New Output field: format  (string, optional, default "flat")
        • "flat": do the flattening as we do today (required with our managed ES)
        • "normal": no flattening or other transformations.

      The default behavior is the same as today, so applications that expect the existing "flat" format will not be broken.

      Applications that want normal format can configure it on the outputs.

      At some point in the future we may deprecate the "flat" format and/or change the default to "normal".

      Acceptance Criteria

      • Log records are flattened by default, and with format: flat
      • Log records are not flattened with format: normal
      • No other flattening, de-dotting or other such transformations are performed with format: normal

      Notesmmulti

      In normal format we do not forward the flat_labels field, only the labels field.

      Part of this story is to review the existing behavior to check for any other mangling of fields that is happening. This story should remove all mangling of the record.

      Attachments

        Issue Links

          Activity

            People

              vimalkum@redhat.com Vimal Kumar
              rhn-engineering-aconway Alan Conway
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: