Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1296

Send JSON logs from containers in the same pod to separate indices

XMLWordPrintable

    • JSON Container to Separate Indices
    • 5
    • False
    • None
    • False
    • Green
    • NEW
    • In Progress
    • Impediment
    • VERIFIED
    • 0% To Do, 0% In Progress, 100% Done
    • Undefined

      Summary

      When JSON log records are forwarded to Elasticsearch, messages with different JSON formats must be directed to different indices.

      The ClusterLogForwarder API can direct logs from different pods to different indices, using the structuredTypeKey and structuredTypeName fields.

      This proposal extends `output.elasticsearch` to allow logs from different containers within a single Pod to be sent to different indices using annotations.

      Motivation

      • Sidecars are a common pattern in kubernetes and openshift clusters, which means there will be multiple containers in a Pod.
      • Many popular sidecars (for example ISTIO) use JSON logging.
      • The JSON log formats of sidecars and application containers may not be compatible, and must be separated to avoid index problems with Elasticsearch.
      • K8s annotations are used to https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/(attach arbitrary non-identifying metadata to objects)

      Acceptance Criteria

      If a pod has

      1. An annotation of the form: containerType.logging.openshift.io/container-name: {}structured-type-name
      2. A container with name = container-name

      Then logs from that container should be routed the as if the structuredTypeName field was set to structured-type-name.

      • Verify that such logs are correctly directed to the desired index.
      • Verify that logs from other containers in the same pod are routed as normal.
      • Verify that if structuredTypeName or structuredTypeKey are set and there is a strutured-type annotation, the annotation takes precedenc

      These annotations should behave in the same way as  structuredTypeName, see LOG-785 and user documentation for more detail.

            jcantril@redhat.com Jeffrey Cantrill
            rhn-engineering-aconway Alan Conway
            Qiaoling Tang Qiaoling Tang
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: