Summary

When JSON log records are forwarded to Elasticsearch, messages with different JSON formats must be directed to different indices.

The ClusterLogForwarder API can direct logs from different pods to different indices, using the structuredTypeKey and structuredTypeName fields.

This proposal extends `output.elasticsearch` to allow logs from different containers within a single Pod to be sent to different indices using annotations.

Motivation

• Sidecars are a common pattern in kubernetes and openshift clusters, which means there will be multiple containers in a Pod.
• Many popular sidecars (for example ISTIO) use JSON logging.
• The JSON log formats of sidecars and application containers may not be compatible, and must be separated to avoid index problems with Elasticsearch.
• K8s annotations are used to https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/(attach arbitrary non-identifying metadata to objects)

Acceptance Criteria

If a pod has

1. An annotation of the form: containerType.logging.openshift.io/container-name: {}structured-type-name
2. A container with name = container-name

Then logs from that container should be routed the as if the structuredTypeName field was set to structured-type-name.

• Verify that such logs are correctly directed to the desired index.
• Verify that logs from other containers in the same pod are routed as normal.
• Verify that if structuredTypeName or structuredTypeKey are set and there is a strutured-type annotation, the annotation takes precedenc

These annotations should behave in the same way as  structuredTypeName, see LOG-785 and user documentation for more detail.