Summary

Refactor the observability API and supporting code to enable users of ClusterLogForwarder to utilize templeting for tenancy
so

• their log records can be stored "dynamically" based upon log fields
• We can drop "special" API (e.g. structuredKeyName)

Acceptance Criteria

• Verify output spec allows a fixed tenancy (e.g. app-write)
• Verify output spec allows a dynamic tenancy based upon a log field e.g:

  app-{{.kubernetes.labels.foo}}

• Verify output spec tenancy is restricted at admission to: curly brackets to indicate the template, characters which allow you to "walk" a log record path

Notes

• Evaluate if Azure.log_type should support templating given there is a RE to restrict what it should be
• Evaludate CW GroupBy to replace enum with freeform...

Questions

• Can we allow a fallback when a dynamic field is missing?
  • Vector doesn’t currently support fallback values. In the interim, you can use the remap transform to set a default value
  • If a field is missing, an error is logged and Vector drops the event. The component_errors_total internal metric is incremented with an error_type tag of template_failed.
• Can we only document record is lost?
• Can we or should we add alert or dashboard to identify count of dropped records?

Informative Links

• Vector's template syntax