Goals

• Bundle a supportable, easy to deploy agent that contains our new collector (see LOG-1321).
• Provide the ability to forward all logs to our internal Loki instance deployed via Observatorium.

Non-Goals

• Fully functional, on-par with our current Fluentd, L5 Operator. We only want to collect certain logs and forward them to a single sink (Loki) at the moment.

Motivation

To continue our journey to tackle an operate-/service-first approach, we'd like to verify the new collector and any limitations internally first before we go and make it generally available. That will help us to remove and understand any problems first which we could immediately build into the GA release and avoid disruptions as much as we can.

Alternatives

Kafka either decides to use their own home-grown "agent" solution (e.g. promtail) or our current Fluentd solution with the Loki forwarding support.

Acceptance Criteria

Risk and Assumptions

Documentation Considerations

Open Questions

Additional Notes

• Kafka metadata requirements (see here) seem to only require namespace_name, pod_name, and container_name. Nevertheless, we also discussed at least identifying the host/cluster this comes from so that they can potentially connect to that and do more troubleshooting.