Loki tenant configuration invalid for fluentd output plugin used

The ClusterLogForwarder CRD support setting a tenantKey in the loki output configuration, which resolves to the tenant configuration key for the fluent-plugin-grafana-loki output plugin when the fluentd collector is used.

The point of specifying a key instead of a static value is to allow having a single configuration for multiple tenants (for example "audit", "application" and "infrastructure").

The version of the fluent-plugin-grafana-loki plugin currently bundled with cluster-logging-operator is 1.2.7 which does not have the ability to replace placeholders in the tenant configuration option yet (see here). The ability to use placeholders was added in version 1.2.12 (PR).

This results in the placeholder that is generated by cluster-logging-operator to show up literally in the tenant ID used for the Loki output. For example using tenantKey: log_type currently results in the tenant ID ${record.dig("log_type")}, which is not desirable.

As a workaround for the older fluentd plugin versions, but also because it might be desirable for a user to set a static tenant configuration anyway, I would suggest adding another key to the Loki configuration struct that allows the user to define a static tenant, for example called Tenant in addition to the current TenantKey.