Problem description

The LokiStack gateway implements the Observatorium API, which is important for use in Observatorium. The cluster log forwarder is implemented to send logs to the plain Loki API. These APIs are mostly the same, but have differences that must be reconciled for the CLO to send logs to LokiStack:

• Plain Loki API expects the tenant-id in HTTP header X-OrgID.
• Observatorium API tenant-ID to be embedded in the request URL.

There are several ways to address this:

1. Modify LokiStack gateway to implement both the Observatorium API and the basic Loki API by rewriting Loki requests with X-OrgID as Observatorium requests with tenant in URLs.
2. Modify the forwarder and console plugin to use the Observatorium API, and embed the tenant ID in the URL instead of sending in X-OrgID header.
3. Configure the ClusterLogForwarder 3 different outputs, one for each of the "[application, infrastructure, audit] "tenants"
4. Have the collector bypass the the LokiStack gateway and talk directlty to the distributor component for in-cluster logging.

Notes

1  (modify gateway) has the advantage that any client of the plain Loki API can talk to Lokistack without modification, not just the log forwarder and console. This may be useful in future, and probably fits well with the operator being based in grafana. There may be complexities in the implementation caused by having 2 paths to the same data.

2 (moidfy forwader/console) should be straightforward but needs to be scheduled ASAP. It does mean the loki output would have 2 different "modes" since it also needs to forward to plain loki instances, as well as lokistack.

3 (complicated CLF config) is surprisingly different from what users are used to, it is complex, error prone, and won't work for (possible future) tenant schemes with a variable number of tenants.

4 (bypass gateway) discards some of the benefits of LokiStack. The gateway can still provide security for users reading logs. The forwarder would need suitable credentials to get privileged access bypassing the gateway.

Goal

Choose the option we want to proceed with for logging 5.5.0 release.

Acceptance Criteria

• Create Jira issues to compete the work for the chosen option.
• Mark them as blockers for the 5.5.0 release.