Goals

• Modify, wrap or proxy to LokiStack so that existing clients can forward logs.
  • Using X-Org-ID header for tenant rather than URL path component.
• Enforce tenancy for such clients, the same as for LokiStack API clients.
• Support "dynamic" tenancy where the client need not know tenants in advance
  • Tenants may be registered with LokiStack if necessary ...
  • ... but it must not be necessary to update clients for new tenants
  • For example: mapping label values or namespaces to the tenant

Non-Goals

• Clients using the Loki API will only have access to plain Loki features
  • No access to additional LokiStack APIs that are not part of Loki.
  • But tenancy rules will be enforced.

Motivation

The Loki and LokiStack APIs are slightly different: Loki passes the tenant in the X-Org-ID HTTP header, LokiStack includes the tenant as a component in the URL path.

This means existing Loki clients (vector, fluetnd, logcli) which can forward logs for multiple tenants to Loki, cannot easily do so for LokiStack.

Alternatives

Create a new "lokistack" output type at the forwarder to handle the new API:

• Update existing clients upstream (at least Fluentd and Vector)
• Write new clients specifically for LokiStack, integrate them into the forwarder
• Deploy a URL-rewriting proxy as part of the forwarder (see notes below)

Acceptance Criteria

• Existing Loki clients - Fluentd, Vector and the CLF - can talk to LokiStack
• Tenancy rules are enforced as for native LokiStack clients
• The CLF can apply arbitrary log record key values as tenant
• CLF forwarding succeeds if the tenant is valid according to LokiStack.
• If the tenant is not valid the forwarder must:
  • log a useful error message
  • do not retry to send the failed record
  • continue to forward logs with valid tenants

Risk and Assumptions

• Complicates Lokistack with multiple paths to the same APIs.
  • May be mitigated by using a separate proxy?
• No access to additional LokiStack APIs
  • We may need native LokiStack clients in future if the forwarder needs these features.

Documentation Considerations

Need doc, should be a simplification of existing doc.

Current solution is to create 3 outputs, one each for application, audit and infrastructure lgos. The new solution will be a single output which applies the `log_type` field value as the tenant.

Open Questions

Not 100% clear yet if we should modify LokiStack, or deal with this on the client side.

The client side seems more complex, and solving it on the LokiStack side opens up access to LokiStack from a wider range of clients - not just Fluentd + Vector but also Loki clients in other languages. This would help adoption of LokiStack.

Additional Notes

This is an nginx configuration that rewrites Loki requests in LokiStack form. This is not a complete solution, it only shows rewriting URLs. There are security issues that are not covered here:

error_log stderr info;
pid nginx-loki.pid;
working_directory .;

events {
}

http {
  access_log access.log;
  client_body_temp_path ./tmp/client_body;
  proxy_temp_path ./tmp/proxy;
  fastcgi_temp_path ./tmp/fastcgi;
  uwsgi_temp_path ./tmp/uwsgi;
  scgi_temp_path ./tmp/scgi;  

  server {
    listen       8080;
    server_name  _;

    # This is the rewriting rule:
    location /loki/api/v1/ {
      rewrite ^/loki/api/v1/(.*)$ /api/v1/logs/$http_x_org_id/$1;
      proxy_pass      http://localhost:8080/; // target URL goes here
    }

    # serve static files for test purposes.
    location / { 
      root html;
    }
  }
}