Goals

• Enable fast-retrieval of a OpenTelemetry metadata regarding signal co-relation across logs.
• Provide first-class ingestion for OTEL-format metadata.

Non-Goals

• Provide OTEL Protocol or similar native APIs in Loki

Motivation

As the prevalence of OpenTelemetry (OTEL) formats across the industry making them the de-facto standard, we need to support queries over these (regardless cardinality) in an efficient way. As per now Loki's/LokiStack's capabilities to index log lines is limited to low-cardinality metadata called the labels which cover the need to partition the search space efficiently (e.g. namespaces, clusters ids, pod names, environments). In addition to such identification-related labels, workloads and infrastructure emitting OTEL logs add another set of characteristic metadata about payload processing (e.g. request ids, trace ids, user ids, etc.). Such metadata have usually a high cardinality and are bound to workload/infrastructure usage patterns. Thus they do not qualify for Loki/LokiStack labels as they would impair Loki's search space partitioning severely and in turn overall performance.

Moving forward Loki/LokiStack >v2.9.0 included expanded capabilities to store efficiently metadata extracted from structured log payloads (e.g. JSON, Logfmt) and store them aside the log lines  (e.g. request ids, trace ids, user ids, etc.). This feature is called structured metadata indexing and enables fast query retrieval without imposing the cost of parsing the structured logs.

Alternatives

• Extended Index Query and Result Caching: Introducing the ability to use several supported external caching systems (e.g. Memcached, Redis) beyond the embedded cache for index query results (i.e. results of label sets to chunk IDs) and query results (i.e. fully parsed log results).

Acceptance Criteria

1. The Cluster Logging Administrator can seamlessly switch to storing OTEL-formatted log lines from any workload and infrastructure without specific LokiStack configuration.
2. The LokiStack administrator can have an overview of chunk utilization and structured metadata index usage by inspecting the LokiStack dashboards.

Risk and Assumptions

1. The Loki structured metadata index APIs and configuration settings are marked as experimental. Breaking changes on APIs as well as the chunk format should be expected.
2. The current LokiStack index configuration defaults to boltb-shipper used with schema V11. This schema version and index store implementation are not eligible to enable the structured metadata index in Loki. Users are required to define (an additional) schema namely V13 and use TSDBv3 as the index store implementation. Therefore only new incoming logs can be considered when using the structured metadata feature.

Documentation Considerations

• The schema migration to V11 to V13 and the store migration from boltdb-shipper to TSDBv3 is a required user task. The current API lokistack.spec.storage.schemas supports schema migrations but not store migrations yet. This will be available with accomplishing this EPIC and requires detailed documentation as per risk to make previous stored logs based upon schema version V11 and boltdb-shipper inaccessible.

Open Questions

Additional Notes

1. Official Loki Documentation: Structured Metadata
2. Open Telemetry Logs Format Specification and Examples
3. Loki v2.9.0 - Pinning chunk and index format to schema version