Goals

• Devise a path to allow Loki Operator to migrate to Loki 4.0

Non-Goals

Motivation

With Loki 4.0 the team is introducing a couple of changes that make it impossible to simply update from Loki 3.x to 4.0. Some of these changes are the following:

1. Moving from row to column storage using dataobj (inspired by Parquet)
2. Kafka/WarpStream to decouple read and write paths
3. New query engine backwards compatible with LogQL but the eventual support to do sql queries

The challenging part is the Kafka dependency on the new architecture. On a meeting about the subject the following approaches were discussed:

1. Prioritization of AutoMQ as a drop-in replacement to WarpStream.
2. Contribution to upstream of a GRPC component that would both shard streams into consumers + provide consumers with a WAL that would allow them to recover from shutdowns.

Prioritization of AutoMQ as a drop-in replacement to WarpStream.

Advantages

1. Closer to the architecture designed for Loki 4.0
2. Customers will also get all the benefits from the new architecture*

Disadvantages

1. Productization of AutoMQ
2. Possibly writing a new operator for it

Contribution to upstream of a GRPC component

Advantages

1. Fewer external dependencies (no need to run Kafka)
2. Possibly lower resource usage

Disadvantages

1. Continued maintainability of the new code path to make sure it provides the necessary APIs to both producers & consumers
2. Customers looking to do analytical queries would possibly still run into performance issues

Alternatives

Acceptance Criteria

A decision has been made on what is solution that we will adopt for Loki Operator

Risk and Assumptions

If we do not adopt Loki 4.0 quickly enough we might run into a situation where our product becomes "outdated" and we are stuck with Loki 3.x.

Documentation Considerations

Open Questions

Additional Notes

• RedHat Internal Design Doc/Write down
• Road to Loki 4.0 Grafana Community Call
• Polarsignals New Database as it deals with similar issues