Goals

• Enable native LokiStack deployment support in the addon for managing log storage for logs forwarded from an entire RHACM managed fleet of clusters.
• Apply the single RHACM Observability tenancy model over log stored in LokiStack instances across the fleet.
• Apply the same RHACM Obserability RBAC controls over logs stored in LokiStack instances across the fleet.
• Enable native LokiStack deployments on selected clustersets on an RHACM managed fleet of clusters.

Non-Goals

• Static support for a single LokiStack instance per Hub cluster.
• Custom visualization for logs stored in RHACM managed LokiStack instancs.

Motivation

The RHACM Multi Cluster Observability (MCO) mission is to provide an end-to-end observability experience for each supported signal from collection over to storage till visualization. The current RHACM product supports metrics collection (via the endpoint-metrics-operator), Thanos-based storage and Grafana-based visualization the on hub cluster (via the multiclusterobservability-operator). To complete compatibility with this mission the journey started with LOG-4539 offering log forwarding needs to continue with adding LokiStack-based log storage and connect to the MCO's Grafana-visualization.

First of all the following EPIC is dedicated to extend the multi-cluster-observability-addon (MCOA) provisioning capabilities to manage Loki-Operator and LokiStack resources on selected clustersets. The emphasis is on selected clusterset because the log storage should be centralized on a dedicated cluster (not necessarily a hub cluster) for a set of clusters forwarding logs, i.e. collecting and storing logs per region/rack/etc.

In extend the EPIC is also dedicated to provide a design and log storage related implementation for:

1. A mutual multi cluster observability tenancy model (e.g. one tenant per clusterset?!?), i.e. managing per-tenant storage, compaction on LokiStack-based installations.
2. A mutual multi cluster RBAC access model (e.g. managing multi-cluster log access per clusterset), i.e. managing multi-cluster observability authorization on LokiStack-based installations.

Alternatives

N/A.

Acceptance Criteria

1. Given the fleet administrator creates a LokiStack resource on a hub cluster annotated with a list of clusterset names when the addon is provisioned on that hub cluster then the it will provision the LokiStack resource on a dedicated cluster (labeled for MCO logs storage) and configure each clusterset as a tenant.
2. Given the fleet administrator updates a ClusterLogForwarder resource to forward logs to a LokiStack installation when the addon is provisioned on that hub cluster then the provision a ClusterLogForwarder resource and a TLS client certificate to forward logs to that LokiStack instance using the clusterset as a tenant.
3. Given the fleet administrator provides a RoleBinding to a user or groups of users to access the logs of one or a set of clustersets when the users access the logs from Grafana then they will be able to access only the logs of the permitted clustersets.

Risk and Assumptions

TBD

Documentation Considerations

N/A

Open Questions

N/A

Additional Notes

1. Multi-Cluster Observability RBAC Dev-Preview: https://github.com/stolostron/multicluster-observability-operator/tree/dev-preview-fine-grain-rbac/dev-previews/fine-grain-rbac
2.