Goals

• Allow a ClusterLogForwarder to be created in any namespace.
• Allow multiple, isolated, RBAC-protected ClusterLogForwarder instances, so that
  • independent groups can forward their choice of logs to their choice of destinations.
  • separate logging groups cannot interfere with each other's configuration.

Motivation

Use case 1: any cluster where there are two or more admin roles with different logging needs. Examples:

• Customer forwards logs to in-cluster log store and/or external customer locations.
• OSD forwards selected logs for monitoring and debugging to central Red Hat log stores.
• ACS forwards audit logs to security analysis services.

Each of these users needs to control their own log forwarding configuration separately, without being affected by the others.

Use case 2: project owners want to forward logs from their own applications without needing cluster admin permissions.

Acceptance Criteria

I can create 2 log administrator accounts and 2 forwarder instances such that:

• Each can create, view and update it's own logging configuration.
• Neither can view or modify the other's configuration.
• Each instance is unaffected by the configuration or existence of others.
• Total resource use for multiple instances is no worse that a single forwarder configured as the sum of the separate instances.

A project admin can create a NamespaceLogForwarder without cluster admin permissions, and configure it to forward their own application logs

• A project admin cannot forward logs from outside their namespace.

Metric labels

Important: Once it is possible to create multiple forwarders, any metrics connected to a forwarder instance must be labeled with the k8s namespace and name of the CLF object, so metrics from different forwarders can be separated. This should be done automatically by the metric system but verify.

Shared vs. separate collectors

This could be implemented by merging forwarders into a single collector configuration, or by deploying a separate collector instance for each forwarder.
Need to prototype and measure the benefits of each approach to decide.

Benefits of separate collectors:

• Collectors can have separate, different resource limits.
• Easier to generate configuration - each collector config is a separate "namespace", no problems of clash
• Greater concurrency (maybe, single process Vector scales well across CPUs)
• Problems/failures/overload of one collector does not affect others (maybe, they are running on the same node regardless.)

Benefits of single collector:

• More efficient use of memory/CPU resources (maybe, needs to be measured)
• Less change to the operator, more like how it currently works (generate config from multiple resources rather than just one)

Do we also need multiple ClusterLogging instances?

In the logging.openshift.io/v2 API versions, we will drop the ClusterLogging API so this will no longer be relevant.

If this epic is completed before the API changes, then we will keep the CL API until the API changes:

• There will be a ClusterLogging instance for every ClusterLogForwarder instance, with same namespace/name
• Each pair of CL/CLF will operate in the same way as the current singleton does.

Do we also need multiple ClusterLogging instances?

• If not, who controls the global resource limits for logging? 
• If  installing an "add on" like RHCAS requires adjusting the resource limits, who does that and with what permissions.

Should separate instances be allowed/required to use separate namespaces or required to live together in the openshift-logging namespace? Consider convenience and security.

Operator Strategy

The operator must support AllNamespaces install-mode to support namespace-bound forwarders as well as multiple cluster-wide forwarders.