Goals

• Allow a ClusterLogForwarder to be created in any namespace.
• Allow multiple, isolated, RBAC-protected ClusterLogForwarder instances, so that
  • independent groups can forward their choice of logs to their choice of destinations.
  • separate logging groups cannot interfere with each other's configuration.

Non-Goals

Not a goal to provide SIEM (or other) compliant storage. Assume that sending over TLS is the end of the forwarders security responsibility.

Not a goal to security-harden the operator or operands (although some of that is happening separately from this story)

Motivation

Use case 1: any cluster where there are two or more admin roles with different logging needs. Examples:

• Customer forwards logs to in-cluster log store and/or external customer locations.
• OSD forwards selected logs for monitoring and debugging to central Red Hat log stores.
• ACS forwards audit logs to security analysis services.

Each of these users needs to control their own log forwarding configuration separately, without being affected by the others.

Use case 2: project owners want to forward logs from their own applications without needing cluster admin permissions.

Alternatives

Manage a single logging API instance that forwards to multiple targets.

The problem is merging settings from multiple parties that may not trust each other. Mistakes will be hard to avoid, and deliberate sabotage hard to prevent.

In the past we have discussed use of webhooks to control access to a single CLF but:

• It's not obvious how to secure this in a simple way with RBAC.
• It's not obvious how to deal with clashes between different uses.
• There is no place where users can view their own configuration without confusion from others.

Given 3 or more users, I think the webhook approach will be hard to explain and use, and difficult to implement without bugs or security holes.

Acceptance Criteria

I can create 2 log administrator accounts and 2 forwarder instances such that:

• Each can create, view and update it's own logging configuration.
• Neither can view or modify the other's configuration.
• Each instance is unaffected by the configuration or existence of others.
• Total resource use for multiple instances is no worse that a single forwarder configured as the sum of the separate instances.

A project admin can create a NamespaceLogForwarder without cluster admin permissions, and configure it to forward their own application logs

• A project admin cannot forward logs from outside their namespace.

Risk and Assumptions

Documentation Considerations

https://docs.google.com/document/d/1ffg7wiceax8LJI7--vbt9hshOvFfnTWgNtrM5A1yG38/edit#heading=h.3zykr6j8zwxz

Open Questions

Additional Notes

Metric labels

Important: Once it is possible to create multiple forwarders, any metrics connected to a forwarder instance must be labeled with the k8s namespace and name of the CLF object, so metrics from different forwarders can be separated. This should be done automatically by the metric system but verify.

Testing

This feature will be of great value for the development team, we will finally be able to run parallel tests with different forwarder configurations on a single cluster.

Shared vs. separate collectors

This could be implemented by merging forwarders into a single collector configuration, or by deploying a separate collector instance for each forwarder.
Need to prototype and measure the benefits of each approach to decide.

Benefits of separate collectors:

• Collectors can have separate, different resource limits.
• Easier to generate configuration - each collector config is a separate "namespace", no problems of clash
• Greater concurrency (maybe, single process Vector scales well across CPUs)
• Problems/failures/overload of one collector does not affect others (maybe, they are running on the same node regardless.)

Benefits of single collector:

• More efficient use of memory/CPU resources (maybe, needs to be measured)
• Less change to the operator, more like how it currently works (generate config from multiple resources rather than just one)

Do we also need multiple ClusterLogging instances?

In the logging.openshift.io/v2 API versions, we will drop the ClusterLogging API so this will no longer be relevant.

If this epic is completed before the API changes, then we will keep the CL API until the API changes:

• There will be a ClusterLogging instance for every ClusterLogForwarder instance, with same namespace/name
• Each pair of CL/CLF will operate in the same way as the current singleton does.

Do we also need multiple ClusterLogging instances?

• If not, who controls the global resource limits for logging? 
• If  installing an "add on" like RHCAS requires adjusting the resource limits, who does that and with what permissions.

Should separate instances be allowed/required to use separate namespaces or required to live together in the openshift-logging namespace? Consider convenience and security.

Operator Strategy

The operator must support AllNamespaces install-mode to support namespace-bound forwarders as well as multiple cluster-wide forwarders.