Per project logging observability

1. Proposed title of this feature request

Per project logging observability

2. What is the nature and description of the request?

With the new ES data model in OpenShift 4.5, we have reduced Operations visibility into the OpenShift logging stack.

Customers leveraged the old data model to observe logging volumes on a per project basis. This helped with capacity planning and sizing of storage requirements

We are looking to enhance the exposed metrics with aggregation queries that provide insights into logging volumes on a per project basis. Additionally, we should expose individual metrics on disk usage and growth of the 3 primary indexes. For these metrics, we should allow for alerting on both index and per project logging thresholds within the retention period.

3. Why does the customer need this? (List the business requirements here)

Observavility is critical to overall cluster stability. In multi-tenant environments, operations teams rely on the ability to observe and manage their clusters at more granular level (such a per namespace) so they can alert and guard against bad behaviour and noisy neighbours impacting the cluster. Being able to accurately monitor and alert on log retention issues will help customers with adoption, growth and overall cluster stability over the long term.

4. List any affected packages or components.

Logging