Data cleanup and retention for logging

1. Proposed title of this feature request

Data cleanup and retention for logging

2. What is the nature and description of the request?

With the new ES data model in OpenShift 4.5, we have reduced Operations control over the retained data within the OpenShift logging stack.

Customers leveraged the old data model to cleanup specific indices if there was excessive logging (debug logging, connection issues etc..) and to tune curator (regex) to control the logging retention of different projects (prod / dev/ qe)

We are looking for some enhanced administrative functions that expose the ability to use the delete_by_query API to give customers the ability to selectively cleanup logs based on specific criteria such as a namespace or time period. In addition, customers want the ability to leverage the rollover api's capabilities around max_size / max_docs in conjunction with regex or other suitable pattern matching method to define retention policies on a more granular level in the app index.

3. Why does the customer need this? (List the business requirements here)

Maintenance is critical to overall cluster stability. In multi-tenant environments, operations teams rely on the ability to manage their clusters at more granular level (such a per namespace) so they can guard against bad behaviour and noisy neighbours impacting the cluster. Being able to act upon log retention issues or excessive growth will help customers with adoption and overall cluster stability.

4. List any affected packages or components.

Logging