Epic Goal*

Add a user-facing API for controlling the verbosity of the Cluster Version Operator (CVO) logs and subsequently audit and categorize the current CVO logs according to the new API to reduce the amount of logs in the normal (default) log level.

Why is this important? (mandatory)

The CVO is a critical core component in an OpenShift cluster. However, there is currently no way to easily change the verbosity of the CVO logs in a live cluster. This results in the need for the CVO to log even necessary important debugging information in the default verbosity level to allow for debugging of issues. This can heavily impact storage and does not meet user expectations, especially in resource constraint environments.

It would be useful to provide functionality for the cluster administrators and OpenShift engineers to easily modify the log level to a desired level using an API similarly as can be done for some other operators. By doing so, we can then categorize better the CVO logs and allow for the expected number of logs in a normal (default) log level. Improving the storage utilization, meeting user expectations, while still providing a way to debug a critical component.
 
Scenarios (mandatory) 

Provide details for user scenarios including actions to be performed, platform specifications, and user personas.  

1. The cluster administrator notices an issue in the cluster and chooses to troubleshoot the issue.
2. The cluster administrator, after some troubleshooting, notices that the logs of the Cluster Version Operator (CVO) might help.
3. The cluster administrator notices that the logs are not detailed enough to troubleshoot the issue.
4. The cluster administrator raises the log level from the default value to a more verbose level by simply modifying the new ClusterVersionOperator resource named cluster via the web console or by patching the resource by using the CLI.
5. The cluster administrator fixes the issue in the cluster.
6. The cluster administrator notices that the CVO outputs too many logs for the administrator's liking.
7. The cluster administrator lowers the log level of the CVO to the lowest level, the default level.
8. The cluster administrator is now a happy cluster administrator.

Dependencies (internal and external) (mandatory)

Code reviews and possible assistance by the HyperShift team will be required while implementing relevant changes for hosted CVOs, as a hosted CVO is deployed in a management cluster and is managed by a control plane operator. HyperShift API changes will be needed to enable the functionality.

Contributing Teams(and contacts) (mandatory) 

• Development - Over The Air Updates (OTA) team
• Documentation - Over The Air Updates (OTA) team (main contact rhn-support-cbippley) for standalone OpenShift; 
• QE - Over The Air Updates (OTA) team

Acceptance Criteria (optional)

• Cluster administrators are able to dynamically modify the log level of a CVO in a standalone cluster.
• Hosted cluster administrators are able to dynamically modify the log level of a hosted CVO.
• The number of logs in the normal log level is significantly fewer than currently observed.
• The CVO utilizes the newly introduced log levels as can be expected by the API. 

Drawbacks or Risk (optional)

Potential drawback or risks are: limited audience for the feature.

Done - Checklist (mandatory)

The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.

• CI Testing - Tests are merged and completing successfully
• Documentation - Content development is complete.
• QE - Test scenarios are written and executed successfully.