Epic Goal

Provide a way for users to configure more environment variables or command line flags that impact how the policy controllers run. For components on managed clusters, the solution should easily apply to a single cluster, or multiple clusters. The controllers on the hub should also have options.

Why is this important?

There are a number of flags and options exposed in our controllers, but only a few of them are currently exposed to users: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.14/html-single/governance/index#policy-controller-advanced-config . For example, the logging level is configurable through an annotation on a ManagedClusterAddOn, but the "evaluation-backoff" option on the config-policy-controller is not exposed in a similar mechanism. If someone needed to adjust these other settings, they would need to do it very manually, and in a potentially unsupported way.

If we want to have feature flags for things, or tune some of these options in the future, we need either a blueprint for exposing and documenting them, or a flexible enough system that enables users to change more things without us explicitly exposing them.

Scenarios

1. As a policy admin, I want to tune some concurrency and evaluation speed options on the config-policy-controller for my different environments. I want to be able to apply these changes to all my test clusters before making the changes to my production clusters.
2. As a policy developer, I want to easily expose the new feature I've written to users in a configurable way.

Acceptance Criteria

• The policy components on both the hub and the managed clusters should be configurable.
• Configuration options may have different types - numbers (evaluation concurrency), strings (a default mode), and maybe lists or objects (for things like https://issues.redhat.com/browse/ACM-23770). 
• We must have automated tests that verify the configurations can be made and take effect quickly and consistently.
• The feature must be well-documented in a way that can accommodate additional options in the future.

Dependencies (internal and external)

None? Maybe the addon-framework and the installer, although they likely already have mechanisms we can utilize.

Previous Work (Optional):

1. https://issues.redhat.com/browse/ACM-17563 

Open questions:

1. What will happen if a user provides an invalid configuration? For example, providing a string when a number is needed, or when multiple configurations work together in a way that requires them to match some constraints (eg option A might require option B also be on). How would we report this to the user?
2. Should we have a way to configure the default options for managed cluster components somewhere centrally on the hub?

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub
  Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub
  Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Doc issue opened with a completed template. Separate doc issue
  opened for any deprecation, removal, or any current known
  issue/troubleshooting removal from the doc, if applicable.
• Considerations were made for Extended Update Support (EUS)