Background

In order for ACM to reuse the monitoring plugin, the plugin needs to connect to a different alert manager. It needs to also contain a new column in alerts to show the source cluster these alerts are generated from

use the #forum-acm-observability channel for questions

Check the ACM documentation around alerts for reference: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.11/html/observability/observing-environments-intro#observability-arch

Outcomes

• The monitoring plugin can discover alert managers present in the cluster
• If multiple alert managers are discovered, the plugin should display a dropdown to select an alert manager to connect to, if no alert manager is discovered the plugin should fallback to the incluster one
• The monitoring plugin can connect to a specific alert manager, to create silences
• The monitoring plugin can connect to a specific prometheus rules endpoint, to read alerts
• Add a new column that can display from which cluster the alert is coming from

Steps

• Use the k8s api to Discover alert managers present in the cluster. A Role for the plugin needs to be created manually so it can read alert managers
• Create an API `/v1/api/alert-managers` that can list the results of the discovery. Return the alert manager name and the service associated with it for the frontend to connect
• Create a proxy to connect to a specific alert manager discovered by the plugin backend. `/proxy/alert-managers/{alert-manager-name}`