Proposed title of this feature request:

Customer-defined platform-wide alerting rules

What is the nature and description of the request?:

By default, platform-alerts are routed to our SRE team, as it is expected. When customers want to create custom alerts, they would do so using the user-workload monitoring. This is also described in our Black Belt page: https://mobb.ninja/docs/rosa/custom-alertmanager/#what-about-cluster-alerts-

However, these custom alerts are limited to a namespace. There is currently no way for customers to create platform-wide alerting rules that can then be forwarded to the customers monitoring / AlertManager. An example for an alerting rule that a customer might want is shown here: https://redhat-internal.slack.com/archives/C021EMWAQ76/p1693920903266459?thread_ts=1693920878.112669&cid=C021EMWAQ76

This request is asking to provide a possibility for customers to create platform-wide alerting rules that are then forwarded to the customers AlertManager instance.

This request has been discussed before:

https://redhat-internal.slack.com/archives/C021EMWAQ76/p1693920878112669
https://redhat-internal.slack.com/archives/C021EMWAQ76/p1689441630527479
One conversation mentions that https://issues.redhat.com/browse/MON-2849 may already provide a partial (or full) solution?

Why does the customer need this? (List the business requirements here):

Customers can do this on their self-managed clusters, but not on ROSA. This may block customers from adopting ROSA as they are unable to create certain security-relevant alerting rules (such as PSA alerting / enforcement) they require.

List any affected packages or components:

OpenShift Monitoring