We got this report from the MT-SRE team regarding the failing image builds alerts:

This alert's severity is marked as "critical" in its basic details but if you take a look at the severity details under "custom details" section, you'd notice that there, its severity is mentioned as "warning" and in the SOP, its severity is marked as "Minor". We (MTSRE) would love to get some insights upon this incoherence i.e. is this alert a critical alert or just a warning, and if it's just a warning, then should we (MTSRE) get paged about these kinds of "warning" alerts

We should audit the alerting rules, the corresponding Pagerduty severity for alerts, and our SOPs documentation to ensure that alert severity is in alignment across all of these locations.

[UPDATE] 

We’ve updated all the alerts that go to the MTSRE team to critical, and kept the severity of the rest.

PRs

• https://github.com/red-hat-data-services/odh-deployer/pull/213
• https://gitlab.cee.redhat.com/service/managed-tenants-sops/-/merge_requests/30