Description of problem:

The RHODS Prometheus alerts that send PagerDuty alerts to MT-SRe are calculated for an SLO of 98%, as that was the SLO defined for RHODS Field Trials

• SLOs-haproxy_backend_http_responses_total
• SLOs-probe_success

As the Service Level Objective is 99.95% for RHODS LA, I think these alerts should be updated

Note: I think those rules were generated using this online tool https://promtools.dev/alerts/errors

Prerequisites (if any, like setup, operators/versions):

Steps to Reproduce

1.  Login to RHODS Prometheus
2.  Go to Status > Rules
3. Verify the expression for SLOs-haproxy_backend_http_responses_total and SLOs-probe_success

Build Details:

RHODS 1.10.0

Live Build:

quay.io/lferrnan/rhods-operator-live-catalog:1.11.1-prometheus-slo 

PR:

https://github.com/red-hat-data-services/odh-deployer/pull/232