Comment from rhn-engineering-theute 

KesselInventoryApiLatency is definitely wrong:

((sum(rate(server_requests_seconds_bucket_seconds_bucket{le="0.25", container="kessel-inventory-api", namespace="kessel-prod", code!="0"}\\{{window}}))
/ sum(rate(server_requests_seconds_bucket_seconds_bucket{container="kessel-inventory-api", namespace="kessel-prod"}\\{{window}}))))

1) we shouldn't filter on code != 0 (but code isn't used so it's not an issue)
2) should indeed be divided by server_requests_seconds_bucket_seconds_count (or server_requests_seconds_bucket_seconds_bucket{le="+Inf"....)

SLO to fix:

KesselInventoryAPILatency

TODOs:

• Change to SLO docs to make sure they reflect the actual SLI/SLO
• Update our prometheus alerts to reflect that. The prometheus alerts are a way you can test your changes using prometheusruletests
  Stage Alert: LINK
  Stage Alert PrometheusRuleTest: LINK

To test, use : https://gitlab.cee.redhat.com/service/app-interface/-/blob/master/docs/app-sre/prometheus-rules-tests-in-app-interface.md?ref_type=heads#hackrun-prometheus-testspy