We got report from teams their request is having high latency. After investigation, it is due to cert auth request to BOP which calls findOwner endpoint in IT.

The issue has been there 6 hours before we notice, we should have a latency alert for ourself. Currently, the overall latency does not help because the cert auth request is just a small fraction of the huge number of total requests.

Here is where we wanna add the alert:
https://gitlab.cee.redhat.com/service/app-interface/-/blob/master/resources/insights-prod/3scale-prod/3scale.prometheusrules.yaml?ref_type=heads
And here is an example:
https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/107379