From the query result:
https://grafana.app-sre.devshift.net/d/xcNqJY24zgdgdgd/mk-7-day-alert?orgId=1

We have CanaryProduceConsumeLatency fired 66 times last week.

CanaryProduceConsumeLatency , the premethues rule is checking the P98 of strimzi_canary_records_consumed_latency. Need further investigation why the long latency.

https://github.com/bf2fc6cc711aee1a0c2a/observability-resources-mk/blob/6de8a925afd9b7bc3662cfca958c6c84961abf9b/resources/prometheus/prometheus-rules.yaml#L445