The flaky tests may actually have bugs in the alerts expressions, Since we see that the alerts are failing after several reruns and when the env is already "dirty" with data.

It could be that all these failures are due to many-to-may error in the alert expressions.
We need to:
1. Crete a list of the flaky tests
2. See if there is a way to catch errors in the alert expressions in the tests.
3. Check the existing flaky tests for many-to-many issue.
4. See how we can automate the unit tests to generate metrics with different labels to mimik possible many-to-many issue.