In https://github.com/openshift/sippy/pull/441 I attempted to add a new API to sippy that would analyze the requested number of most recent payloads, find the tests that fail most often, and attempt to determine if a particular test looks like it's permablocking the payload.

End goal would be to fire alerts that would tell us precisely that something is wrong and what it is.

This needs some work to improve, specifically:

• calculate a score of how likely it is to be a blocker vs just something that failed a lot lately
• heavily weight tests that fail in the same job over multiple rejected payloads in a row
• heavily weight tests that fail in higher numbers of payloads (2+), by 4 we are near certain.
• more lightly weight tests that fail in different jobs over multiple rejected payloads
• skip handle failed payloads (ART problems usually, as opposed to rejected)