There are a number of jobs performing tests that are expected to very rarely break, but provide important coverage we don't want to miss in the context of "are we ready to release or not".

Examples include:

• QE jobs
• etcd scaling job
• cpu partitioning
• kube-apiserver repeated rollout (new)
• exotic install methods

The ask is for these jobs to be able to turn component readiness cells red, even though they're run rarely, once a week, maybe less.

We're essentially looking for raw pass rate comparison here, on each test. However, our current component readiness sample/basis time windows are not going to work, we need to look back further, perhaps to a fixed number of runs. (last 10 has been floated) Then if any test in those last X runs is below say 90% pass rate, consider it a regression.

Design doc: https://docs.google.com/document/d/1Qm9Yry8T0ILQARRVnMEMNYu8EEPdxc_FUewiRaLsyNg/edit?usp=sharing

We need to have a plan for clearing a regression on such a long timeframe, 10 runs on a weekly job spans 2.5 months, we cannot wait that long for data to roll off.