-
Story
-
Resolution: Done
-
Undefined
-
None
-
None
-
None
-
None
-
False
-
-
False
-
None
-
None
-
None
-
None
-
None
Jobs that make up an aggregated run are already tracked in Sippy. We are
injecting unwanted hysteresis that will cause changes to pass rates to
lag behind. The effect is probably small, but I think it's worth being
correct.
Let's take a simple example:
The follow test runs 10x and passes 7.
[sig-pizza] Pizza should be delivered within 30 minutes
It has a pass rate of 70%, in line with historical results.
If it is aggregated, we have an 11th entry in Sippy of a pass. Sippy
actually thinks the test passed 8 times out of 11 – a pass rate of
72.7%.