Goal: implement a more efficient test re-running strategy using the CANNIER method

The CANNIER method describes a method to make the re-running of tests more efficient using a pre-categorization of tests with a mixture of static analysis and other data into three categories: probable flaky, probable non-flaky and uncertain. Only re-running the tests in the uncertain category cuts down on total number of test executions.

User Stories

As a CI maintainer I want to have a model that I can feed the CANNIER feature set and receive a prediction value, so that I can use that value for decision of rerunning a certain test

• implement the data retrieval to gather the data from the test code
• feed the model with the existing data
• deploy the model with a service into our workloads cluster

As a PR author I want to get feedback on the test code I produced, so that I can get information about which tests are likely flaky and where I can improve them

Outline of a job run:

• evaluates the flakyness probability of the set of tests having been updated or created
• then re-runs those tests
• attaches a comment to the PR that gives the PR author feedback on test flakyness

As a CI maintainer I want to update the CANNIER model in regular intervals so that advice on the test cases will be current

Outline:

• regularly rerun the model creation
• replace the model with an updated version

Notes

Sources:

• 2023 paper by Kapfhammer et al