Epic: Start tracking code coverage in OpenShift GitOps

Epic Goal

• Be able to see the coverage percentage of our unit tests in each of the repos involved in OpenShift GitOps
• Changes in coverage are trackable over time

Why is this important?

Code coverage for the sake of a high percentage isn’t important. The benefits our team gets from being able to see our current coverage, track changes over time, and having a target coverage percentage are many. 

• We can start adding unit test coverage as acceptance criteria in our stories - this will ensure that new code is well tested and understood
• To increase our coverage to meet a target we would add more unit tests to our current test suite which touch more parts of the code - this would increase our understanding of the existing codebase
• With a deeper understanding of our codebase we’ll move towards the outcomes we defined in our Developer Engineering goals for FY23 FY23-Goals-Outerloop
  • GitOps team members are recognized as SMEs internally and externally
  • GitOps team members feel confident in their ability to manage incidents and empowered to make changes to mitigate recurring problems

• Another of the goals from the document above was to reduce regression bugs. With more test coverage we build a better safety net for ourselves when introducing new features or changing existing ones. With fewer regression bugs we’ll be able to directly impact another of the outcomes we defined for this year:
  • Increase customer confidence in our releases 

Acceptance Criteria (Mandatory)

• CI - MUST be running successfully with tests automated
• Codecov is installed in the relevant repos
• We have somewhere that we can track changes in coverage over time (spreadsheet, dashboard, etc.)

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. KAM repo already has Codecov enabled
2. The GitOps Service team already uses Codecov
3. The RHTAP QE group tracks coverage over time here: Stonesoup Quality Status
4. Related Jiras: GITOPS-2239

Open questions:

1. Do we want to block merges when codecov is reduced? RHTAP has a goal for all teams to do this: https://issues.redhat.com/browse/GITOPSRVCE-372 

Done Checklist

• Acceptance criteria are met