Labeling and/or tagging tests for filtering purposes

Epic Goal

• This epic has been created with the idea of implementing a new labeling or tagging system that allows the different teams to select and filter which tests they want to run in the different stages of the builds.
• Right now is focusing on Operator-E2E repo, but as far as new suites would be included in the Acceptance Test (Kam, DevConsole...), we may need to think of something that not only doesn't break the current automation but also that could be extended to different test suites under the same criteria. A solution based on a directory-tree naming convention could be the best suit since you can access the test folders by REGEX, and almost every language and framework can deal with this structure.

Why is this important?

1. We need a solution in place to easily select which features we want to test. Doesn't make any sense to run a whole suit (up to 2h), for instance, in every git push within the CI system. Allowing each team to pick up the necessary tests would optimize the testing efforts.
2. Right now there is no way to identify which test corresponds with which component, and let alone group them. Letting the team quickly identify where to find where looking for some specific thing would also enhance our general performance.

Acceptance Criteria

• The GitOps team implements a system to filter and select which tests to execute based on:
  • Component (ArgoCD, Dex, Operator Controller...)
  • Feature (App Syncing, Authentication, Git connectivity...)
  • The gitops variant (GitOps Product, GitOps Managed Service, GitOps Core) - Not sure about this, we may discuss it first.
• All Kuttl tests used in Acceptance Pipeline are properly labeled and are identifiable.
• We have a Make target in the operator-e2e repository to run specific tests based on the mentioned system.

Previous Work:

• You can take a look at the current repo holding the tests here.

Open questions::

• Makes it sense to apply the same to Ginkgo tests as well?
• Shall we maintain the tests in two different folders based on if they are intended to be run parallel or sequentially? Or either we could put "seq" or "par" as another label, so we can group tests by component instead?

Please, feel free to add comments and suggestions.

Thanks in advance.