-
Story
-
Resolution: Done
-
Undefined
-
None
-
None
-
None
-
False
-
None
-
False
-
-
openshift-tests-extension should support receiving environmental flags from origin (–platform, --arch etc) according to the design in https://github.com/openshift/enhancements/pull/1676. See the "List Tests - Extension Test Listing" section.
Environment flags could be grouped together and placed in an implementation in https://github.com/openshift-eng/openshift-tests-extension/tree/main/pkg/flags like other CLI flags.
Acceptance criteria:
- Flags get version information attached (e.g. v1.0 of OTE supports X, Y, Z flags, Q was added in v1.1).
- Tests are filtered according to the environmental flags passed in `list tests`
The second part needs some design work. Currently tests get skipped based on annotations in the name, and it's handled on the origin side. This means shifting test selection to the external binary (it decides what it wants to run in a particular environment).
We may want to provide a way that uses the existing annotations method inside the OTE binary, i.e. examine the test names to filter, but we also need a secondary way that doesn't involve changing the test name (which affects a lot of our tooling) that is the future way to do things.
The enhancement starts to call out a way this is done, e.g.
// Here, OpenShift convention was used to determine which tests to run on AWS// As a contrived example, let's say we want to run these same tests on GovCloudawsTests := ginkgoExtensionSpecs.Select(extensions.PlatformSelector(platforms.AWS))awsTests.AddPlatform(platforms.GovCloud)
`Select` is useful to filter a group of tests, but I am not sure about the rest of the implementation.
Alternative to be discussed:
One way to do this might be to include this information in the ExtensionTestSpec with a CEL expr:
type ExtensionTestSpec struct { Name string [...] // EnvironmentSelector defines the inclusion and exclusion rules for this test. EnvironmentSelector EnvironmentSelector `json:"environmentSelector"` }
type EnvironmentSelector struct { // Include specifies the CEL expression that must evaluate to true for the test to be considered. // If Exclude also evaluates to true, the test will be excluded regardless of Include. Include string `json:"include,omitempty"` // Exclude specifies the CEL expression that, if true, will override Include and exclude the test. Exclude string `json:"exclude,omitempty"` }
Example test specs
name: "TestA" environmentSelector: exclude: "topology == 'single'"
name: "TestB" environmentSelector: exclude: "network_stack == 'IPv6'"
name: "TestC" environmentSelector: include: "network == 'ovn' && platform == 'aws'"