Owner: Architect:

pmacik@redhat.com

Story (Required)

As an SBO developer I'd like to run the @upgrade-with-olm tests as PR checks with supported versions of OpenShift on-demand in addition to Minikube.

Background (Required)

Currently, there is a set of tests for upgrading SBO with OLM (scenarios tagged with @upgrade-with-olm) that are executed with Minikube as PR checks for SBO repo via Github Actions. We need to run the same set of tests with supported versions of Openshift, too.

Out of scope

Adding scenarios.

In Scope

Adding new Openshit CI jobs (each for one of the supported version of Openshift (4.9-4.12)).

Approach(Required)

Currently there is a set of acceptance testing jobs already set in Openshift CI that run all the acceptance tests for supported versions of OpenShift as PR checks.

We cannot include the "upgrade" scenarios in the main set of acceptance tests because those assume and require SBO to be installed before running the tests. On the other hand, the "upgrade" scenarios include steps to actually install and upgrade SBO and so require SBO to not be installed, which makes the two sets mutually exclusive.

That is why we need to introduce additional jobs - one for each supported version of Openshift - to run just the "upgrade" scenarios. Each job would execute the acceptance testing framework as in case of the main acceptance tests with the environmental variable set: EXTRA_BEHAVE_ARGS="--tags=@upgrade-with-olm".

Since that will introduce 4 additional PR checking jobs that require provisioning of an Openshift cluster that take ~1hour and will increase chance failures due to glitches in Openshift CI, as well as the additional costs to Red Hat (the Openshift clusters runs on AWS), those jobs would be executed on-demand only - especially before releases - similarily as the performance testing jobs are. Triggering the "upgrade" tests will be done by adding a comment like /test 4.12-acceptance-upgrade to the respective PR.

Demo requirements(Required)

The jobs would not be apparent out-of-the-box so the users/contributors would not be aware of the jobs existence - the point of the demo is to advertise/introduce the new PR checks available to contributors/PR authors.

Demo scenario: The demo introduces new optional PR checks, shows that PR does not trigger the Openshift CI jobs automatically. Then, it is shown to the audience that the jobs can triggered be by adding the respective comments to the PR.

Acceptance Criteria

Development:

• The Openshift CI jobs (for Openshift v4.9, v4.10, v4.11 and v4.12) are set to run the "upgrade" acceptance tests as PR check, triggered on-demand by adding appropriate comments to the PR.

QE:
Documentation: Yes/No (needs-docs|upstream-docs / no-doc)

Upstream: <Inputs/Requirement details: Concept/Procedure>/ Not
Applicable

Downstream: <Inputs/Requirement details: Concept/Procedure>/ Not
Applicable

Release Notes Type: <New Feature/Enhancement/Known Issue/Bug
fix/Breaking change/Deprecated Functionality/Technology Preview>

Legend

Unknown

Verified

Unsatisfied