-
Task
-
Resolution: Unresolved
-
Major
-
None
-
None
-
None
-
False
-
None
-
False
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
-
The functional tests [1] that we run as part of the Container Verification Pipeline (CVP) (which is triggered after every regular Brew build), are not really useful any more:
- They deploy from an openshift template, which is not a supported installation method any more.
- They were never designed to work on Z / P, and are often failing.
- They have too broad a scope, as they deploy all of 3scale. Instead, they should have a more limited, per-container-image, scope: rather than deploy all of 3scale, the tests should just check the individual container image, because the QE Group Tests take care of 3scale-wide deployments. For some components, there is already a `make smoke` target (e.g. [2]) that we used to run as part of the old, pre-cpaas 3scale productization pipeline, so perhaps we could re-use those.
After discussing this issue with mjaros@redhat.com , we concluded that we can sunset these particular tests. We can then replace them with some new tests (perhaps based on the `make smoke` targets, where available) that focus on just running the individual container image on the respective platform the test is running on and check whatever else is possible/feasible to test on the container-image level. Input from the app maintainers will be required on that front. Perhaps some checks from community CI can be used as well.