-
Spike
-
Resolution: Unresolved
-
Normal
-
None
-
None
-
None
-
False
-
-
False
USER STORY
As a certification team member, I want a baseline of certification test results for X version of OpenShift so that I can compare to real partner test results.
DESCRIPTION
OpenShift certification tool is currently run periodically on the latest released version of OpenShift. These results are stored temporarily (medium term) in Prow CI. However, these results are kept inside of an archive that isn't quickly or easily parseable; similar issue with must-gather.
A summarized report of the periodic job executions would be very helpful to certification team members.{}{}
Consider a filterable report that displays the following:
- Date of job execution
- OpenShift version + platform
- OPCT version
- Overall success of test (did OPCT fail out or not)
- Count of successful/failed/skipped tests
- Ability to zoom into specific tests that succeeded/failed/skipped
- Difference in test results between job executions
- Consistent failures between job executions
- which else?
Metrics we expected on the results for each execution:
- Total counters[1] for each result by plugin: Total/Passed/Skipped/Failed[...]
- Failure ratio (counter of the operation failed/total) by plugin
- Failure ratio after applying filters (recently added 'report' command)
- Total time to create the cluster*, run the tool, destroy the cluster*
*not mandatory, important to have
ACCEPTANCE CRITERIA
| OPCT mode | OCP Topology | OCP Version | OPCT Run time | Conf. OCP S/T/P/F/S* | Conf. K8S S/T/P/F/S* |
|---|---|---|---|---|---|
| Regular | HA | 4.9.? | |||
| Regular | HA | 4.10.? | |||
| Regular | HA | 4.11.? | |||
| Regular | HA | 4.12.? | |||
| Upgrade | HA | 4.10.?->4.11.? | |||
| Upgrade | HA | 4.11.?->4.12.? |
- Nice to have - timers:
| OPCT mode | OCP Topology | OCP Version | Total Time | Cluster Install Time | OPCT Exec Time | Cluster Destroy Time |
|---|---|---|---|---|---|---|
| Regular | HA | 4.9.? | ||||
| Regular | HA | 4.10.? | ||||
| Regular | HA | 4.11.? | ||||
| Regular | HA | 4.12.? | ||||
| Upgrade | HA | 4.10.?->4.11.? | ||||
| Upgrade | HA | 4.11.?->4.12.? |
ENGINEERING DETAILS
[1] Previous work / examples:
[2] WIP automation to automatically create AWS cluster [agnostic], run OPCT, and destroy: https://github.com/redhat-openshift-ecosystem/provider-certification-tool/pull/38
1.
|
[CI] collect insights from the CI periodic jobs |
|
To Do | 
|
Unassigned |