Owner: Architect:

Martin Mulholland

Story (Required)

As an OpenShift user I want the chart certification process to work consistently free of bugs. 

Background (Required)

Work is underway to add a report digest to a report which can be used to detect when a report is modified. The current workflow test suite is based on using a single report which is them modified to create a report which can be used for a test. Tests using this technique will all start failing once the report includes a sha value.   

Glossary

Report digest - digest value of a report which is then added to a report. When the workflow reads a report it check the digest matched the report content. 

Out of scope

Adding new tests.

Automation required to regenerate report files.

In Scope

Modification of test framework and existing tests to match the framework.

Approach(Required)

Every test will need its own unique report file, which is generated by the chart verifier to include a digest. One approach is to pre-create the reports and add, for example:

development/tests/data/HC-16/report_sha_bad/report.yaml

Another approach is to generate the report as part of the test which will need the test to specify the parameters required to use for the verify command.

Some tests (for example chart uri tests) require a modified report because it will not be possible to generate a report using the uri required. Two possibilities here:

 1. Remove tests which require a modified report as not needed.

 2. Locally modify the chart verifier report command code to output the digest value of the modified the report and manually add it.

Also, any test which needs a pre-generated repot file should include a file which specifies the command required to generate the report. Possibly in json format which will enable future automation to re-create reports when chart-verifier changes.   

Dependencies

Chart-verifier version which includes the report-sha. For testing this may need a chart verifier image created specifically for the testing.

Edge Case

If we start generating reports as part of the test we may need to revisit the content of the smoke test dues to the added time needed.

Acceptance Criteria

Passing test suite when using a version of chart-verifier which adds and checks a report digest value. 

Documented method for creating each report fie required.

INVEST Checklist

Dependencies identified
Blockers noted and expected delivery timelines set
Design is implementable
Acceptance criteria agreed upon
Story estimated

Legend

Unknown
Verified
Unsatisfied