{}USER STORY:{}

As as Partner, I would like to be able to troubleshoot failed tests by Name and link to the documentation guiding the way/options to fix the issue, so I can resolve the problems before submitting the results to Red Hat.

{}DESCRIPTION:{}

This idea of create a helper document was discussed previously with the sonobuoy community[1]. The community shared a post-processor feature that could help the tool to hook the results processor to customize (link to existing documents) when some pattern has been found, one example is 'when failed tests have been found'. The post-processor feature is evaluated/tested by the team on SPLAT-623 .

The idea of this card is to create the helper documentation, and create the process to continuous feeding this document based on the Partner Review process - mainly for failures find on their environment (lessons learned).

The kube certification has a good practice to document e2e tests by code comments, then a document is created for each release. For each section there is a link for the "Test Name" and the "Defined in code as". It's very good standard and helpful to anyone who would like to troubleshoot the test failures:

https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.25.md

It's not a goal do document ALL tests as KubeConformance does. It was discussed on the Slack thread[2] and can be tracked by another card.

Goal:

• Create a document indexing by failed test names
• reference a sort of helper document (OPCT, OCP, KCS, etc) that can help to resolve the problem

Not Goal:

• Implement automation to link to this document (it is covered on SPLAT-623 )
• It's not a goal to create the index and document ALL tests on the suite for a given release. This card intent to document failures find on the partners linking to KBs (OPCT and OCP docs, KCS, etc)

Example of Documentation could be something like that:

# e2e Tests

## <Name>

- <test_id : as defined on the suite>

Short Description: 
<description>

Know failures:

1) short failure reason
<short code snippet>

References:
- References for a documentation (OPCT, OCP, KCS...)

{}Required:{}

• Initial documentation with the first partner failures
• the process to continue feeding this documentation

{}Nice to have:{}

...

{}ACCEPTANCE CRITERIA:{}

• OPCT failures with lessons learned from the partner evaluated the preview release should be documented as the first delivery

{}ENGINEERING DETAILS:{}

 [1] Discussion with Sonobuoy community about the post-processor feature. https://kubernetes.slack.com/archives/C6L3G051C/p1653506433016089?thread_ts=1651669470.529919&cid=C6L3G051C 

 [2] Slack thread discussing ideas to document OCP conformance e2e tests: https://coreos.slack.com/archives/C02DSMAP4LA/p1669134489097869