Summary

Once this work is completed, we would like to have a mechanism in place to ensure that the Default Catalog Sources distributed in OCP are built in a consistent and reliable way. This consistency is important to prevent poor user experiences, avoid technical issues, and reduce the number of support cases.

Success Criteria

• Infrastructure and validation tests are running automatically as part of the workflow
• Mechanism allows for early detection of issues, inconsistencies, or regressions
• A clear and comprehensive set of validations is defined and implemented for this first stage

Expected Results (what, how, when)

What: Reduce the number of support cases related to inconsistencies or regressions in the Default Catalog Sources
When: The automated mechanism should be in place from the moment it detects known issues and resolves them within the pipelines. Each new test should target a specific, known failure scenario to prevent regressions.
Measurement:

• Track support tickets related to catalog inconsistencies and regressions
• Ensure no recurrence of these issues in future support cases for what has been validated

What: Introduce a new mechanism that enables us to define and implement automated validations and tests for Default Catalog Sources
When: When the system is capable of automatically notifying us about detected regressions or issues during the pipeline execution
Measurement: 

• Automatic alerts or reports are triggered when inconsistencies or regressions are introduced
• Successful detection of known problematic scenarios via the new mechanism