Outcome Overview (aka. Goal Summary)

Extraction of end-to-end tests for plugins from the RHDH core repository into a dedicated repository to enable better separation of concerns, improved test management, and more efficient testing workflows for both the core platform and dynamic plugins. Since this is a large effort that will be covered in stages over a few releases, progress can be tracked under this outcome.  

Adjusted to incorporate the RHDH Test arch document for 1.8: https://docs.google.com/document/d/1UBpmZQ2PJ2FpEyniPDDdGTqpwd0GBAt_8YWqrf2q3CE/edit?tab=t.0#heading=h.lfcobaa82e1

Goals (aka. expected user outcomes)

• dedicated repository for plugin integration tests, decoupled from the RHDH core platform
• Developers will be able to run tests for specific plugins without running the entire test suite
• nightly integration test that runs all tests for all plugins
• Plugin developers will have clear documentation on how to run tests locally against an RHDH instance

Requirements (aka. Acceptance Criteria):

• Create a new repository for plugin integration tests that follows the same structure as the current test setup in the RHDH repository
• Move all plugin-specific end-to-end tests from the RHDH core repository to the new repository
• Configure the new repository to reference dynamic plugin configurations:
  • Initially for plugins shipped with the core RHDH image
  • Future support for OCI image references
• nightly integration test that runs all tests for all plugins
• mechanism to selectively run tests for individual plugins when changes are made to those specific plugin tests
  • probably by checking changes to source paths and by setting environment variables to selectively run tests
• Establish clear separation between different test types:
  • Unit tests remain in plugin repositories
  • Integration tests with external dependencies moved to new repository
  • RHDH core tests to be created for core functionality only
• Support for automated PR workflows:
  • Plugin PRs trigger corresponding overlay repository PRs for integration testing
  • Test results commented back on original plugin PRs
  • RHDH core PRs gated only by core-specific tests
• Cross-platform testing support:
  • Automated testing across AKS, GKE, OCP, and EKS platforms
  • Platform-specific test configurations and validations
• FIPS compliance testing capability
• Local test execution environment setup
  • Minimal setup required to run tests against local RHDH instance
  • Test results matching CI environment behavior
• Test failure diagnostics and investigation support:
  • Clear indication whether failure is plugin-specific or core-related
  • Integration with issue tracking for failure patterns
• Provide documentation on how to run the tests locally against an RHDH instance
• Ensure the transition is seamless with no loss of test coverage

Out of Scope (Optional)

• Changes to the actual test implementation logic (only moving the tests, not rewriting them)
• Tests for the RHDH core platform (these remain in the original repository)
• Integration with third-party testing frameworks beyond what's currently used
• Manual test coordination workflows (handled separately)
• Third-party/community plugin testing infrastructure (separate consideration)

Customer Considerations (Optional)

• This change is primarily internal and should not directly impact customers
• Improved test isolation may lead to higher quality plugins and faster release cycles

Documentation Considerations

• Create documentation on:
  • How to run plugin tests locally against an RHDH instance
  • How to add new plugin tests to the repository
  • How the nightly integration test pipeline works
  • How selective test execution is implemented
• Update existing RHDH documentation to reference the new repository for plugin testing