Feature Overview (aka. Goal Summary)

Extraction of end-to-end tests for plugins from the RHDH core repository into a dedicated repository to enable better separation of concerns, improved test management, and more efficient testing workflows for both the core platform and dynamic plugins.

Adjusted to incorporate the RHDH Test arch document for 1.8: https://docs.google.com/document/d/1UBpmZQ2PJ2FpEyniPDDdGTqpwd0GBAt_8YWqrf2q3CE/edit?tab=t.0#heading=h.lfcobaa82e1

Goals (aka. expected user outcomes)

• dedicated repository for plugin integration tests, decoupled from the RHDH core platform
• Developers will be able to run tests for specific plugins without running the entire test suite
• nightly integration test that runs all tests for all plugins
• Plugin developers will have clear documentation on how to run tests locally against an RHDH instance

Requirements (aka. Acceptance Criteria):

• Create a new repository for plugin integration tests that follows the same structure as the current test setup in the RHDH repository
• Move all plugin-specific end-to-end tests from the RHDH core repository to the new repository
• Configure the new repository to reference dynamic plugin configurations:
  • Initially for plugins shipped with the core RHDH image
  • Future support for OCI image references
• nightly integration test that runs all tests for all plugins
• mechanism to selectively run tests for individual plugins when changes are made to those specific plugin tests
  • probably by checking changes to source paths and by setting environment variables to selectively run tests
• Establish clear separation between different test types:
  • Unit tests remain in plugin repositories
  • Integration tests with external dependencies moved to new repository
  • RHDH core tests to be created for core functionality only
• Support for automated PR workflows:
  • Plugin PRs trigger corresponding overlay repository PRs for integration testing
  • Test results commented back on original plugin PRs
  • RHDH core PRs gated only by core-specific tests
• Cross-platform testing support:
  • Automated testing across AKS, GKE, OCP, and EKS platforms
  • Platform-specific test configurations and validations
• FIPS compliance testing capability
• Local test execution environment setup
  • Minimal setup required to run tests against local RHDH instance
  • Test results matching CI environment behavior
• Test failure diagnostics and investigation support:
  • Clear indication whether failure is plugin-specific or core-related
  • Integration with issue tracking for failure patterns
• Provide documentation on how to run the tests locally against an RHDH instance
• Ensure the transition is seamless with no loss of test coverage

Out of Scope (Optional)

• Changes to the actual test implementation logic (only moving the tests, not rewriting them)
• Tests for the RHDH core platform (these remain in the original repository)
• Integration with third-party testing frameworks beyond what's currently used
• Manual test coordination workflows (handled separately)
• Third-party/community plugin testing infrastructure (separate consideration)

Customer Considerations (Optional)

• This change is primarily internal and should not directly impact customers
• Improved test isolation may lead to higher quality plugins and faster release cycles

Documentation Considerations

• Create documentation on:
  • How to run plugin tests locally against an RHDH instance
  • How to add new plugin tests to the repository
  • How the nightly integration test pipeline works
  • How selective test execution is implemented
• Update existing RHDH documentation to reference the new repository for plugin testing