integration test cases with DSP, CodeFlare and KubeRay in a multi-step pipeline

Feature description

Create configurible integration test cases with Data Science Pipelines, CodeFlare and KubeRay, with the following characteristics:

• Small enough footprint that can be executed in a test environment;
• Container image and data should be free of intellectual property concerns;
• Easy to increase or decrease KubeRay cluster footprint;
• Easy to go from a single-step to N-step pipelines;

The test should run in the kind integration gh workflow in odh.

Describe alternatives you've considered

IBM Research has been running pipelines with DSP, CodeFlare, and KubeRay internally but the configuration and images contain proprietary libraries that are not suitable for integration testing purposes.

Anything else?

No response

Migrated from GitHub: https://github.com/opendatahub-io/data-science-pipelines/issues/179