In the past we have encountered scenarios where images would be leveraged as part of different hidden pipeline steps in dsp v1. These images were hard coded and pulled from public repositories like dockerhub resulting in various issues with disconnected, rate limitation, etc.

We have accounted for some of these in DSP V2 by parameterizing newly introduced images like kfp-launcher and kfp-driver, but a thorough inspection of the kfp v2 code bases is needed. Ensure no other images are being hard coded during dspa deployment, during pipeline runs, etc. Attempt to run different types of pipelines and inspect their container images. Ensure all images being used (except for the user provided executor image) are planned and accounted for, we should be providing these via cpaas builds (or in the process of doing so).

Another good way to confirm this with a high degree of certainty is to deploy dsp in a disconnected environment, mirror needed images, and run pipelines, and confirm all images are accounted for.