Description of problem:

Reinstall of RHODS fails to start operator:

Due to:

Reading manifest in brew.registry.redhat.io/rhods/odh-deployer-rhel8: name unknown: Digest not found

Prerequisites (if any, like setup, operators/versions):

Steps to Reproduce

1. Install 1.28
2. Upgrade to RHODS 1.29
3. Reinstall 1.29 few more times on the same cluster

Actual results:

Jenkins Job output:

https://opendatascience-jenkins-csb-rhods.apps.ocp-c1.prod.psi.redhat.com/job/rhods-ci-pr-test/1914/console
1) Timeout- 0 pods found with the label selector app=rhods-dashboard in redhat-ods-applications namespace

2) Timeout- 0 pods found with the label selector app=notebook-controller in redhat-ods-applications namespace

3) Timeout- 0 pods found with the label selector app=odh-notebook-controller in redhat-ods-applications namespace

4) Timeout- 0 pods found with the label selector app.kubernetes.io/created-by=data-science-pipelines-operator in redhat-ods-applications namespace

5) Timeout- 0 pods found with the label selector prometheus=rhods-model-monitoring in redhat-ods-monitoring namespace
 

OCP Events shows to root cause:
61m Warning Failed pod/rhods-operator-78d59b47cb-4tzn7 Failed to pull image "registry.redhat.io/rhods/odh-deployer-rhel8@sha256:a355e4e6efd1be7dbbdb9ae5885e4bdc8353dfaad45ffe00cb22b9b2512e3e67": rpc error: code = Unknown desc = fetching target platform image selected from manifest list: reading manifest sha256:85cda0b6ad380314b7f0617ba8ec04daac1c3c9efcba3d407875861c8f4fc617 in brew.registry.redhat.io/rhods/odh-deployer-rhel8: name unknown: Digest not found

Expected results:

Reproducibility (Always/Intermittent/Only Once):

Many times, but only on cluster ODS-QE-08-MESH (that also had Service Mesh 2.4 installed)

Build Details:

Workaround:

Additional info:

This could be related to existing bug: https://issues.redhat.com/browse/OCPBUGS-15778