TopoLVM container image download is stuck occasionally, causing MicroShift CI to fail greenboot checks after creating VMs

4.15

Sometimes

1. Watch greenboot output

VM startup fails due to greenboot errors

+ 10:21:48.963462058 ./bin/scenario.sh:273     ssh -oConnectTimeout=10 -oBatchMode=yes -oStrictHostKeyChecking=accept-new redhat@192.168.122.136 'sudo journalctl -n 10 -u greenboot-healthcheck'
Nov 28 05:21:44 el92-src-storage-host1 40_microshift_running_check.sh[1253]: openshift-storage                   5m7s        Normal    Started                       pod/topolvm-node-4bbxk                          Started container csi-registrar
Nov 28 05:21:44 el92-src-storage-host1 40_microshift_running_check.sh[1253]: openshift-storage                   5m7s        Normal    Pulling                       pod/topolvm-node-4bbxk                          Pulling image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:829a8e4d34404abbd22fddb6ebfa0f74daa55f2697fb147da77b83fc8b473d8c"
Nov 28 05:21:44 el92-src-storage-host1 40_microshift_running_check.sh[1253]: openshift-storage                   5m4s        Normal    Pulled                        pod/topolvm-node-4bbxk                          Successfully pulled image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:829a8e4d34404abbd22fddb6ebfa0f74daa55f2697fb147da77b83fc8b473d8c" in 3.605s (3.605s including waiting)
Nov 28 05:21:44 el92-src-storage-host1 40_microshift_running_check.sh[1253]: openshift-storage                   5m3s        Normal    Created                       pod/topolvm-node-4bbxk                          Created container liveness-probe
Nov 28 05:21:44 el92-src-storage-host1 40_microshift_running_check.sh[1253]: openshift-storage                   5m3s        Normal    Started                       pod/topolvm-node-4bbxk                          Started container liveness-probe
Nov 28 05:21:44 el92-src-storage-host1 40_microshift_running_check.sh[1253]: openshift-storage                   5m25s       Normal    SuccessfulCreate              daemonset/topolvm-node                          Created pod: topolvm-node-4bbxk
Nov 28 05:21:44 el92-src-storage-host1 40_microshift_running_check.sh[1253]: ------
Nov 28 05:21:44 el92-src-storage-host1 40_microshift_running_check.sh[1253]: FAILURE
Nov 28 05:21:44 el92-src-storage-host1 systemd[1]: greenboot-healthcheck.service: Triggering OnFailure= dependencies.
Nov 28 05:21:44 el92-src-storage-host1 systemd[1]: greenboot-healthcheck.service: Consumed 1min 1.582s CPU time.

VM startup finish successfully

- containerID: cri-o://3064b18cc0fc10bd381c6935898f5835549f050a4f22658af77b5b1e0d38367e
    image: registry.access.redhat.com/ubi8/openssl@sha256:9e743d947be073808f7f1750a791a3dbd81e694e37161e8c6c6057c2c342d671
    imageID: registry.access.redhat.com/ubi8/openssl@sha256:9e743d947be073808f7f1750a791a3dbd81e694e37161e8c6c6057c2c342d671
    lastState: {}
    name: self-signed-cert-generator
    ready: true
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: cri-o://3064b18cc0fc10bd381c6935898f5835549f050a4f22658af77b5b1e0d38367e
        exitCode: 0
        finishedAt: "2023-11-28T09:01:30Z"
        reason: Completed
        startedAt: "2023-11-28T09:01:29Z"