Monitor tests running against IBM Cloud are failing during setup, mainly the network-availability tests. These failures are causing the CI to fail for PR's involving IBM Cloud related testing (e.g., e2e-ibmcloud-ovn), showing up in PR reviews every time monitor testing is run.

    4.15/4.16 (master)

    90% (likely 100% if CI was stable to reach OCP Conformance testing every build)

    1. Open a PR to installer
    2. Trigger/retrigger e2e-ibmcloud-ovn test
    

      * could not run steps: step e2e-ibmcloud-ovn failed: ["e2e-ibmcloud-ovn" test steps failed: "e2e-ibmcloud-ovn" pod "e2e-ibmcloud-ovn-openshift-e2e-test" failed: could not watch pod: the pod ci-op-gdhnxf4y/e2e-ibmcloud-ovn-openshift-e2e-test failed after 56m43s (failed containers: test): ContainerFailed one or more containers exited9054190542Container test exited with code 1, reason Error90543---90544:"4.16","Tests":[{"Name":"[Jira:\"Network / ovn-kubernetes\"] monitor test pod-network-avalibility setup","Risk":{"Level":{"Name":"Low","Level":1},"Reasons":["This test has passed 33.33% of 3 runs on release 4.16 [amd64 ha ovn] in the last week."]},"OpenBugs":[]},{"Name":"[sig-network] pods should successfully create sandboxes by adding pod to network","Risk":{"Level":{"Name":"Medium","Level":50},"Reasons":["This test has passed 90.35% of 549 runs on release 4.16 [Overall] in the last week."]},"OpenBugs":[{"id":15598546,"key":"OCPBUGS-22839","created_at":"2023-12-07T19:04:19.254794Z","updated_at":"2023-12-12T00:24:53.711025Z","deleted_at":null,"status":"New","last_change_time":"2023-11-02T12:04:39Z","summary":"Failed to create the sandbox-plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF","affects_versions":["4.14.z"],"fix_versions":[],"components":["Networking / multus"],"labels":["rollback","rollback"],"url":"https://issues.redhat.com/browse/OCPBUGS-22839"}]},{"Name":"[sig-network] can collect host-to-service poller pod logs","Risk":{"Level":{"Name":"Low","Level":1},"Reasons":["This test has passed 66.67% of 3 runs on release 4.16 [amd64 ha ovn] in the last week."]},"OpenBugs":[]},{"Name":"[sig-network] can collect pod-to-service poller pod logs","Risk":{"Level":{"Name":"Low","Level":1},"Reasons":["This test has passed 66.67% of 3 runs on release 4.16 [amd64 ha ovn] in the last week."]},"OpenBugs":[]}],"OverallRisk":{"Level":{"Name":"Medium","Level":50},"Reasons":["Maximum failed test risk: Medium"]},"OpenBugs":[]}9054590546Successfully wrote: /logs/artifacts/junit/risk-analysis.json90547{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"k8s.io/test-infra/prow/entrypoint/run.go:84","func":"k8s.io/test-infra/prow/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2023-12-12T01:16:47Z"}90548error: failed to execute wrapped command: exit status 1

    Passing OCP Conformance tests on IBM Cloud

    The normal test logs don't contain too much in terms of helping track down the issue, besides a setup failure:

: [Jira:"Network / ovn-kubernetes"] monitor test pod-network-avalibility setup expand_less1m2s{  failed during setup
context deadline exceeded}

: [sig-network] can collect pod-to-service poller pod logs expand_less0s{  no pods found for poller %q
pod-to-service}

: [sig-network] can collect host-to-service poller pod logs expand_less0s{  no pods found for poller %q
host-to-service}


But the junit xml does have more details, although I have yet to understand it in full so far.
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_installer/7805/pull-ci-openshift-installer-master-e2e-ibmcloud-ovn/1734324484720037888/artifacts/e2e-ibmcloud-ovn/openshift-e2e-test/artifacts/junit/e2e-monitor-tests__20231212-002038.xml