Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-25246

[IBMCloud] Monitor tests failing to setup

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • 4.15.0, 4.16.0
    • Test Framework
    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

          Monitor tests running against IBM Cloud are failing during setup, mainly the network-availability tests. These failures are causing the CI to fail for PR's involving IBM Cloud related testing (e.g., e2e-ibmcloud-ovn), showing up in PR reviews every time monitor testing is run.

      Version-Release number of selected component (if applicable):

          4.15/4.16 (master)

      How reproducible:

          90% (likely 100% if CI was stable to reach OCP Conformance testing every build)

      Steps to Reproduce:

          1. Open a PR to installer
          2. Trigger/retrigger e2e-ibmcloud-ovn test
          

      Actual results:

            * could not run steps: step e2e-ibmcloud-ovn failed: ["e2e-ibmcloud-ovn" test steps failed: "e2e-ibmcloud-ovn" pod "e2e-ibmcloud-ovn-openshift-e2e-test" failed: could not watch pod: the pod ci-op-gdhnxf4y/e2e-ibmcloud-ovn-openshift-e2e-test failed after 56m43s (failed containers: test): ContainerFailed one or more containers exited9054190542Container test exited with code 1, reason Error90543---90544:"4.16","Tests":[{"Name":"[Jira:\"Network / ovn-kubernetes\"] monitor test pod-network-avalibility setup","Risk":{"Level":{"Name":"Low","Level":1},"Reasons":["This test has passed 33.33% of 3 runs on release 4.16 [amd64 ha ovn] in the last week."]},"OpenBugs":[]},{"Name":"[sig-network] pods should successfully create sandboxes by adding pod to network","Risk":{"Level":{"Name":"Medium","Level":50},"Reasons":["This test has passed 90.35% of 549 runs on release 4.16 [Overall] in the last week."]},"OpenBugs":[{"id":15598546,"key":"OCPBUGS-22839","created_at":"2023-12-07T19:04:19.254794Z","updated_at":"2023-12-12T00:24:53.711025Z","deleted_at":null,"status":"New","last_change_time":"2023-11-02T12:04:39Z","summary":"Failed to create the sandbox-plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF","affects_versions":["4.14.z"],"fix_versions":[],"components":["Networking / multus"],"labels":["rollback","rollback"],"url":"https://issues.redhat.com/browse/OCPBUGS-22839"}]},{"Name":"[sig-network] can collect host-to-service poller pod logs","Risk":{"Level":{"Name":"Low","Level":1},"Reasons":["This test has passed 66.67% of 3 runs on release 4.16 [amd64 ha ovn] in the last week."]},"OpenBugs":[]},{"Name":"[sig-network] can collect pod-to-service poller pod logs","Risk":{"Level":{"Name":"Low","Level":1},"Reasons":["This test has passed 66.67% of 3 runs on release 4.16 [amd64 ha ovn] in the last week."]},"OpenBugs":[]}],"OverallRisk":{"Level":{"Name":"Medium","Level":50},"Reasons":["Maximum failed test risk: Medium"]},"OpenBugs":[]}9054590546Successfully wrote: /logs/artifacts/junit/risk-analysis.json90547{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"k8s.io/test-infra/prow/entrypoint/run.go:84","func":"k8s.io/test-infra/prow/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2023-12-12T01:16:47Z"}90548error: failed to execute wrapped command: exit status 1

      Expected results:

          Passing OCP Conformance tests on IBM Cloud

      Additional info:

          The normal test logs don't contain too much in terms of helping track down the issue, besides a setup failure:
      
      : [Jira:"Network / ovn-kubernetes"] monitor test pod-network-avalibility setup expand_less1m2s{  failed during setup
      context deadline exceeded}
      
      : [sig-network] can collect pod-to-service poller pod logs expand_less0s{  no pods found for poller %q
      pod-to-service}
      
      : [sig-network] can collect host-to-service poller pod logs expand_less0s{  no pods found for poller %q
      host-to-service}
      
      
      But the junit xml does have more details, although I have yet to understand it in full so far.
      https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_installer/7805/pull-ci-openshift-installer-master-e2e-ibmcloud-ovn/1734324484720037888/artifacts/e2e-ibmcloud-ovn/openshift-e2e-test/artifacts/junit/e2e-monitor-tests__20231212-002038.xml

      Attachments

        Activity

          People

            kenzhang@redhat.com Ken Zhang
            cschaefe@redhat.com Christopher Schaefer
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: