Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-37245

High rate of pod sandbox errors detected on metal

XMLWordPrintable

    • Important
    • None
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Component Readiness has found a potential regression in the following test:

      [sig-network] pods should successfully create sandboxes by adding pod to network

      Probability of significant regression: 99.93%

      Sample (being evaluated) Release: 4.17
      Start Time: 2024-07-12T00:00:00Z
      End Time: 2024-07-18T23:59:59Z
      Success Rate: 74.29%
      Successes: 25
      Failures: 9
      Flakes: 1

      Base (historical) Release: 4.16
      Start Time: 2024-05-31T00:00:00Z
      End Time: 2024-06-27T23:59:59Z
      Success Rate: 98.18%
      Successes: 54
      Failures: 1
      Flakes: 0

      View the test details report at https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=metal&Platform=metal&Scheduler=default&SecurityMode=default&Suite=unknown&Suite=unknown&Topology=ha&Topology=ha&Upgrade=minor&Upgrade=minor&baseEndTime=2024-06-27%2023%3A59%3A59&baseRelease=4.16&baseStartTime=2024-05-31%2000%3A00%3A00&capability=Other&columnGroupBy=Platform&columnGroupBy=Architecture&columnGroupBy=Network&component=Networking%20%2F%20cluster-network-operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20metal%20unknown%20ha%20minor&ignoreDisruption=1&ignoreMissing=0&includeVariant=Architecture%3Aamd64&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&pity=5&sampleEndTime=2024-07-18%2023%3A59%3A59&samplePRNumber=&samplePROrg=&samplePRRepo=&sampleRelease=4.17&sampleStartTime=2024-07-12%2000%3A00%3A00&testId=openshift-tests%3A65e48733eb0b6115134b2b8c6a365f16&testName=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network

      This test appears to be failing roughly 50% of the time on periodic-ci-openshift-release-master-nightly-4.17-upgrade-from-stable-4.16-e2e-metal-ipi-ovn-upgrade and the error looks workable:

       [sig-network] pods should successfully create sandboxes by adding pod to network expand_less 	0s
{  1 failures to create the sandbox

namespace/e2e-test-ns-global-srg5f node/worker-1 pod/test-ipv6-podtm8vn hmsg/da5d303f42 - never deleted - firstTimestamp/2024-07-18T11:26:41Z interesting/true lastTimestamp/2024-07-18T11:26:41Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-ipv6-podtm8vn_e2e-test-ns-global-srg5f_65c4722e-d832-4ec8-8209-39587a81d95d_0(d11ec24638e2d578486e57851a419e52ddd4367d48b33e46825f7c42687c9f7f): error adding pod e2e-test-ns-global-srg5f_test-ipv6-podtm8vn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d11ec24638e2d578486e57851a419e52ddd4367d48b33e46825f7c42687c9f7f" Netns:"/var/run/netns/7bb7a08a-9352-49d6-a211-02046349dba6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=e2e-test-ns-global-srg5f;K8S_POD_NAME=test-ipv6-podtm8vn;K8S_POD_INFRA_CONTAINER_ID=d11ec24638e2d578486e57851a419e52ddd4367d48b33e46825f7c42687c9f7f;K8S_POD_UID=65c4722e-d832-4ec8-8209-39587a81d95d" Path:"" ERRORED: error configuring pod [e2e-test-ns-global-srg5f/test-ipv6-podtm8vn] networking: [e2e-test-ns-global-srg5f/test-ipv6-podtm8vn/65c4722e-d832-4ec8-8209-39587a81d95d:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[e2e-test-ns-global-srg5f/test-ipv6-podtm8vn d11ec24638e2d578486e57851a419e52ddd4367d48b33e46825f7c42687c9f7f network default NAD default] [e2e-test-ns-global-srg5f/test-ipv6-podtm8vn d11ec24638e2d578486e57851a419e52ddd4367d48b33e46825f7c42687c9f7f network default NAD default] failed to configure pod interface: timed out waiting for OVS port binding (ovn-installed) for 0a:58:0a:83:00:e3 [10.131.0.227/23]
'
': StdinData: {"binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}}

      Taken from: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-upgrade-from-stable-4.16-e2e-metal-ipi-ovn-upgrade/1813846390107803648

              rhn-engineering-dgoodwin Devan Goodwin
              rhn-engineering-dgoodwin Devan Goodwin
              Anurag Saxena Anurag Saxena
              Nadia Pinaeva
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: