Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-16008

pods assigned with Multus whereabouts IP get stuck in ContainerCreating state after OCP upgrading [Backport 4.12]

    XMLWordPrintable

Details

    • No
    • CNF Network Sprint 239, CNF Network Sprint 240, CNF Network Sprint 241, CNF Network Sprint 247, CNF Network Sprint 248
    • 5
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, pods assigned an IP from the pool created by the Whereabouts CNI plugin were getting stuck in `ContainerCreating` state after a node force reboot. With this release, the Whereabouts CNI plugin issue associated with the IP allocation after a node force reboot is resolved. (link:https://issues.redhat.com/browse/OCPBUGS-16008[OCPBUGS-16008])
      Show
      Previously, pods assigned an IP from the pool created by the Whereabouts CNI plugin were getting stuck in `ContainerCreating` state after a node force reboot. With this release, the Whereabouts CNI plugin issue associated with the IP allocation after a node force reboot is resolved. (link: https://issues.redhat.com/browse/OCPBUGS-16008 [ OCPBUGS-16008 ])
    • Bug Fix
    • Done
    • Customer Escalated
    • Hide
      11/7: tbd for 4.12.z, testing in progress
      9/10: u/s PR has merged, waiting for backports
      8/22: PR pending merge then backports; KNIECO-8019
      Show
      11/7: tbd for 4.12.z, testing in progress 9/10: u/s PR has merged, waiting for backports 8/22: PR pending merge then backports; KNIECO-8019

    Description

      Description of problem:

      pods assigned with Multus whereabouts IP get stuck in ContainerCreating state after OCP upgrading from 4.12.15 to 4.12.22. Not sure if upgrading cause the issue or node rebooting directly cause the issue.

The error message is:
(combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox mypod-0-0-1-0_testproject_8c8500e1-1643-4716-8fd7-e032292c62ab_0(2baa045a1b19291769ed56bab288b60802179ff3138ffe0d16a14e78f9cb5e4f): error adding pod testproject_mypod-0-0-1-0 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [testproject/mypod-0-0-1-0/8c8500e1-1643-4716-8fd7-e032292c62ab:testproject-net-svc-kernel-bond]: error adding container to network "testproject-net-svc-kernel-bond": error at storage engine: k8s get error: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

      Version-Release number of selected component (if applicable):

      How reproducible:

      Not sure if it is reproducible

      Steps to Reproduce:

      1.
2.
3.

      Actual results:

      Pods stuck in ContainerCreating state

      Expected results:

      Pods creates normally

      Additional info:

      Customer responded deleting statefulset and recreated it didn't work.
The pods can be created normally after deleting corresponding ippools.whereabouts.cni.cncf.io manually
$ oc delete ippools.whereabouts.cni.cncf.io 172.21.24.0-22 -n openshift-multus

      Attachments

        Issue Links

          depends on

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-27367 pods assigned with Multus whereabouts IP get stuck in ContainerCreating state after OCP upgrading [Backport 4.13]

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-18893 pods assigned with Multus whereabouts IP get stuck in ContainerCreating state after OCP upgrading

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-26553 pods assigned with Multus whereabouts IP get stuck in ContainerCreating state after OCP upgrading [Backport 4.14]

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is duplicated by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-16404 Pod gets stuck in ContainerCreating state with exhausted Whereabouts IPAM range

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is related to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. NP-805 Several pending pods at once will cause the reconcile cycle to stall

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • In Progress
          links to

          GitHub openshift/whereabouts-cni#238: OCPBUGS-16008: Cherry pick fix assignment 4.12

          Web Link RHBA-2024:0664 OpenShift Container Platform 4.12.z bug fix update

          Activity

            People

              lnoy@redhat.com Lior Noy
              rhn-support-yhuang Ying Huang
              Weibin Liang Weibin Liang
              Carlos Goncalves
              Votes:
              2 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: