Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-57127

Deleted pod stays in terminating state indefinitely when having whereabout as a secondary network plugin

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • 4.12.z
    • Networking / multus
    • None
    • Incidents & Support
    • True
    • Hide

      Mavenir's end customer is not able to get the application pods up without applying workaround ie Node Reboot (which itself require downtime). 

       

      Following is the email from Mavenir from yesterday: 

       

      "

      RedHat SME has collected the logs 24 hours ago from problematic cluster, but we haven’t received any feedback. The only working cluster also has been degraded now hence we don’t have any healthy cluster in network and this will result in complete network outage.

       

      We need immediate recovery plan from RedHat.

      "

      Show
      Mavenir's end customer is not able to get the application pods up without applying workaround ie Node Reboot (which itself require downtime).    Following is the email from Mavenir from yesterday:    " RedHat SME has collected the logs 24 hours ago from problematic cluster, but we haven’t received any feedback. The only working cluster also has been degraded now hence we don’t have any healthy cluster in network and this will result in complete network outage.   We need immediate recovery plan from RedHat. "
    • None
    • Critical
    • None
    • None
    • None
    • None
    • CNF Network Sprint 272, CNF Network Sprint 273
    • 2
    • Customer Escalated
    • None
    • None
    • None

      Description of problem:

      When deleting a pod, it stays in Terminating state until we force delete the pod.

      Jun 04 17:26:03 lab1ocpims1-wrk-11.ims.lab1.ocp.net.ice.no crio[17788]: time="2025-06-04 17:26:03.903711411+02:00" level=info msg="Got pod network &{Name:slc-0 Namespace:mvnr-msg-sms-lab1 ID:ae0df4bc2b6991ee55b495e3d2bb4bc55f93f2bd5a16c9efbea5422d09441a1b UID:3d4b2899-3836-4abc-b609-bc43ee687604 NetNS:/var/run/netns/5952530d-023d-4604-9428-a2be34907c91 Networks:[{Name:multus-cni-network Ifname:eth0}] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}"Jun 04 17:26:03 lab1ocpims1-wrk-11.ims.lab1.ocp.net.ice.no crio[17788]: time="2025-06-04 17:26:03.903927022+02:00" level=info msg="Deleting pod mvnr-msg-sms-lab1_slc-0 from CNI network \"multus-cni-network\" (type=multus)"Jun 04 17:27:03 lab1ocpims1-wrk-11.ims.lab1.ocp.net.ice.no kubenswrapper[18216]: E0604 17:27:03.905909   18216 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_slc-0_mvnr-msg-sms-lab1_3d4b2899-3836-4abc-b609-bc43ee687604_0(ae0df4bc2b6991ee55b495e3d2bb4bc55f93f2bd5a16c9efbea5422d09441a1b): error removing pod mvnr-msg-sms-lab1_slc-0 from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2025-06-04T17:26:03+02:00 [verbose] Del: mvnr-msg-sms-lab1:slc-0:3d4b2899-3836-4abc-b609-bc43ee687604:msg-sms-dcca-b:net2 { \\\"cniVersion\\\": \\\"0.3.1\\\", \\\"name\\\": \\\"msg-sms-dcca-b\\\", \\\"type\\\": \\\"macvlan\\\", \\\"master\\\": \\\"bond1.1163\\\", \\\"ipMasq\\\": false, \\\"ipam\\\": { \\\"type\\\": \\\"whereabouts\\\", \\\"datastore\\\": \\\"kubernetes\\\", \\\"kubernetes\\\": {\\\"kubeconfig\\\": \\\"/etc/kubernetes/cni/net.d/whereabouts.d/whereabouts.kubeconfig\\\"}, \\\"range\\\": \\\"10.7.216.64/28\\\", \\\"range_start\\\": \\\"10.7.216.68\\\", \\\"range_end\\\": \\\"10.7.216.69\\\", \\\"routes\\\": [ { \\\"dst\\\": \\\"10.7.214.83/32\\\" } ], \\\"gateway\\\": \\\"10.7.216.65\\\"\\n} }\\n\"" podSandboxID="ae0df4bc2b6991ee55b495e3d2bb4bc55f93f2bd5a16c9efbea5422d09441a1b"

       

      Version-Release number of selected component (if applicable):

          4.12.53 multus

      Actual results:

      Pod stuck in terminating state.

      Expected results:

          It should delete the pods properly    

      Additional info:

      Customer: Mavnier

              rh-ee-marguerr Marcelo Guerrero Viveros (Inactive)
              rhn-support-nbhatt Neeraj Bhatt
              None
              None
              Weibin Liang Weibin Liang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: