Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-16634

[OVN-Kubernetes] IP currently assigned to multiple pods

XMLWordPrintable

    • No
    • SDN Sprint 241, SDN Sprint 242, SDN Sprint 243, SDN Sprint 244
    • 4
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, a race condition occurred between the handling of pods that had been scheduled and the pods that had been completed on a node when OVN-Kubernetes started. This often happened when nodes would reboot. Consequently, the same IP was erroneously assigned to multiple pods. This update fixes the race condition so that the same IP is not assigned to multiple pods in those circumstances. (link:https://issues.redhat.com/browse/OCPBUGS-16634[*OCPBUGS-16634*])

      Cause: Race condition between the handling of pods that had been scheduled and the pods that had been completed on a node when openshift-ovn-kubernetes is starting up after not being running, like during certain time windows of a node reboot.
      Consequence: The same IP is erroneously assigned to multiple pods.
      Fix: Fixed the race.
      Result: The same IP should not be assigned to multiple pods in those circumstances.
      Show
      * Previously, a race condition occurred between the handling of pods that had been scheduled and the pods that had been completed on a node when OVN-Kubernetes started. This often happened when nodes would reboot. Consequently, the same IP was erroneously assigned to multiple pods. This update fixes the race condition so that the same IP is not assigned to multiple pods in those circumstances. (link: https://issues.redhat.com/browse/OCPBUGS-16634 [* OCPBUGS-16634 *]) Cause: Race condition between the handling of pods that had been scheduled and the pods that had been completed on a node when openshift-ovn-kubernetes is starting up after not being running, like during certain time windows of a node reboot. Consequence: The same IP is erroneously assigned to multiple pods. Fix: Fixed the race. Result: The same IP should not be assigned to multiple pods in those circumstances.
    • Bug Fix
    • Done

      {  2023-07-19T16:52:37Z reason/ReusedPodIP podIP 10.128.0.39 is currently assigned to multiple pods: ns/e2e-replicaset-4951 pod/test-rs-ddhkn node/ip-10-0-151-233.us-west-1.compute.internal uid/117115dd-dc8f-4333-b972-ed880fcf8dd9;ns/openshift-apiserver pod/apiserver-5f7d4599b4-dvpdk node/ip-10-0-151-233.us-west-1.compute.internal uid/293cba9c-11ea-4258-9d38-4ff5b2cb52bd
      2023-07-19T16:58:40Z reason/ReusedPodIP podIP 10.128.0.39 is currently assigned to multiple pods: ns/e2e-job-1076 pod/pod-disruption-failure-ignore-2-qlxp2 node/ip-10-0-151-233.us-west-1.compute.internal uid/3dda8eea-b221-433a-b254-fc7cf487189b;ns/openshift-apiserver pod/apiserver-5f7d4599b4-dvpdk node/ip-10-0-151-233.us-west-1.compute.internal uid/293cba9c-11ea-4258-9d38-4ff5b2cb52bd}
      
      I0719 16:44:56.659916   49761 base_network_controller_pods.go:444] [default/openshift-apiserver/apiserver-5f7d4599b4-dvpdk] creating logical port openshift-apiserver_apiserver-5f7d4599b4-dvpdk for pod on switch ip-10-0-151-233.us-west-1.compute.internal
      
      W0719 16:44:56.666407   49761 base_network_controller_pods.go:198] No cached port info for deleting pod default/openshift-kube-controller-manager/installer-7-ip-10-0-151-233.us-west-1.compute.internal. Using logical switch ip-10-0-151-233.us-west-1.compute.internal port uuid  and addrs [10.128.0.39/23]
      
      I0719 16:44:56.680604   49761 base_network_controller_pods.go:234] Releasing IPs for Completed pod: openshift-kube-controller-manager/installer-7-ip-10-0-151-233.us-west-1.compute.internal, ips: 10.128.0.39
      
      I0719 16:44:56.699279   49761 pods.go:134] Attempting to release IPs for pod: openshift-kube-controller-manager/installer-7-ip-10-0-151-233.us-west-1.compute.internal, ips: 10.128.0.39
      
      I0719 16:44:56.790903   49761 client.go:783]  "msg"="transacting operations" "database"="OVN_Northbound" "operations"="[\{Op:insert Table:Logical_Switch_Port Row:map[addresses:{GoSet:[0a:58:0a:80:00:27 10.128.0.39]} external_ids:\{GoMap:map[namespace:openshift-apiserver pod:true]} name:openshift-apiserver_apiserver-5f7d4599b4-dvpdk 
      

      Observed in 
      https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-net[…]perator-master-e2e-aws-ovn-single-node/1681699276796727296
       
      https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_clus[…]netes_ovnkube-node-bsbt9_ovnkube-controller.log

            jcaamano@redhat.com Jaime Caamaño Ruiz
            jcaamano@redhat.com Jaime Caamaño Ruiz
            Weibin Liang Weibin Liang
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: