static container pod cannot be running due to get different pod UID

This bug was initially created as a copy of Bug #2038019

I am copying this bug because:

Description of problem:

Static container pod ( create pod yaml file in /etc/kubernetes/manifests/ on worker and let kubelet create the test pod)

met the following error:

Warning FailedCreatePodSandBox <invalid> kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-212-43.ap-southeast-1.compute.internal_zhouyt_937a3fb4b41f9b5b073f252037c0c642_0(ab9baefc8c6d7bf1664ec01cf81ea37bc76069556d7cedbfaca6150ac1e62541): error adding pod zhouyt_static-web-ip-10-0-212-43.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-212-43.ap-southeast-1.compute.internal/937a3fb4b41f9b5b073f252037c0c642]: expected pod UID "937a3fb4b41f9b5b073f252037c0c642" but got "62ebfd59-5409-4c3e-8c10-069f5b1092e7" from Kube API

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-06-183041

How reproducible:
always

Steps to Reproduce:
1. Create pod yaml file in /etc/kubernetes/manifests/ of worker

$oc debug node/worker1

sh-4.4# cd /etc/kubernetes/manifests/
sh-4.4# cat static-web.yaml
apiVersion: v1
kind: Pod
metadata:
name: static-web
namespace: zhouyt
labels:
role: myrole
spec:
containers:

• name: web
  image: nginx
  ports:
• name: web
  containerPort: 80
  protocol: TCP

2. Check the pod cannot be running
3. oc describe pod

Actual results:

$ oc get pod -n zhouyt -o wide
static-web-ip-10-0-142-146.ap-southeast-1.compute.internal 0/1 ContainerCreating 0 75m <none> ip-10-0-142-146.ap-southeast-1.compute.internal <none> <none>

$ oc describe pod show the following error:

Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 77m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(12c7c4a724cea9090dc507c100986355e23eca8c9fdbdc624f9d2f8b7238d751): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
Warning FailedCreatePodSandBox 77m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(d08facb68721d241f66b4b9997fa93be0fb0c76d055085855cb2eb5a749ec1a5): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(4831e6b22dbc166257e19725433f4f32f5661ed828e096c45572faf51fb53766): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(f194239aefcdcf0aab433a88a8dd3b6a3d455bf505c88fc1efa578a73bc85b83): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(c7371edc8aea60be92507837808493eb3403361bbed24d12d726872327e2dd02): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(b3e2879ba9967893799448f64f0c7aa5a02f4b4c222a1ba52a2dc9fbe531afd0): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(82f37c94e06faf1f1fb862e86e7aed48475f658a533f251a6e1ad6e4a86cf5ab): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
Warning FailedCreatePodSandBox 75m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(7ab12e9f6d606943ee678b0b8400bd4136b07d28acb3a8c65e94772736e0acb4): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
Warning FailedCreatePodSandBox 75m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(6ccb2d7f18c313beb489073bc54a243e67d84403d7888ca942d197f29761f000): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
Warning FailedCreatePodSandBox 2m5s (x336 over 75m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(512df204bea415d20618e085ab3dd9cade5816bbf21f49a8b2a7b4e547ccc367): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API

Expected results:

Additional info:

Change the pod to 'hostnetwork' pod. it works well

sh-4.4# cat test-static.yaml
apiVersion: v1
kind: Pod
metadata:
name: static-hellozy
namespace: testzy
spec:
containers:

• image: "quay.io/openshifttest/hello-pod@sha256:04b6af86b03c1836211be2589db870dba09b7811c197c47c07fbbe33c7f80ef7"
  name: hello-pod
  hostNetwork: true

[root@localhost ~]# oc get po
NAME READY STATUS RESTARTS AGE
static-hellozy-ip-10-0-136-177.ap-southeast-1.compute.internal 1/1 Running 0 5s