Description of problem:

1. Deleting the virt-launcher pod calls cnv-bridge to remove it from the cni network.
2. cnv-bridge calls nft (macspoof), and nft can take its time if there are many rules, see BZ2173485
3. If it takes more than 60s, its considered failed and tries to delete the pod network again
4. Another cnv-bridge is spawned, calling another nft
5. Goto 3

If nft is taking > 60s, this keeps piling up forever, like a slow fork bomb. The customer had issues during drain and hit out of memory killer due to this.

Version-Release number of selected component (if applicable):
4.10.45 (customer)
4.12.5 (reproduced)

How reproducible:
100%

Steps to Reproduce:1
1. Create a bridge NAD with macspoof enabled

spec:
config: >-
{"name":"virt.toca","type":"cnv-bridge","cniVersion":"0.3.1","bridge":"virt.toca","macspoofchk":true,"ipam":{}}

2. Create 1+ VMs attached to that network and run them on, all on node X.

3. On node X, where all Vms are running, add a bunch of fake nft rules to slow it down (easier than configuring tons of services and won't break your cluster)

$ cat /tmp/nft.sh
for a in

do
nft add chain ip table$a chain$b
done
done

4. Shutdown the VMs

Actual results:

• cnv-bridge and nft comes pile up in a loop, increasing every 60, never completes
• nft slows down ever further with all duplicate nft commands running, it just gets worse over time with even less chances of recoverying.

Expected results:

• something better than this. Maybe somehow wait for the already spawned one to complete, just don't keep forking in a loop, trying again and again a million times will not fix anything and just makes it all worse, as nft slows down further with every added command.

Additional info:

For a single VM shutdown, number of cnv-bridge and nft goes up every 60s.

....
Fri Mar 3 00:40:35 UTC 2023
systemd(1)───cnv-bridge(2616979)───nft(2616988)
systemd(1)───cnv-bridge(2618673)───nft(2618679)
systemd(1)───cnv-bridge(2620404)───nft(2620410)
systemd(1)───cnv-bridge(2622061)───nft(2622067)
Fri Mar 3 00:40:40 UTC 2023
systemd(1)───cnv-bridge(2616979)───nft(2616988)
systemd(1)───cnv-bridge(2618673)───nft(2618679)
systemd(1)───cnv-bridge(2620404)───nft(2620410)
systemd(1)───cnv-bridge(2622061)───nft(2622067)
systemd(1)───cnv-bridge(2623736)───nft(2623742)
Fri Mar 3 00:41:39 UTC 2023
systemd(1)───cnv-bridge(2616979)───nft(2616988)
systemd(1)───cnv-bridge(2618673)───nft(2618679)
systemd(1)───cnv-bridge(2620404)───nft(2620410)
systemd(1)───cnv-bridge(2622061)───nft(2622067)
systemd(1)───cnv-bridge(2623736)───nft(2623742)
systemd(1)───cnv-bridge(2625513)───nft(2625519)
Fri Mar 3 00:42:38 UTC 2023
systemd(1)───cnv-bridge(2616979)───nft(2616988)
systemd(1)───cnv-bridge(2618673)───nft(2618679)
systemd(1)───cnv-bridge(2620404)───nft(2620410)
systemd(1)───cnv-bridge(2622061)───nft(2622067)
systemd(1)───cnv-bridge(2623736)───nft(2623742)
systemd(1)───cnv-bridge(2625513)───nft(2625519)
systemd(1)───cnv-bridge(2627285)───nft(2627304)
...

Here is the loop causing it:

Mar 03 00:36:33 indigo.toca.local kubenswrapper[2896]: E0303 00:36:33.782374 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:35:33Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:36:34 indigo.toca.local crio[2516]: time="2023-03-03 00:36:34.337437914Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:37:34 indigo.toca.local kubenswrapper[2896]: E0303 00:37:34.339490 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:36:34Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:37:34 indigo.toca.local crio[2516]: time="2023-03-03 00:37:34.622870373Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:38:34 indigo.toca.local kubenswrapper[2896]: E0303 00:38:34.627033 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:37:34Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:38:35 indigo.toca.local crio[2516]: time="2023-03-03 00:38:35.505327803Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:39:35 indigo.toca.local kubenswrapper[2896]: E0303 00:39:35.512319 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:38:35Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:39:36 indigo.toca.local crio[2516]: time="2023-03-03 00:39:36.405370684Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:40:36 indigo.toca.local kubenswrapper[2896]: E0303 00:40:36.418374 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:39:36Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:40:37 indigo.toca.local crio[2516]: time="2023-03-03 00:40:37.391753278Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:41:37 indigo.toca.local kubenswrapper[2896]: E0303 00:41:37.400993 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:40:37Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:41:38 indigo.toca.local crio[2516]: time="2023-03-03 00:41:38.426007341Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:42:38 indigo.toca.local kubenswrapper[2896]: E0303 00:42:38.446835 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:41:38Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:42:39 indigo.toca.local crio[2516]: time="2023-03-03 00:42:39.452727743Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:43:39 indigo.toca.local kubenswrapper[2896]: E0303 00:43:39.506179 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:42:39Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:43:40 indigo.toca.local crio[2516]: time="2023-03-03 00:43:40.517316110Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:44:40 indigo.toca.local kubenswrapper[2896]: E0303 00:44:40.542166 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:43:40Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:44:40 indigo.toca.local crio[2516]: time="2023-03-03 00:44:40.584368061Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:45:40 indigo.toca.local kubenswrapper[2896]: E0303 00:45:40.620160 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:44:40Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:45:40 indigo.toca.local crio[2516]: time="2023-03-03 00:45:40.646295338Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:46:40 indigo.toca.local kubenswrapper[2896]: E0303 00:46:40.681181 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:45:40Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:46:41 indigo.toca.local crio[2516]: time="2023-03-03 00:46:41.751600469Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:47:41 indigo.toca.local kubenswrapper[2896]: E0303 00:47:41.766650 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:46:41Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:47:42 indigo.toca.local crio[2516]: time="2023-03-03 00:47:42.746573748Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:48:42 indigo.toca.local kubenswrapper[2896]: E0303 00:48:42.761576 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:47:42Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:48:42 indigo.toca.local crio[2516]: time="2023-03-03 00:48:42.851162161Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:49:42 indigo.toca.local kubenswrapper[2896]: E0303 00:49:42.871138 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:48:43Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:49:43 indigo.toca.local crio[2516]: time="2023-03-03 00:49:43.028061955Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"
Mar 03 00:50:43 indigo.toca.local kubenswrapper[2896]: E0303 00:50:43.046580 2896 kubelet.go:1829] failed to "KillPodSandbox" for "6dfbddf9-a5c0-49b2-ab50-28989e8b696d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_virt-launcher-rhel-91-fqvbg_openshift-cnv_6dfbddf9-a5c0-49b2-ab50-28989e8b696d_0(007fb03b13f4251e995382574b71f9dfd7185e049241db4d9a7c29706d1738d2): error removing pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2023-03-03T00:49:43Z [verbose] Del: openshift-cnv:virt-launcher-rhel-91-fqvbg:6dfbddf9-a5c0-49b2-ab50-28989e8b696d:virt.toca:net1 {\\\"name\\\":\\\"virt.toca\\\",\\\"type\\\":\\\"cnv-bridge\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"bridge\\\":\\\"virt.toca\\\",\\\"macspoofchk\\\":true,\\\"ipam\\\":{}}\\n\""
Mar 03 00:50:43 indigo.toca.local crio[2516]: time="2023-03-03 00:50:43.169293744Z" level=info msg="Deleting pod openshift-cnv_virt-launcher-rhel-91-fqvbg from CNI network \"multus-cni-network\" (type=multus)"