Description of problem:
When a VM is defined w/ the default podnet and an additional multus network, the ordering matters: if podnet is defined first, all works fine; if podnet is not listed first, that network is not reachable

Version-Release number of selected component (if applicable):
kubevirt-hyperconverged-operator.v4.9.3 OpenShift Virtualization 4.9.3
OCP 4.9.13

How reproducible:
Every time

Steps to Reproduce:
1. Create a VM where the podnet is not the first interface, ex:

interfaces:

• bridge: {}
  model: virtio
  name: nic0
• masquerade: {}
  name: default
  [...]
  networks:
• multus:
  networkName: linux-bridge
  name: nic0
• name: default
  pod: {}

Try to ping the pod ip from the host node, it is not reachable.

Actual results:
VM podnet does not work if it is not listed first.

Expected results:
VM podnet should be reachable no matter the interface ordering.

Additional info:

• • VM - podnet NOT listed first:

interfaces:

• bridge: {}
  model: virtio
  name: nic0
• masquerade: {}
  name: default

VMI output:
interfaces:

• interfaceName: eth0
  ipAddress: 192.168.222.72
  ipAddresses:
• 192.168.222.72
• fe80::fb:1cff:fe00:1e
  mac: 02:fb:1c:00:00:1e
  name: nic0
• interfaceName: eth1
  ipAddress: 10.131.0.235
  ipAddresses:
• 10.131.0.235
  mac: 02:fb:1c:00:00:1f
  name: default

pod IP 10.131.0.235 does not ping from the host worker node. The bridge IP works fine (192.168.222.72).

Virt-launcher pod:
[root@test1 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if2215: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 0a:58:0a:83:00:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.131.0.235/23 brd 10.131.1.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::f489:dbff:feb9:6ab0/64 scope link
valid_lft forever preferred_lft forever
4: net1@if2216: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master k6t-net1 state UP group default
link/ether 02:fb:1c:0f:d8:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::fb:1cff:fe0f:d8d2/64 scope link
valid_lft forever preferred_lft forever
5: k6t-net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:fb:1c:0f:d8:d2 brd ff:ff:ff:ff:ff:ff
inet 169.254.75.10/32 scope global k6t-net1
valid_lft forever preferred_lft forever
inet6 fe80::fb:1cff:fe0f:d8d2/64 scope link
valid_lft forever preferred_lft forever
6: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master k6t-net1 state UP group default qlen 1000
link/ether d6:58:8a:4a:08:43 brd ff:ff:ff:ff:ff:ff
inet6 fe80::d458:8aff:fe4a:843/64 scope link
valid_lft forever preferred_lft forever
7: k6t-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 02:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.1/24 brd 10.0.2.255 scope global k6t-eth0
valid_lft forever preferred_lft forever
inet6 fe80::ff:fe00:0/64 scope link
valid_lft forever preferred_lft forever
8: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq master k6t-eth0 state UP group default qlen 1000
link/ether 8a:f8:a3:bb:ce:28 brd ff:ff:ff:ff:ff:ff
inet6 fe80::88f8:a3ff:febb:ce28/64 scope link
valid_lft forever preferred_lft forever
[root@test1 /]# ip route
default via 10.131.0.1 dev eth0
10.0.2.0/24 dev k6t-eth0 proto kernel scope link src 10.0.2.1
10.128.0.0/14 dev eth0
10.131.0.0/23 dev eth0 proto kernel scope link src 10.131.0.235
172.30.0.0/16 via 10.131.0.1 dev eth0
224.0.0.0/4 dev eth0

Inside VM:
[fedora@test1 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:fb:1c:00:00:1e brd ff:ff:ff:ff:ff:ff
altname enp1s0
inet 192.168.222.72/24 brd 192.168.222.255 scope global dynamic noprefixroute eth0
valid_lft 2318sec preferred_lft 2318sec
inet6 fe80::fb:1cff:fe00:1e/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
link/ether 02:fb:1c:00:00:1f brd ff:ff:ff:ff:ff:ff
altname enp2s0
inet 10.0.2.2/24 brd 10.0.2.255 scope global dynamic noprefixroute eth1
valid_lft 86298936sec preferred_lft 86298936sec
inet6 fe80::dbd2:3998:9ece:40af/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[fedora@test1 ~]$ ip route
default via 192.168.222.1 dev eth0 proto dhcp metric 100
default via 10.0.2.1 dev eth1 proto dhcp metric 101
10.0.2.0/24 dev eth1 proto kernel scope link src 10.0.2.2 metric 101
192.168.222.0/24 dev eth0 proto kernel scope link src 192.168.222.72 metric 100

• • VM - podnet listed first:

interfaces:

• masquerade: {}
  name: default
• bridge: {}
  model: virtio
  name: nic0

VMI output:
interfaces:

• interfaceName: eth0
  ipAddress: 10.131.0.236
  ipAddresses:
• 10.131.0.236
  mac: 02:fb:1c:00:00:20
  name: default
• interfaceName: eth1
  ipAddress: 192.168.222.75
  ipAddresses:
• 192.168.222.75
• fe80::341d:ebed:b93d:52b4
  mac: 02:fb:1c:00:00:21
  name: nic0

Both IPs are reachable from the host worker node.

Virt-launcher pod:
[root@test2 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if2217: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 0a:58:0a:83:00:ec brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.131.0.236/23 brd 10.131.1.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::700f:6fff:fe0f:7706/64 scope link
valid_lft forever preferred_lft forever
4: net1@if2218: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master k6t-net1 state UP group default
link/ether 02:fb:1c:90:2b:9c brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::fb:1cff:fe90:2b9c/64 scope link
valid_lft forever preferred_lft forever
5: k6t-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 02:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.1/24 brd 10.0.2.255 scope global k6t-eth0
valid_lft forever preferred_lft forever
inet6 fe80::ff:fe00:0/64 scope link
valid_lft forever preferred_lft forever
6: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq master k6t-eth0 state UP group default qlen 1000
link/ether 46:7d:99:a1:82:92 brd ff:ff:ff:ff:ff:ff
inet6 fe80::447d:99ff:fea1:8292/64 scope link
valid_lft forever preferred_lft forever
7: k6t-net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:fb:1c:90:2b:9c brd ff:ff:ff:ff:ff:ff
inet 169.254.75.11/32 scope global k6t-net1
valid_lft forever preferred_lft forever
inet6 fe80::fb:1cff:fe90:2b9c/64 scope link
valid_lft forever preferred_lft forever
8: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master k6t-net1 state UP group default qlen 1000
link/ether 06:3b:c7:71:00:d6 brd ff:ff:ff:ff:ff:ff
inet6 fe80::43b:c7ff:fe71:d6/64 scope link
valid_lft forever preferred_lft forever
[root@test2 /]# ip route
default via 10.131.0.1 dev eth0
10.0.2.0/24 dev k6t-eth0 proto kernel scope link src 10.0.2.1
10.128.0.0/14 dev eth0
10.131.0.0/23 dev eth0 proto kernel scope link src 10.131.0.236
172.30.0.0/16 via 10.131.0.1 dev eth0
224.0.0.0/4 dev eth0

Inside VM:
[fedora@test2 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
link/ether 02:fb:1c:00:00:20 brd ff:ff:ff:ff:ff:ff
altname enp1s0
inet 10.0.2.2/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
valid_lft 86299064sec preferred_lft 86299064sec
inet6 fe80::fb:1cff:fe00:20/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:fb:1c:00:00:21 brd ff:ff:ff:ff:ff:ff
altname enp2s0
inet 192.168.222.75/24 brd 192.168.222.255 scope global dynamic noprefixroute eth1
valid_lft 2269sec preferred_lft 2269sec
inet6 fe80::341d:ebed:b93d:52b4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[fedora@test2 ~]$ ip route
default via 10.0.2.1 dev eth0 proto dhcp metric 100
default via 192.168.222.1 dev eth1 proto dhcp metric 101
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.2 metric 100
192.168.222.0/24 dev eth1 proto kernel scope link src 192.168.222.75 metric 101