-
Bug
-
Resolution: Done-Errata
-
Critical
-
None
-
13
-
False
-
None
-
True
-
-
-
Important
Description of problem:
Created a pod with annotation v1.multus-cni.io/default-network:
metadata:
name: my-pod
annotations:
v1.multus-cni.io/default-network: nijin-cnv/localnet-network
localnet-network is a localnet NAD:
# oc get net-attach-def localnet-network -o yaml |yq '.spec' config: "{\n \"cniVersion\": \"0.3.1\", \n \"name\": \"localnet1\", \n \"type\": \"ovn-k8s-cni-overlay\", \n \"topology\": \"localnet\", \n \"netAttachDefName\": \"nijin-cnv/localnet-network\" \n}\n"
Pod fails to start and multus have following error message:
2024-11-01T09:00:13Z [error] [nijin-cnv/my-pod/72f5ea65-b2b2-461b-8752-96f94550aaff:localnet1]: error adding container to network "localnet1": CNI request failed with status 400: '[nijin-cnv/my-pod 9313a3184c15351fa333afd67db76dfb8af09419dd9ec8e2d6d234cb48c51270 network localnet1 NAD nijin-cnv/localnet-network] [nijin-cnv/my-pod 9313a3184c15351fa333afd67db76dfb8af09419dd9ec8e2d6d234cb48c51270 network localnet1 NAD nijin-cnv/localnet-network] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded
The ovn-kubernetes running on the node doesn't have any error message. I cannot see the events related to adding the logical port and related events.
The NAD is working if I add the annotation as "k8s.v1.cni.cncf.io/networks: nijin-cnv/localnet-network".
Version-Release number of selected component (if applicable):
4.16.8
How reproducible:
100%
Steps to Reproduce:
1. Create a pod and define OVN secondary network in annotation "v1.multus-cni.io/default-network".
2. Pod is stuck in ContainerCreating state with above mentioned error in multus and events.
Actual results:
OVN secondary network is not working with multus default network override.
Expected results:
I am not really sure if using v1.multus-cni.io/default-network is supported for OVN secondary network. However, some of the OpenShift Virtualization component use this to select the default network for the pod.
Additional info:
The issue is observed while customer is using Migration Toolkit for Virtualization (MTV) to migrate VMs where they selected one of the OVN secondary network as transfer network. The MTV will create importer pods with annotation v1.multus-cni.io/default-network on to use the selected transfer network NAD.
- is blocked by
-
-
- Closed
-
- is caused by
-
-
- Closed
-
- links to
-
RHBA-2025:145593
MTV 2.7.10 Images
- mentioned on
[MTV-1645] OVN secondary network is not working with multus default network override
According to the testing result for warm/cold migrate VM from vSphere for using localnet with second ovs-bridge/ br-ex, set this bug as verified.
Tested on MTV 2.7.10 (IIB: 912790) with CNV 4.17.5
Scenario 3: Localnet mapping to br-ex - Warm/cold migration PASSED
Create the NodeNetworkConfigurationPolicy to map the localnet with br-ex,
create NetworkAttachmentDefinition with "annotations: forklift.konveyor.io/route: 192.168.50.1",
Warm/cold migrate win2019 vm/rhel7 from vsphere7 to ocp cluster,
The migration plan executed successfully, can login to vm, testing passed.
Steps:
1)Create NodeNetworkConfigurationPolicy to map localnet1 with br-ex
2) Create localnetwork: localnet1 with networkAttachmentDefinition and set: forklift.konveyor.io/route: "192.168.50.1", "subnets": "192.168.50.0/24","excludeSubnets": "192.168.50.1/32"
3) Create warm/cold migration plan to migrate vm: mtv-esx-win2019 from vSphere7, select the "Transfer Network: test/localnet1"
4)Check the warm importer pod/cold virt-v2v convert pod with the localnet1 route setting:
k8s.v1.cni.cncf.io/networks: [{"name":"localnet1","namespace":"test","default-route":["192.168.50.1"]}]
5) Check the migration plan executed successfully, login to vm, check the static ipaddress no change
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in merge request !1180 of mtv / MTV on branch MTV-2.7-rhel-8_upstream_6c67a49aed820c7223189ab34237b003:
Updated 3 upstream sources
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in merge request !1180 of mtv / MTV on branch MTV-2.7-rhel-8_ upstream _6c67a49aed820c7223189ab34237b003 : Updated 3 upstream sources 
Miguel Duarte de Mora Barroso
added a comment - Can the bug be moved to `VERIFIED` then ?
Are there more scenarios that need testing ? chhu@redhat.com
Miguel Duarte de Mora Barroso
added a comment - Can the bug be moved to `VERIFIED` then ?
Are there more scenarios that need testing ? chhu@redhat.com Tested on MTV 2.7.10(IIB: 912790) with CNV 4.17.4
Scenario 2: Localnet use another ovs-bridge - Cold migration PASSED
Cold migrate rhel7-9 vm from vSphere7 to ocp cluster with localnet: test/tenantblue, the plan executed successfully, can login to VM on target ocp cluster, static ip interface/ip address no change.
Tested on MTV 2.7.10(IIB: 912790) with CNV 4.17.4
Scenario 1: Localnet use another ovs-bridge - Warm migration PASSED
Create the NodeNetworkConfigurationPolicy to map the localnet with br-tenantblue(interface: eno8403 with trunk vlan 20),
create NetworkAttachmentDefinition with "annotations: forklift.konveyor.io/route: 192.168.20.1",
warm migrate win2019 vm from vsphere 7 to ocp cluster, check the importer pod with localnet route settings,
plan executed successfully, can login to VM on target ocp cluster
Steps:
1)Create NodeNetworkConfigurationPolicy to map tenantblue to br-tenantblue
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br-withvlan
spec:
desiredState:
interfaces:
- name: br-withvlan
type: ovs-bridge
state: up
bridge:
options:
stp: true
port:
- name: eno8403
vlan:
mode: trunk
trunk-tags:
- id: 20
ovn:
bridge-mappings:
- localnet: tenantblue
bridge: br-withvlan
state: present
2)Create localnetwork: tenantblue with networkAttachmentDefinition and set: forklift.konveyor.io/route: "192.168.20.1"
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: tenantblue
namespace: test
annotations:
forklift.konveyor.io/route: 192.168.20.1
spec:
config: |2
{
"cniVersion": "0.4.0",
"name": "tenantblue",
"type": "ovn-k8s-cni-overlay",
"topology":"localnet",
"subnets": "192.168.20.0/24",
"excludeSubnets": "192.168.20.1/32",
"vlanID": 20,
"netAttachDefName": "test/tenantblue"
}
3)Create warm migration plan to migrate vm: mtv-function-win2019-79 from vSphere7, select the "Transfer Network: test/tenantblue"
4)Check the DV importer pod with localnet settings:
k8s.v1.cni.cncf.io/networks: '[{"name":"tenantblue","namespace":"test","default-route":["192.168.20.1"]}]'
4)Check the migration plan is executed successfully, and can login to the VM on target cluster
We have some challenges with verifying this bug, due to this, and the related OCPBUGS-50898, this bug is getting pushed out to 2.7.11
Tested on MTV 2.7.10 (IIB: 911185) with CNV 4.17.5
Create the NodeNetworkConfigurationPolicy to map the localnet with br-ex,
create NetworkAttachmentDefinition with "annotations: forklift.konveyor.io/route: 192.168.50.1",
warm migrate vm from vsphere7 to ocp cluster, check the importer pod with k8s.v1.cni.cncf.io/networks:
k8s.v1.cni.cncf.io/networks: '[
{"name":"localnet-network","namespace":"mtv-test","default-route":["192.168.50.1"]}]
but the pod failed to connect worker node/outside network, track this in OCPBUGS-50898
Steps:
1) Create NodeNetworkConfigurationPolicy to map localnet1 to br-ex
# oc get nncp mapping -o yaml apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: ..... name: mapping spec: desiredState: ovn: bridge-mappings: - bridge: br-ex localnet: localnet-network state: present nodeSelector: node-role.kubernetes.io/worker: ""
2) Create localnetwork: localnet1 with networkAttachmentDefinition and set: forklift.konveyor.io/route: "192.168.50.1"
# oc get net-attach-def localnet-network -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: localnet-network
namespace: mtv-test
annotations:
forklift.konveyor.io/route: 192.168.50.1
spec:
config: |
{
"cniVersion": "0.4.0",
"name": "localnet-network",
"type": "ovn-k8s-cni-overlay",
"topology": "localnet",
"subnets": "192.168.50.0/24",
"excludeSubnets": "192.168.50.1/24",
"vlanID": 50,
"netAttachDefName": "mtv-test/localnet-network"
}
3) Create warm migration plan to migrate vm: mtv-function-rhel8-8 from vSphere7, select the
"Transfer Network: mtv-test/localnet-network"
4) Check the DV importer pod with
"k8s.v1.cni.cncf.io/networks: '[
]'"
more details in attached file: importer-prime-05557c5c-71ce-462a-936d-50fad64bd91c-checkpoint-snapshot-2585.yaml
$ oc get pod|grep importer importer-prime-05557c5c-71ce-462a-936d-50fad64bd91c-checkpoint-snapshot-2585 0/1 Init:0/1 0 18s importer-prime-05557c5c-71ce-462a-936d-50fad64bd91c-checkpoint-snapshot-2585.yaml: ---------------------------- apiVersion: v1 kind: Pod metadata: annotations: cdi.kubevirt.io/storage.createdByController: "yes" k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.129.2.170/23"],"mac_address":"0a:58:0a:81:02:aa","routes":[{"dest":"10.128.0.0/14","nextHop":"10.129.2.1"},{"dest":"172.30.0.0/16","nextHop":"10.129.2.1"},{"dest":"169.254.169.5/32","nextHop":"10.129.2.1"},{"dest":"100.64.0.0/16","nextHop":"10.129.2.1"}],"ip_address":"10.129.2.170/23"}}' k8s.v1.cni.cncf.io/networks: '[{"name":"localnet-network","namespace":"mtv-test","default-route":["192.168.50.1"]}]
5) Check the importer pod with errors, more details in attached file: importer-prime-05557c5c-71ce-462a-936d-50fad64bd91c-checkpoint-snapshot-2585.describe
networking: [mtv-test/importer-prime-05557c5c-71ce-462a-936d-50fad64bd91c-checkpoint-snapshot-2585/c8920ab5-9b37-4b98-ad1f-bb2ad0775c0b:localnet-network]: error adding container to network "localnet-network": CNI request failed with status 400: '[mtv-test/importer-prime-05557c5c-71ce-462a-936d-50fad64bd91c-checkpoint-snapshot-2585 930d7977e17ec63e9b1c7f53642932ef75c96e737e1bb4fee507d155ed628e82 network localnet-network NAD mtv-test/localnet-network] [mtv-test/importer-prime-05557c5c-71ce-462a-936d-50fad64bd91c-checkpoint-snapshot-2585 930d7977e17ec63e9b1c7f53642932ef75c96e737e1bb4fee507d155ed628e82 network localnet-network NAD mtv-test/localnet-network] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded
6) Check the localnet network through net1 in pod:
Start two pods without "forklift.konveyor.io/route: 192.168.50.1" in the ocp cluster, the pods: pod1, pod2 are running, login to pod1 check the network through net1, can't ping pod2/ pod1 worker node/ outside network, track this in OCPBUGS-50898
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in merge request !1163 of mtv / MTV on branch MTV-2.7-rhel-8_upstream_e258667192cf6f2b0b56f1bd25f5033f:
Updated 3 upstream sources
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in merge request !1163 of mtv / MTV on branch MTV-2.7-rhel-8_ upstream _e258667192cf6f2b0b56f1bd25f5033f : Updated 3 upstream sources 
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (MTV 2.7.10 Images), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2025:1754