Description of problem:

Created a pod with annotation v1.multus-cni.io/default-network:

metadata:
  name: my-pod
  annotations:
    v1.multus-cni.io/default-network: nijin-cnv/localnet-network

localnet-network is a localnet NAD:

# oc get net-attach-def localnet-network -o yaml |yq '.spec'

config: "{\n        \"cniVersion\": \"0.3.1\", \n        \"name\": \"localnet1\", \n        \"type\": \"ovn-k8s-cni-overlay\", \n        \"topology\": \"localnet\", \n        \"netAttachDefName\": \"nijin-cnv/localnet-network\" \n}\n"

Pod fails to start and multus have following error message:

2024-11-01T09:00:13Z [error] [nijin-cnv/my-pod/72f5ea65-b2b2-461b-8752-96f94550aaff:localnet1]: error adding container to network "localnet1": CNI request failed with status 400: '[nijin-cnv/my-pod 9313a3184c15351fa333afd67db76dfb8af09419dd9ec8e2d6d234cb48c51270 network localnet1 NAD nijin-cnv/localnet-network] [nijin-cnv/my-pod 9313a3184c15351fa333afd67db76dfb8af09419dd9ec8e2d6d234cb48c51270 network localnet1 NAD nijin-cnv/localnet-network] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded

The ovn-kubernetes running on the node doesn't have any error message. I cannot see the events related to adding the logical port and related events. 

The NAD is working if I add the annotation as "k8s.v1.cni.cncf.io/networks: nijin-cnv/localnet-network".

Version-Release number of selected component (if applicable):

4.16.8

How reproducible:

100%

Steps to Reproduce:

1. Create a pod and define OVN secondary network in annotation "v1.multus-cni.io/default-network".

2.  Pod is stuck in ContainerCreating state with above mentioned error in multus and events.

Actual results:

OVN secondary network is not working with  multus default network override.

Expected results:

I am not really sure if using v1.multus-cni.io/default-network is supported for OVN secondary network. However, some of the OpenShift Virtualization component use this to select the default network for the pod.

Additional info:

The issue is observed while customer is using Migration Toolkit for Virtualization (MTV) to migrate VMs where they selected one of the OVN secondary network as transfer network. The MTV will create importer pods with annotation v1.multus-cni.io/default-network on to use the selected transfer network NAD.