Description of problem:

Creating the following NodeNetworkConfigurationPolicy resource fails:

$ cat 01-vlan-interface-nncp.yml
apiVersion: nmstate.io/v1beta1
kind: NodeNetworkConfigurationPolicy
metadata:
name: ontap-vlan-interface
spec:
nodeSelector:
kubernetes.io/hostname: worker-1
desiredState:
interfaces:

• name: "eno5np0.350"
  description: vlan interface for NetApp ONTAP storage.
  type: vlan
  state: up
  ipv4:
  enabled: true
  dhcp: true
  auto-gateway: false
  vlan:
  base-iface: "eno5np0"
  id: 350

Error:
Traceback (most recent call last):
File "/usr/bin/nmstatectl", line 11, in <module>
load_entry_point('nmstate==1.0.2', 'console_scripts', 'nmstatectl')()
File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 73, in main
return args.func(args)
File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 326, in set
return apply(args)
File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 354, in apply
args.save_to_disk,
File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 407, in apply_state
save_to_disk=save_to_disk,
File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 81, in apply
_apply_ifaces_state(plugins, net_state, verify_change, save_to_disk)
File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 114, in _apply_ifaces_state
plugin.apply_changes(net_state, save_to_disk)
File "/usr/lib/python3.6/site-packages/libnmstate/nm/plugin.py", line 233, in apply_changes
NmProfiles(self.context).apply_config(net_state, save_to_disk)
File "/usr/lib/python3.6/site-packages/libnmstate/nm/profiles.py", line 95, in apply_config
self._ctx.wait_all_finish()
File "/usr/lib/python3.6/site-packages/libnmstate/nm/context.py", line 213, in wait_all_finish
raise tmp_error
libnmstate.error.NmstateLibnmError: Activate profile uuid:9c545b9c-3233-4863-af07-5113d9eaaf46 iface:eno5np0.350 type: vlan failed: error=nm-manager-error-quark: Failed to find a compatible device for this connection (3)

Version-Release number of selected component (if applicable):

$ oc version
Client Version: 4.8.17
Server Version: 4.8.17
Kubernetes Version: v1.21.1+6438632

$ oc get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
kubevirt-hyperconverged-operator.v4.8.2 OpenShift Virtualization 4.8.2 kubevirt-hyperconverged-operator.v4.8.1 Succeeded

$ oc -n openshift-cnv exec -it nmstate-handler-8c9z8 – rpm -q nmstate
nmstate-1.0.2-14.el8_4.noarch

[core@worker-1 ~]$ rpm -qa | grep NetworkManager
NetworkManager-libnm-1.30.0-10.el8_4.x86_64
NetworkManager-cloud-setup-1.30.0-10.el8_4.x86_64
NetworkManager-1.30.0-10.el8_4.x86_64
NetworkManager-ovs-1.30.0-10.el8_4.x86_64
NetworkManager-team-1.30.0-10.el8_4.x86_64
NetworkManager-tui-1.30.0-10.el8_4.x86_64

How reproducible:

100%

Steps to Reproduce:

• Create the vlan interface with the data above.
  $ oc create -f 01-vlan-interface-nncp.yml
  nodenetworkconfigurationpolicy.nmstate.io/ontap-vlan-interface created

• Validate the nncp status
  $ oc get nncp
  NAME STATUS
  ontap-vlan-interface FailedToConfigure

• Verify the enactment of the node
  $ oc get nnce
  NAME STATUS
  master-0.ontap-vlan-interface NodeSelectorNotMatching
  master-1.ontap-vlan-interface NodeSelectorNotMatching
  master-2.ontap-vlan-interface NodeSelectorNotMatching
  worker-0.ontap-vlan-interface NodeSelectorNotMatching
  worker-1.ontap-vlan-interface FailedToConfigure
  worker-2.ontap-vlan-interface NodeSelectorNotMatching
  worker-3.ontap-vlan-interface NodeSelectorNotMatching

• Get more details of the error. ( see attached log worker-1.ontap-vlan-interface_nnce.log )
  $ oc get nnce worker-1.ontap-vlan-interface -o jsonpath=' {.status.conditions[?(@.type=="Failing")].message}

  '

• From the NetworkManager logs in the node you'll see similar messages (Attached log worker-1-NetworkManager.log)
  [core@worker-1 ~]$ sudo journalctl -u NetworkManager -f

• Logs from the nmstate-handler pod of the node, also show similar messages (Attached log worker-1-nmstate-handler.log)

Actual results:

• VLAN is not setup through NMState policy

• If we setup it manually with nmcli (just to test) it works:

[core@worker-1 ~]$ nmcli c
NAME UUID TYPE DEVICE
ovs-if-br-ex bea2fb68-e23d-40f3-8191-2b77caae4f71 ovs-interface br-ex
Wired Connection e8764ffe-444b-41e0-bace-4fc6a07756ab ethernet eno5np0
Wired Connection e8764ffe-444b-41e0-bace-4fc6a07756ab ethernet enp1s0f4u4
br-ex 458e00ef-79fb-4a62-b5eb-18d3412ba5bf ovs-bridge br-ex
ovs-if-phys0 4b9e1ed3-bee0-4b12-836a-c7e498750368 ethernet ens3f1
ovs-port-br-ex 1ad3bd72-baee-4776-bb00-941fc0f72f4f ovs-port br-ex
ovs-port-phys0 61cf1f80-e6f4-4767-9296-de7aa1936c98 ovs-port ens3f1
Wired Connection 809bf544-3bbf-4f7d-849f-96c0e4a28f8e ethernet –

[core@worker-1 ~]$ sudo nmcli con add type vlan con-name eno5np0.350 ifname eno5np0.350 dev eno5np0 id 350
Connection 'eno5np0.350' (8c519760-10c2-4cd6-a3b3-28ccbfcf44a4) successfully added.

[core@worker-1 ~]$ nmcli c
NAME UUID TYPE DEVICE
ovs-if-br-ex bea2fb68-e23d-40f3-8191-2b77caae4f71 ovs-interface br-ex
eno5np0.350 8c519760-10c2-4cd6-a3b3-28ccbfcf44a4 vlan eno5np0.350
Wired Connection e8764ffe-444b-41e0-bace-4fc6a07756ab ethernet eno5np0
Wired Connection e8764ffe-444b-41e0-bace-4fc6a07756ab ethernet enp1s0f4u4
br-ex 458e00ef-79fb-4a62-b5eb-18d3412ba5bf ovs-bridge br-ex
ovs-if-phys0 4b9e1ed3-bee0-4b12-836a-c7e498750368 ethernet ens3f1
ovs-port-br-ex 1ad3bd72-baee-4776-bb00-941fc0f72f4f ovs-port br-ex
ovs-port-phys0 61cf1f80-e6f4-4767-9296-de7aa1936c98 ovs-port ens3f1
Wired Connection 809bf544-3bbf-4f7d-849f-96c0e4a28f8e ethernet –

[core@worker-1 ~]$ ip a s eno5np0.350
37: eno5np0.350@eno5np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether f4:03:43:cc:c1:90 brd ff:ff:ff:ff:ff:ff
inet 192.168.53.21/24 brd 192.168.53.255 scope global dynamic noprefixroute eno5np0.350
valid_lft 7186sec preferred_lft 7186sec
inet6 fe80::a58f:9bc8:c440:aaee/64 scope link noprefixroute
valid_lft forever preferred_lft forever

Expected results:

NNCP should setup the VLAN interface

Additional info:

• I setup a NodeSelector just to test one server, but it fails too when applying to all nodes.
• I tested with v1alpha1 version too, with same results
• I tried applying the nncp a few dozens of times and one node was successful, but it stops because it fail in the others.