What were you trying to do that didn't work?

There is no option to route between NAD ipvlan and NodeNetworkConfigurationPolicy with macvlan.  NodeNetworkConfigurationPolicy is not support ipvlan

Expected results

Please provide the package NVR for which bug is seen:
1.Deploy OCP4.16 [4.16.0-0.nightly-2024-06-14-130320] on BM
2.Install LSO4.16 [local-storage-operator.v4.16.0-202406061206]
3.Install NMStaste4.16 [kubernetes-nmstate-operator.4.16.0-202406061206] and create instance
4.Install ODF4.16 [odf-operator.v4.16.0-127.stable]
How reproducible:
Steps to reproduce

1.Configure NAD with ipvlan
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  creationTimestamp: "2024-04-30T14:28:31Z"
  generation: 3
  name: public-net
  namespace: default
  resourceVersion: "291428"
  uid: 74ccacd3-0d5a-4743-b182-39fdf5a21974
spec:
  config: '{ "cniVersion": "0.3.1", "type": "ipvlan", "master": "enp1s0f1", "ipam":

]} }'

2.Installed NMState Operator and create an instance 

3.Configure NodeNetworkConfigurationPolicy
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ceph-public-net-shim-argo005
  namespace: openshift-storage
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ""
    kubernetes.io/hostname: argo005.ceph.redhat.com
  desiredState:
    interfaces:
      - name: odf-pub-shim
        description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
        type: mac-vlan
        state: up
        mac-vlan:
          base-iface: enp1s0f1
          mode: bridge
          promiscuous: true
        ipv4:
          enabled: true
          dhcp: false
          address:
            - ip: 192.168.252.1 # STATIC IP FOR compute-0
              prefix-length: 24
    routes:
      config:
        - destination: 192.168.20.0/24
          next-hop-interface: odf-pub-shim

4.The  NodeNetworkConfigurationPolicy failed to configure
$ oc get NodeNetworkConfigurationPolicy  ceph-public-net-shim-argo005
NAME                           STATUS     REASON
ceph-public-net-shim-argo005   Degraded   FailedToConfigure

[2024-06-18T16:54:17Z INFO  nmstate::nm::query_apply::profile] Will retry activation 16 seconds
[2024-06-18T16:54:33Z INFO  nmstate::nm::query_apply::profile] Activating connection 3fcb9a02-69e0-4cae-b267-6fb7c0d23a3a: odf-pub-shim/macvlan
[2024-06-18T16:54:33Z INFO  nmstate::nm::query_apply::profile] Got activation failure Bug: Device(CreationFailed): odf-pub-shim failed to create resources: Failed to create macvlan interface 'odf-pub-shim' for 'odf-pub-shim': Device or resource busy
[2024-06-18T16:54:33Z INFO  nmstate::nm::query_apply::profile] Will retry activation 32 seconds
[2024-06-18T16:55:05Z INFO  nmstate::nm::query_apply::profile] Activating connection 3fcb9a02-69e0-4cae-b267-6fb7c0d23a3a: odf-pub-shim/macvlan
[2024-06-18T16:55:05Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/1
NmstateError: Bug: Device(CreationFailed): odf-pub-shim failed to create resources: Failed to create macvlan interface 'odf-pub-shim' for 'odf-pub-shim': Device or resource busy

5.Try to change the macvlan block to ipvlan:
        mac-vlan:
          base-iface: enp1s0f1
          mode: bridge
          promiscuous: true

=>

        ipvlan:
          base-iface: enp1s0f1
          mode: l3

6.Got this  error:
Events:
  Type     Reason           Age   From                                     Message
  ----     ------           ----  ----                                     -------
  Warning  ReconcileFailed  36s   argo005.ceph.redhat.com.nmstate-handler  error reconciling NodeNetworkConfigurationPolicy on node argo005.ceph.redhat.com at desired state apply: "",
 failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' '[2024-06-19T08:15:48Z INFO  nmstatectl] Nmstate version: 2.2.29
Using 'set' is deprecated, use 'apply' instead.
Provide file is not valid NetworkState or NetworkPolicy: interfaces: unknown field `ipvlan` at line 2 column 1
'

Full NodeNetworkConfigurationPolicy CR yaml:
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ceph-public-net-shim-argo005
  namespace: openshift-storage
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ""
    kubernetes.io/hostname: argo005.ceph.redhat.com
  desiredState:
    interfaces:
      - name: odf-pub-shim
        description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
        type: mac-vlan
        state: up
        ipvlan:
          base-iface: enp1s0f1
          mode: l3
        ipv4:
          enabled: true
          dhcp: false
          address:
            - ip: 192.168.252.1 # STATIC IP FOR compute-0
              prefix-length: 24
    routes:
      config:
        - destination: 192.168.20.0/24
          next-hop-interface: odf-pub-shim
—

There is no option to route between NAD ipvlan and NodeNetworkConfigurationPolicy with macvlan

Expected results

Actual results