Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-8619

[1901859] NodeNetworkConfigurationPolicy failed to retrieve default gw - create VLAN interface

XMLWordPrintable

    • No

      Created attachment 1733676 [details]
      NodeNetworkState

      Description of problem:

      We want to create a VLAN interface:

      oc apply -f - <<EOF
      apiVersion: nmstate.io/v1alpha1
      kind: NodeNetworkConfigurationPolicy
      metadata:
      name: vlan-ens3f1-policy
      spec:
      nodeSelector:
      kubernetes.io/hostname: "ocp-master1"
      desiredState:
      interfaces:

      • name: ens3f1.602
        description: VLAN using ens3f1
        type: vlan
        state: up
        vlan:
        base-iface: ens3f1
        id: 602
        EOF

      and got the error: "error reconciling NodeNetworkConfigurationPolicy at desired state apply: , rolling back desired state configuration: failed runnig probes after network changes: failed to retrieve default gw at runProbes: timed out waiting for the condition"

      During the installation, we created a VLAN interface via Live ISO for the OpenShiftSDN:

      [root@ocp-lb ~]# oc debug node/ocp-master1
      Creating debug namespace/openshift-debug-node-m6wfb ...
      Starting pod/ocp-master1-debug ...
      To use host binaries, run `chroot /host`
      Pod IP: 172.29.26.52
      If you don't see a command prompt, try pressing enter.
      sh-4.4# chroot /host
      sh-4.4# nmcli con show
      NAME UUID TYPE DEVICE
      VLAN connection 1 561dcb98-d2e2-49cb-9485-8d43bfbdd1d1 vlan ens2f1.602
      Wired connection 1 00a6d56a-4cd5-3e15-ba80-7a9358522368 ethernet –
      Wired connection 2 b7bab4a4-99f7-379e-bda7-acd3e1e2396d ethernet –
      Wired connection 3 b59db2e4-6794-3d45-ba16-4e32aa2ea89a ethernet –
      Wired connection 4 4172378f-325c-386f-865b-c043539360dd ethernet –
      Wired connection 5 2a8f681d-0481-3382-856f-4a85859b06ab ethernet –
      Wired connection 6 f6bad38f-7e2c-3153-b8a8-a232ee3fec8d ethernet –

      During the NodeNetworkConfigurationPolicy progressing the con show's up:
      sh-4.4# nmcli con show
      NAME UUID TYPE DEVICE
      VLAN connection 1 561dcb98-d2e2-49cb-9485-8d43bfbdd1d1 vlan ens2f1.602
      ens3f1.602 64cebaa5-5d79-435d-af1d-849c5545cd6a vlan ens3f1.602
      ...

      Attached NodeNetworkState and NodeNetworkConfigurationEnactment

      I tried to add
      ipv4:
      enabled: true
      dhcp: false
      to my policy, another error:

      [root@ocp-lb ~]# oc get nnce/ocp-master1.vlan-ens3f1-policy -o json | jq '.status.conditions[0].message' -r
      error reconciling NodeNetworkConfigurationPolicy at desired state apply: , failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' '2020-11-26 09:29:31,434 root DEBUG Checkpoint /org/freedesktop/NetworkManager/Checkpoint/13 created for all devices: 480
      2020-11-26 09:29:31,435 root DEBUG Adding new interfaces: ['ens3f1.602']
      2020-11-26 09:29:31,437 root DEBUG Editing interfaces: ['ens2f1.602']
      2020-11-26 09:29:31,439 root DEBUG Executing NM action: func=add_connection_async
      2020-11-26 09:29:31,455 root DEBUG Connection adding succeeded: dev=ens3f1.602
      2020-11-26 09:29:31,455 root DEBUG Executing NM action: func=commit_changes_async
      2020-11-26 09:29:31,459 root DEBUG Connection update succeeded: dev=ens2f1.602
      2020-11-26 09:29:31,459 root DEBUG Executing NM action: func=_safe_modify_async
      2020-11-26 09:29:31,463 root DEBUG Device reapply succeeded: dev=ens2f1.602
      2020-11-26 09:29:31,463 root DEBUG Executing NM action: func=safe_activate_async
      2020-11-26 09:29:31,465 root DEBUG Connection activation initiated: dev=ens3f1.602, con-state=<enum NM_ACTIVE_CONNECTION_STATE_ACTIVATING of type NM.ActiveConnectionState>
      2020-11-26 09:29:31,484 root DEBUG Connection activation succeeded: dev=ens3f1.602, con-state=<enum NM_ACTIVE_CONNECTION_STATE_ACTIVATED of type NM.ActiveConnectionState>, dev-state=<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState>, state-flags=<flags NM_ACTIVATION_STATE_FLAG_LAYER2_READY | NM_ACTIVATION_STATE_FLAG_IP4_READY | NM_ACTIVATION_STATE_FLAG_IP6_READY of type NM.ActivationStateFlags>
      2020-11-26 09:29:31,484 root DEBUG NM action queue exhausted, quiting mainloop
      2020-11-26 09:29:36,867 root DEBUG Checkpoint /org/freedesktop/NetworkManager/Checkpoint/13 rollback executed: dbus.Dictionary(

      {dbus.String('/org/freedesktop/NetworkManager/Devices/92'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/451'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/96'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/118'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/67'): db us.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/49'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/81'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/127'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/102'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/57'): dbus.UInt32(0), dbus.String('/org/freedesktop/Network Manager/Devices/68'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/91'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/65'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/93'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/95'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/82'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/99'): dbus.UInt32(0), dbus.String('/ org/freedesktop/NetworkManager/Devices/35'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/30'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/34'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/39'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/449'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/64'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/134'): dbus. UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/128'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/452'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/72'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/24'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/141'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/142'): dbus.UInt32(0), dbus.String('/org/freedesktop/Network Manager/Devices/53'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/122'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/146'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/63'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/148'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String(' /org/freedesktop/NetworkManager/Devices/33'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/98'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/140'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/20'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/136'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/55'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus. UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/110'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/108'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/143'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/130'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/145'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/25'): dbus.UInt32(0), dbus.String('/org/freedesktop/Networ kManager/Devices/74'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/23'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/104'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/97'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/29'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/22'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/50'): dbus.UInt32(0), dbus.String( '/org/freedesktop/NetworkManager/Devices/101'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/150'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/459'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/137'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/28'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/54'): db us.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/117'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/36'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/103'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/115'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/66'): dbus.UInt32(0), dbus.String('/org/freedesktop/Networ kManager/Devices/73'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/71'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/51'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/26'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/138'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/58'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/47'): dbus.UInt32(0), dbus.String( '/org/freedesktop/NetworkManager/Devices/144'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/120'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/52'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/121'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/21'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/32'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/135'): d bus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/133'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/119'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/100'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/129'): dbus.UInt32(0), dbus.String('/org/freedesktop/Netwo rkManager/Devices/107'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/139'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/113'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/458'): dbus.UInt32(0)}

      , signature=dbus.Signature('su'))
      Traceback (most recent call last):
      File "/usr/bin/nmstatectl", line 11, in <module>
      load_entry_point('nmstate==0.2.6', 'console_scripts', 'nmstatectl')()
      File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 59, in main
      return args.func(args)
      File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 221, in apply
      return apply_state(statedata, args.verify, args.commit, args.timeout)
      File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 241, in apply_state
      rollback_timeout=timeout,
      File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper
      return func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped
      ret = func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply
      state.State(desired_state), verify_change, commit, rollback_timeout
      File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 175, in _apply_ifaces_state
      _verify_change(desired_state)
      File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 221, in _verify_change
      verifiable_desired_state.verify_interfaces(current_state)
      File "/usr/lib/python3.6/site-packages/libnmstate/state.py", line 331, in verify_interfaces
      self._assert_interfaces_equal(other_state)
      File "/usr/lib/python3.6/site-packages/libnmstate/state.py", line 751, in _assert_interfaces_equal
      current_state.interfaces[ifname],
      libnmstate.error.NmstateVerificationError:
      desired
      =======

      name: ens3f1.602
      type: vlan
      state: up
      description: VLAN using ens3f1
      ipv4:
      dhcp: false
      enabled: true
      ipv6:
      enabled: false
      mac-address: D4:F5:EF:1A:25:78
      mtu: 1500
      vlan:
      base-iface: ens3f1
      id: 602

      current
      =======

      name: ens3f1.602
      type: vlan
      state: up
      description: VLAN using ens3f1
      ipv4:
      enabled: false
      ipv6:
      enabled: false
      mac-address: D4:F5:EF:1A:25:78
      mtu: 1500
      vlan:
      base-iface: ens3f1
      id: 602

      difference
      ==========
      — desired
      +++ current
      @@ -4,8 +4,7 @@
      state: up
      description: VLAN using ens3f1
      ipv4:

      • dhcp: false
      • enabled: true
        + enabled: false
        ipv6:
        enabled: false
        mac-address: D4:F5:EF:1A:25:78

      '

      Attached NodeNetworkState-with-ip and NodeNetworkConfigurationEnactment-with-ip

      Version-Release number of selected component (if applicable):

      CNV: 2.5.1
      OCP: 2.6.4

      How reproducible:

      Steps to Reproduce:
      1. Apply NodeNetworkConfigurationPolicy from above.
      2.
      3.

      Actual results:

      Interface ens3f1.602 is missing

      Expected results:

      Create a ens3f1.602 interface

      Additional info:

      The final setup would be: linux-bridge for VM's, linux bridge is attached to ens3f1.602

      We followed the official documentation: https://docs.openshift.com/container-platform/4.6/virt/node_network/virt-updating-node-network-config.html#virt-example-vlan-nncp_virt-updating-node-network-config

              ellorent Felix Enrique Llorente Pastora
              rbohne Robert Bohne
              Meni Yakove Meni Yakove
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: