Issue with NMState in 4.14 and bondings with upgrade of EUS(EUS upgrade from version 4.12.47, aiming for version 4.14.14) where we had to delete all the connections and manually set the configuration using nmstatectl to make it work again. 

All network interfaces were then correctly set and configured, allowing us to complete the upgrade, this required us to pause the upgrade process after each node reboot to adjust the network configuration before proceeding. I will share some data to see if someone can share some light in case any buggy behaviour ir observed

    OCP 4.14.14

Status before we started with the upgrade:


$ oc get nncp; oc get nnce
NAME                      STATUS      REASON
bond-dell-r740            Available   SuccessfullyConfigured
bond-dell-r750            Available   SuccessfullyConfigured
bond-hpe-dl380gen10       Available   NoMatchingNode
bond-hpe-dl380gen10plus   Available   NoMatchingNode
NAME                                     STATUS      REASON
occ01bm01.infra.cc.corp.bond-dell-r740   Available   SuccessfullyConfigured
occ01bm02.infra.cc.corp.bond-dell-r740   Available   SuccessfullyConfigured
occ01bm03.infra.cc.corp.bond-dell-r740   Available   SuccessfullyConfigured
occ01bm04.infra.cc.corp.bond-dell-r750   Available   SuccessfullyConfigured
occ01i01.infra.cc.corp.bond-dell-r740    Available   SuccessfullyConfigured
occ01i02.infra.cc.corp.bond-dell-r740    Available   SuccessfullyConfigured
occ01i03.infra.cc.corp.bond-dell-r740    Available   SuccessfullyConfiguredAll

Worker nodes were drained rebooted prior to the upgrade, and all of them booted up correctly
Upgraded to 4.13.35 while the worker nodes were paused
Upgraded the NMState Operator 4.12.0-202402211639 > 4.13.0-202402211707
Upgraded the MetalLB Operator 4.12.0-202402081808 > v4.13.0-202402081808
nnce status after operators upgrade:

oc get nncp; oc get nnce
NAME                      STATUS      REASON
bond-dell-r740            Degraded    FailedToConfigure
bond-dell-r750            Degraded    FailedToConfigure
bond-hpe-dl380gen10       Available   NoMatchingNode
bond-hpe-dl380gen10plus   Available   NoMatchingNode
NAME                                     STATUS    REASON
occ01bm01.infra.cc.corp.bond-dell-r740   Aborted   ConfigurationAborted
occ01bm02.infra.cc.corp.bond-dell-r740   Aborted   ConfigurationAborted
occ01bm03.infra.cc.corp.bond-dell-r740   Failing   FailedToConfigure
occ01bm04.infra.cc.corp.bond-dell-r750   Failing   FailedToConfigure
occ01i01.infra.cc.corp.bond-dell-r740    Aborted   ConfigurationAborted
occ01i02.infra.cc.corp.bond-dell-r740    Aborted   ConfigurationAborted
occ01i03.infra.cc.corp.bond-dell-r740    Aborted   ConfigurationAborted

- Bond down


ip a |grep br1-ext
11: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue master br1-ext state DOWN group default qlen 1000
13: br1-ext: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000Next, we:
Upgraded to 4.14.14 while the worker nodes were paused
Upgraded the NMState Operator 4.13.0-202402211707 > 4.14.0-202402091039
Upgraded the MetalLB Operator 4.13.0-202402081808 > v4.14.0-202402161909
nnce status after operators upgrade:

oc get nncp; oc get nnce
NAME                      STATUS      REASON
bond-dell-r740            Degraded    FailedToConfigure
bond-dell-r750            Degraded    FailedToConfigure
bond-hpe-dl380gen10       Available   NoMatchingNode
bond-hpe-dl380gen10plus   Available   NoMatchingNode
NAME                                     STATUS      STATUS AGE   REASON
occ01bm01.infra.cc.corp.bond-dell-r740   Available   84s          SuccessfullyConfigured
occ01bm02.infra.cc.corp.bond-dell-r740   Aborted     5m33s        ConfigurationAborted
occ01bm03.infra.cc.corp.bond-dell-r740   Available   5m3s         SuccessfullyConfigured
occ01bm04.infra.cc.corp.bond-dell-r750   Failing     2m23s        FailedToConfigure
occ01i01.infra.cc.corp.bond-dell-r740    Available   110s         SuccessfullyConfigured
occ01i02.infra.cc.corp.bond-dell-r740    Available   67s          SuccessfullyConfigured
occ01i03.infra.cc.corp.bond-dell-r740    Available   2m9s         SuccessfullyConfigured
Finally, we resume the upgrade on worker nodes.

    1.Config (nmstate was complaining about was sriov not being enabled, it is disabled in bios) (edited) nodenetworkconfigurationenactment-ocs05i03.infra.dc.corp.bond-dell-r740.yaml 

apiVersion: nmstate.io/v1beta1
kind: NodeNetworkConfigurationEnactment
metadata:
  creationTimestamp: '2023-06-20T11:55:12Z'
  generation: 1
  labels:
    nmstate.io/node: ocs05i03.infra.dc.corp
    nmstate.io/policy: bond-dell-r740
  managedFields:
    - apiVersion: nmstate.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:nmstate.io/node': {}
            'f:nmstate.io/policy': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"26e4ac58-b636-485f-99cd-cbd10b6abc5f"}': {}
      manager: manager
      operation: Update
      time: '2023-06-20T11:55:12Z'
    - apiVersion: nmstate.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          .: {}
          'f:conditions': {}
          'f:desiredState':
            .: {}
            'f:interfaces': {}
          'f:desiredStateMetaInfo': {}
          'f:policyGeneration': {}
      manager: manager
      operation: Update
      subresource: status
      time: '2024-03-07T13:31:24Z'
  name: ocs05i03.infra.dc.corp.bond-dell-r740
  ownerReferences:
    - apiVersion: v1
      kind: Node
      name: ocs05i03.infra.dc.corp
      uid: 26e4ac58-b636-485f-99cd-cbd10b6abc5f
  resourceVersion: '417002817'
  uid: 8a768628-0a27-4b5b-b261-cbf138d74e50
status:
  conditions:
    - lastHeartbeatTime: '2024-03-07T13:31:24Z'
      lastTransitionTime: '2024-03-07T13:31:24Z'
      reason: FailedToConfigure
      status: 'False'
      type: Progressing
    - lastHeartbeatTime: '2024-03-07T13:31:24Z'
      lastTransitionTime: '2024-03-07T13:31:24Z'
      message: |-
        error reconciling NodeNetworkConfigurationPolicy on node ocs05i03.infra.dc.corp at desired state apply: "",
         failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' 'Using 'set' is deprecated, use 'apply' instead.
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type 
        [2024-03-07T13:28:49Z WARN  nmstate::ovsdb::show] Unknown OVS interface type vxlan
        [2024-03-07T13:28:49Z WARN  nmstate::ifaces::bond] The `slaves` is deprecated, please replace with `ports`.
        [2024-03-07T13:28:49Z WARN  nmstate::ifaces::bond] The `slaves` is deprecated, please replace with `ports`.
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth0cdceffb type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface vethcbf7cc68 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth31bb6df0 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth26d06c08 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth924187d3 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface vethaf470c09 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface vethb8d13ba5 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth72ca94ae type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface vetha2d66180 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth0522b895 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth939b87e7 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface vethc7f1ea5d type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface br0 type ovs-interface
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface tun0 type ovs-interface
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth902dc596 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface vxlan_sys_4789 type vxlan
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth0e8f551a type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface veth8dffcc94 type ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/2
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("c7352604-9b70-4e69-b9f5-0975d7d21d4e"), ID Some("bond0"), type Some("bond") name Some("bond0")
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("7c5eaf74-6120-4366-b469-a3fdfe86c7bc"), ID Some("bond1"), type Some("bond") name Some("bond1")
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("f4f5a693-a30a-4ad0-b47f-6ed1802436fb"), ID Some("br1-ext"), type Some("bridge") name Some("br1-ext")
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("dbae7c4d-7c9a-4883-a0a2-38bdb3620029"), ID Some("bond0.1713"), type Some("vlan") name Some("bond0.1713")
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("87c4826e-5052-3de8-85e4-b01584d49c90"), ID Some("Wired connection 3"), type Some("802-3-ethernet") name Some("eno1np0")
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("dfce3def-6ebd-3bf6-bd48-1aab4e580e2a"), ID Some("Wired connection 4"), type Some("802-3-ethernet") name Some("eno2np1")
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("e4d42a7a-7743-4afb-8a39-3b1ded71dd0f"), ID Some("ens7f0np0"), type Some("802-3-ethernet") name Some("ens7f0np0")
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("3f5fdc20-fd80-4fc0-9589-0e85be6c96ba"), ID Some("ens7f1np1"), type Some("802-3-ethernet") name Some("ens7f1np1")
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Activating connection c7352604-9b70-4e69-b9f5-0975d7d21d4e: bond0/bond
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Activating connection 7c5eaf74-6120-4366-b469-a3fdfe86c7bc: bond1/bond
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Activating connection f4f5a693-a30a-4ad0-b47f-6ed1802436fb: br1-ext/bridge
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Activating connection dbae7c4d-7c9a-4883-a0a2-38bdb3620029: bond0.1713/vlan
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 87c4826e-5052-3de8-85e4-b01584d49c90: eno1np0/802-3-ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Reapplying connection dfce3def-6ebd-3bf6-bd48-1aab4e580e2a: eno2np1/802-3-ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Activating connection e4d42a7a-7743-4afb-8a39-3b1ded71dd0f: ens7f0np0/802-3-ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Activating connection 3f5fdc20-fd80-4fc0-9589-0e85be6c96ba: ens7f1np1/802-3-ethernet
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Got activation failure Bug: Manager(UnknownDevice): No suitable device found for this connection (device eno1np0 not available because profile is not compatible with device (device does not support SR-IOV)).
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Got activation failure Bug: Manager(UnknownDevice): No suitable device found for this connection (device eno1np0 not available because profile is not compatible with device (device does not support SR-IOV)).
        [2024-03-07T13:28:49Z INFO  nmstate::nm::query_apply::profile] Will retry activation 2 seconds
        [2024-03-07T13:28:51Z INFO  nmstate::nm::query_apply::profile] Activating connection e4d42a7a-7743-4afb-8a39-3b1ded71dd0f: ens7f0np0/802-3-ethernet
        [2024-03-07T13:28:51Z INFO  nmstate::nm::query_apply::profile] Activating connection 3f5fdc20-fd80-4fc0-9589-0e85be6c96ba: ens7f1np1/802-3-ethernet
        [2024-03-07T13:28:51Z INFO  nmstate::nm::query_apply::profile] Got activation failure Bug: Manager(UnknownDevice): No suitable device found for this connection (device eno1np0 not available because profile is not compatible with device (device does not support SR-IOV)).
        [2024-03-07T13:28:51Z INFO  nmstate::nm::query_apply::profile] Got activation failure Bug: Manager(UnknownDevice): No suitable device found for this connection (device eno1np0 not available because profile is not compatible with device (device does not support SR-IOV)).
        [2024-03-07T13:28:51Z INFO  nmstate::nm::query_apply::profile] Will retry activation 4 seconds
        [2024-03-07T13:28:55Z INFO  nmstate::nm::query_apply::profile] Activating connection e4d42a7a-7743-4afb-8a39-3b1ded71dd0f: ens7f0np0/802-3-ethernet
        [2024-03-07T13:28:55Z INFO  nmstate::nm::query_apply::profile] Activating connection 3f5fdc20-fd80-4fc0-9589-0e85be6c96ba: ens7f1np1/802-3-ethernet
        [2024-03-07T13:28:55Z INFO  nmstate::nm::query_apply::profile] Got activation failure Bug: Manager(UnknownDevice): No suitable device found for this connection (device eno1np0 not available because profile is not compatible with device (device does not support SR-IOV)).
        [2024-03-07T13:28:55Z INFO  nmstate::nm::query_apply::profile] Got activation failure Bug: Manager(UnknownDevice): No suitable device found for this connection (device eno1np0 not available because profile is not compatible with device (device does not support SR-IOV)).
        [2024-03-07T13:28:55Z INFO  nmstate::nm::query_apply::profile] Will retry activation 8 seconds
        [2024-03-07T13:29:03Z INFO  nmstate::nm::query_apply::profile] Activating connection e4d42a7a-7743-4afb-8a39-3b1ded71dd0f: ens7f0np0/802-3-ethernet
        [2024-03-07T13:29:03Z INFO  nmstate::nm::query_apply::profile] Activating connection 3f5fdc20-fd80-4fc0-9589-0e85be6c96ba: ens7f1np1/802-3-ethernet
        [2024-03-07T13:29:03Z INFO  nmstate::nm::query_apply::profile] Got activation failure Bug: Manager(UnknownDevice): No suitable device found for this connection (device eno1np0 not available because profile is not compatible with device (device does not support SR-IOV)).
        [2024-03-07T13:29:03Z INFO  nmstate::nm::query_apply::profile] Got activation failure Bug: Manager(UnknownDevice): No suitable device found for this connection (device eno1np0 not available because profile is not compatible with device (device does not support SR-IOV)).
        [2024-03-07T13:29:03Z INFO  nmstate::nm::query_apply::profile] Will retry activation 16 seconds
        [2024-03-07T13:29:19Z INFO  nmstate::nm::query_apply::profile] Activating connection e4d42a7a-7743-4afb-8a39-3b1ded71dd0f: ens7f0np0/802-3-ethernet
        [2024-03-07T13:29:19Z INFO  nmstate::nm::query_apply::profile] Activating connection 3f5fdc20-fd80-4fc0-9589-0e85be6c96ba: ens7f1np1/802-3-ethernet

2024-03-13T13:45:21.754104965Z {"level":"info","ts":"2024-03-13T13:45:21.754Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 6, enactments count: {failed: {true: 1, false: 5, unknown: 0}, progressing: {true: 0, false: 6, unknown: 0}, pending: {true: 0, false: 6, unknown: 0}, available: {true: 0, false: 6, unknown: 0}, aborted: {true: 5, false: 1, unknown: 0}}","policy":"bond-dell-r740"}

2024-03-13T13:45:34.402454577Z {"level":"error","ts":"2024-03-13T13:45:34.402Z","logger":"policyconditions","msg":"failed to reset policy conditions","policy":"bond-dell-r740","error":"Internal error occurred: failed calling webhook \"nodenetworkconfigurationpolicies-timestamp-mutate.nmstate.io\": failed to call webhook: Post \"https://nmstate-webhook.openshift-nmstate.svc:443/nodenetworkconfigurationpolicies-timestamp-mutate?timeout=10s\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"nmstate\")","stacktrace":"github.com/nmstate/kubernetes-nmstate/pkg/policyconditions.Reset.func1\n\t/go/src/github.com/openshift/kubernetes-nmstate/pkg/policyconditions/conditions.go:303\nk8s.io/client-go/util/retry.OnError.func1\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/k8s.io/client-go/util/retry/util.go:51\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:461\nk8s.io/client-go/util/retry.OnError\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/k8s.io/client-go/util/retry/util.go:50\nk8s.io/client-go/util/retry.RetryOnConflict\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/k8s.io/client-go/util/retry/util.go:104\ngithub.com/nmstate/kubernetes-nmstate/pkg/policyconditions.Reset\n\t/go/src/github.com/openshift/kubernetes-nmstate/pkg/policyconditions/conditions.go:291\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/go/src/github.com/openshift/kubernetes-nmstate/controllers/handler/nodenetworkconfigurationpolicy_controller.go:151\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}

2024-03-13T13:45:21.775326622Z {"level":"error","ts":"2024-03-13T13:45:21.775Z","logger":"policyconditions","msg":"failed to update policy status using cached client. Retrying with non-cached.","policy":"bond-dell-r740","error":"Internal error occurred: failed calling webhook \"nodenetworkconfigurationpolicies-status-mutate.nmstate.io\": failed to call webhook: Post \"https://nmstate-webhook.openshift-nmstate.svc:443/nodenetworkconfigurationpolicies-status-mutate?timeout=10s\": no endpoints available for service \"nmstate-webhook\"","stacktrace":"github.com/nmstate/kubernetes-nmstate/pkg/policyconditions.Update\n\t/go/src/github.com/openshift/kubernetes-nmstate/pkg/policyconditions/conditions.go:161\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/go/src/github.com/openshift/kubernetes-nmstate/controllers/handler/nodenetworkconfigurationpolicy_controller.go:205\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/openshift/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}    
    

    Having to reconfigure manually

    Not having to configure it manually, smooth upgrade.

    Logs included in the linked case.