Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-84756

DNS NNCP is stuck in Degraded state during reconsiliation [rhel-10.0.z]

XMLWordPrintable

    • nmstate-2.2.43-1.el10_0
    • No
    • Important
    • 0day
    • rhel-sst-network-management
    • ssg_networking
    • 1
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • Hide

      Definition of Done:

      Please mark each item below with ( / ) if completed or ( x ) if incomplete:

      ( ) The acceptance criteria defined below are met.

      None

      ( ) Code changes are included in a downstream build attached to an errata.

      ( ) All required testing (manual and/or automated) passes successfully.

      ( ) Related documentation updates (if applicable) have been completed.

      Show
      Definition of Done: Please mark each item below with ( / ) if completed or ( x ) if incomplete: ( ) The acceptance criteria defined below are met. None ( ) Code changes are included in a downstream build attached to an errata. ( ) All required testing (manual and/or automated) passes successfully. ( ) Related documentation updates (if applicable) have been completed.
    • Requested
    • None
    • Unspecified
    • Unspecified
    • Unspecified
    • None

      This is a clone of issue RHEL-82269 to use for version rhel-10.0.z
      –
      Original description:
      This is a clone of issue RHEL-82100 to use for version rhel-10.1
      –
      Original description:

      What were you trying to do that didn't work?

      • After a OCP cluster upgrade from 4.12.28 to 4.14.38.
      • The applied NNCP on one of the node is stuck with `Degraded` and `FailedToConfigure` state.
      • Where as same nncp is successfullyconfigured on node only one node is affected.

      What is the impact of this issue to you?

      • NNCP is stuck in failedtoconfigure state

        Please provide the package NVR for which the bug is seen:

        handlerNetworkManagerVersion: 1.30.0-18.el8_4
        handlerNmstateVersion: nmstatectl 2.2.39
        hostNetworkManagerVersion: 1.42.2

      How reproducible is this bug?: In customer environment

      Steps to reproduce

      1.  
      2.  
      3.  

      Expected results : NNCP should successfully configured on node.

      Actual results : It is getting failed with below errors

        - lastHeartbeatTime: "2025-03-04T00:36:46Z"
    lastTransitionTime: "2025-03-04T00:36:46Z"
    message: "error reconciling NodeNetworkConfigurationPolicy on node wat-rtcs-ocpwrk08p-p.rtcs-prod.otg.om
      at desired state apply: \"\",\n failed to execute nmstatectl set --no-commit
      --timeout 480: 'exit status 1' '' '[2025-03-04T00:36:44Z INFO  nmstatectl] Nmstate
      version: 2.2.39\nUsing 'set' is deprecated, use 'apply' instead.\n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type vxlan\n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type vxlan\n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      WARN  nmstate::ovsdb::show] Unknown OVS interface type \n[2025-03-04T00:36:45Z
      INFO  nmstate::ifaces::inter_ifaces] Ignoring interface tun0 type ovs-interface\n[2025-03-04T00:36:45Z
      INFO  nmstate::ifaces::inter_ifaces] Ignoring interface vxlan_sys_4789 type
      vxlan\n[2025-03-04T00:36:45Z INFO  nmstate::ifaces::inter_ifaces] Ignoring interface
      br0 type ovs-interface\n[2025-03-04T00:36:45Z INFO  nmstate::ifaces::inter_ifaces]
      Ignoring interface veth9d36b47e type ethernet\n[2025-03-04T00:36:45Z INFO  nmstate::ifaces::inter_ifaces]
      Ignoring interface veth951a4b86 type ethernet\n[2025-03-04T00:36:45Z INFO  nmstate::ifaces::inter_ifaces]
      Ignoring interface veth24ef9094 type ethernet\n[2025-03-04T00:36:45Z INFO  nmstate::ifaces::inter_ifaces]
      Ignoring interface veth15f6ffbe type ethernet\n[2025-03-04T00:36:45Z INFO  nmstate::ifaces::inter_ifaces]
      Ignoring interface veth8c8cba79 type ethernet\n[2025-03-04T00:36:45Z INFO  nmstate::query_apply::net_state]
      Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/10\n[2025-03-04T00:36:45Z
      INFO  nmstate::nm::query_apply::dns] Using interface level DNS for special use
      case: appending static DNS nameserver before dynamic ones.\n[2025-03-04T00:36:45Z
      INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some(\"20785f41-f25f-4b62-8a88-2f1b6ccba4f8\"),
      ID Some(\"bond0.218\"), type Some(Vlan) name Some(\"bond0.218\")\n[2025-03-04T00:36:45Z
      INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some(\"08a5bca2-4d0c-42af-bd47-beb5239e5dc8\"),
      ID Some(\"bond0.868\"), type Some(Vlan) name Some(\"bond0.868\")\n[2025-03-04T00:36:45Z
      INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/10\nNmstateError:
      InvalidArgument: Connection(MissingSetting):vlan: setting required for connection
      of type 'vlan'\n'"
    reason: FailedToConfigure
    status: "True"
    type: Failing

      Additional Info :

      Applied NNCP as below-

      apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"nmstate.io/v1","kind":"NodeNetworkConfigurationPolicy","metadata":{"annotations":{},"name":"static-route-for-worker8"},"spec":{"desiredState":{"dns-resolver":{"config":{"search":[],"server":["10.164.103.20","10.72.103.20","10.80.203.20"]}},"routes":{"config":[{"destination":"195.145.98.210/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"151.253.156.33/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"212.93.182.12/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"94.200.104.142/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"96.9.152.160/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"52.17.228.11/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.64.163.150/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.64.163.182/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.64.172.30/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.72.84.10/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.164.130.10/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.230.0.1/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.164.159.40/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.164.61.63/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.164.130.72/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.72.223.81/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.64.103.30/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254},{"destination":"10.72.219.31/32","metric":0,"next-hop-address":"10.164.222.22","next-hop-interface":"bond0.868","table-id":254}]}},"nodeSelector":{"kubernetes.io/hostname":"wat-rtcs-ocpwrk08p-p.rtcs-prod.otg.om"}}}
    nmstate.io/webhook-mutating-timestamp: "1741048604380150143"
  creationTimestamp: "2025-03-04T00:36:44Z"
  generation: 1
  name: static-route-for-worker8
  resourceVersion: "1705652026"
  uid: 02e3e188-f29e-4f23-b6eb-fbdb8906fb8b
spec:
  desiredState:
    dns-resolver:
      config:
        search: []
        server:
        - 10.164.103.20
        - 10.72.103.20
        - 10.80.203.20
    routes:
      config:
      - destination: 195.145.98.210/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 151.253.156.33/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 212.93.182.12/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 94.200.104.142/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 96.9.152.160/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 52.17.228.11/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.64.163.150/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.64.163.182/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.64.172.30/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.72.84.10/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.164.130.10/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.230.0.1/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.164.159.40/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.164.61.63/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.164.130.72/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.72.223.81/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.64.103.30/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
      - destination: 10.72.219.31/32
        metric: 0
        next-hop-address: 10.164.222.22
        next-hop-interface: bond0.868
        table-id: 254
  nodeSelector:
    kubernetes.io/hostname: wat-rtcs-ocpwrk08p-p.rtcs-prod.otg.om
status:
  conditions:
  - lastHeartbeatTime: "2025-03-04T00:36:46Z"
    lastTransitionTime: "2025-03-04T00:36:46Z"
    reason: FailedToConfigure
    status: "False"
    type: Available
  - lastHeartbeatTime: "2025-03-04T00:36:46Z"
    lastTransitionTime: "2025-03-04T00:36:46Z"
    message: 1/1 nodes failed to configure
    reason: FailedToConfigure
    status: "True"
    type: Degraded
  - lastHeartbeatTime: "2025-03-04T00:36:46Z"
    lastTransitionTime: "2025-03-04T00:36:46Z"
    reason: ConfigurationProgressing
    status: "False"
    type: Progressing
  lastUnavailableNodeCountUpdate: "2025-03-04T00:36:46Z"

      Nmstate handler logs available in below pastebin link : https://privatebin.corp.redhat.com/?d3047958752a9de7#gerhpzhz9eHt95XJrgRZjrdZ1mnMXi93P354ZPJYHB7

              rh-ee-mshi1 Mingyu Shi
              watson-automation Watson Automation
              se-network management se-network management
              Network Management Team Network Management Team
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: