Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-41509

cannot modify br-ex configuration via nncp when using configure-ovs alternative in day1

XMLWordPrintable

    • Critical
    • Yes
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      cannot modify br-ex configuration via nncp when using configure-ovs alternative in day1, the nncp always in Progressing, and node become NotReady.

      Version-Release number of selected component (if applicable):

      4.17.0, 4.16.z

      How reproducible:

      always

      Steps to Reproduce:

      1. setup bm with dev-script, add customerized br-ex configuration under extra-assets directory 
      # pwd
      /root/qiowang/opnet282/dev-scripts/extra-assets/nmstate-configuration
      # ls -l
      total 8
      -rw-r--r--. 1 root root 2554 Jun 18 06:11 nmstate-br-ex-master.yaml [1]
      -rw-r--r--. 1 root root 2554 Jun 18 06:12 nmstate-br-ex.yaml [2]
      
      
      2. installation done, check on one worker node, br-ex configured correctly
      # oc debug node/worker-0
      Starting pod/worker-0-debug-d54wb ...
      To use host binaries, run `chroot /host`
      Pod IP: 192.168.111.23
      If you don't see a command prompt, try pressing enter.
      sh-5.1# chroot /host
      sh-5.1# nmcli con show
      NAME              UUID                                  TYPE           DEVICE 
      br-ex-if          2225e79b-21f5-4f8e-b2a8-9f5e5f60b865  ovs-interface  br-ex 
      Wired Connection  5b6d4977-1cb4-494d-a8d0-d41d2dc8a769  ethernet       enp1s0 
      br-ex-br          ffd2411a-1945-4fb6-98ef-9739bbf81cb3  ovs-bridge     br-ex 
      br-ex-port        7654760b-9f4f-428b-b2d5-9af0c301d5c9  ovs-port       br-ex 
      enp2s0            b5b18af2-d977-4074-9cdd-b958074fa95d  ethernet       enp2s0 
      enp2s0-port       3991c1eb-c620-4303-8c44-9a0397a23a65  ovs-port       enp2s0 
      lo                7217b3fa-ab86-4f3b-b7c9-6c99d07be5b0  loopback       lo     
      Wired Connection  b5bc4fa3-d0b7-4f19-8ce0-b3682bae0406  ethernet       --     
      sh-5.1# ip a show br-ex
      5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
          link/ether 00:a4:b1:df:24:37 brd ff:ff:ff:ff:ff:ff
          inet 192.168.111.23/24 brd 192.168.111.255 scope global dynamic noprefixroute br-ex
             valid_lft 2265sec preferred_lft 2265sec
          inet 169.254.0.2/17 brd 169.254.127.255 scope global br-ex
             valid_lft forever preferred_lft forever
          inet 192.168.111.4/32 scope global vip
             valid_lft forever preferred_lft forever
      
      
      3. install knmstate operator
      # oc get csv
      NAME                                              DISPLAY                       VERSION               REPLACES   PHASE
      kubernetes-nmstate-operator.4.17.0-202409041336   Kubernetes NMState Operator   4.17.0-202409041336              Succeeded
      
      
      
      4. apply nncp containing the same NMState config as the day 1 manifests
      apiVersion: nmstate.io/v1
      kind: NodeNetworkConfigurationPolicy
      metadata:
        name: worker-0-br-ex
      spec:
        nodeSelector:
          kubernetes.io/hostname: worker-0
        desiredState:
          interfaces:
          - name: enp2s0
            type: ethernet
            state: up
            ipv4:
              enabled: false
            ipv6:
              enabled: false
          - name: br-ex
            type: ovs-bridge
            state: up
            ipv4:
              enabled: false
              dhcp: false
            ipv6:
              enabled: false
              dhcp: false
            bridge:
              port:
              - name: enp2s0
              - name: br-ex
          - name: br-ex
            type: ovs-interface
            state: up
            copy-mac-from: enp2s0
            #mtu: 1550
            ipv4:
              enabled: true
              dhcp: true
              address:
              - ip: "169.254.169.2"
                prefix-length: 29
            ipv6:
              enabled: false
              dhcp: false 

      Actual results:

      nncp always in Progressing, node become NotReady
      
      # oc get nncp
      NAME             STATUS        REASON
      worker-0-br-ex   Progressing   ConfigurationProgressing
      # oc get nnce
      NAME                      STATUS        STATUS AGE   REASON
      worker-0.worker-0-br-ex   Progressing   15m          ConfigurationProgressing
      
      [core@worker-0 ~]$$ nmcli con show
      NAME              UUID                                  TYPE           DEVICE
      Wired Connection  5b6d4977-1cb4-494d-a8d0-d41d2dc8a769  ethernet       enp1s0
      br-ex-br          ffd2411a-1945-4fb6-98ef-9739bbf81cb3  ovs-bridge     br-ex
      br-ex-port        7654760b-9f4f-428b-b2d5-9af0c301d5c9  ovs-port       br-ex
      enp2s0            b5b18af2-d977-4074-9cdd-b958074fa95d  ethernet       enp2s0
      enp2s0-port       3991c1eb-c620-4303-8c44-9a0397a23a65  ovs-port       enp2s0
      br-ex-if          2225e79b-21f5-4f8e-b2a8-9f5e5f60b865  ovs-interface  br-ex
      lo                7217b3fa-ab86-4f3b-b7c9-6c99d07be5b0  loopback       lo
      Wired Connection  b5bc4fa3-d0b7-4f19-8ce0-b3682bae0406  ethernet       --
      [core@worker-0 ~]$
      [core@worker-0 ~]$ ip a show br-ex
      24: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
          link/ether 00:a4:b1:df:24:37 brd ff:ff:ff:ff:ff:ff

      Expected results:

      nncp is Available, node is healthy, br-ex configuration is still correct

      Additional info:

      [1]nmstate-br-ex-master.yaml
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: master
        name: 10-br-ex-master
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
            - contents:
                source: data:text/plain;charset=utf-8;base64,aW50ZXJmYWNlczoKLSBuYW1lOiBlbnAyczAKICB0eXBlOiBldGhlcm5ldAogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogIGlwdjY6CiAgICBlbmFibGVkOiBmYWxzZQotIG5hbWU6IGJyLWV4CiAgdHlwZTogb3ZzLWJyaWRnZQogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogICAgZGhjcDogZmFsc2UKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNlCiAgYnJpZGdlOgogICAgcG9ydDoKICAgIC0gbmFtZTogZW5wMnMwCiAgICAtIG5hbWU6IGJyLWV4Ci0gbmFtZTogYnItZXgKICB0eXBlOiBvdnMtaW50ZXJmYWNlCiAgc3RhdGU6IHVwCiAgY29weS1tYWMtZnJvbTogZW5wMnMwCiAgaXB2NDoKICAgIGVuYWJsZWQ6IHRydWUKICAgIGRoY3A6IHRydWUKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNl
              mode: 0644
              overwrite: true
              path: /etc/nmstate/openshift/master-0.yml
            - contents:
                source: data:text/plain;charset=utf-8;base64,aW50ZXJmYWNlczoKLSBuYW1lOiBlbnAyczAKICB0eXBlOiBldGhlcm5ldAogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogIGlwdjY6CiAgICBlbmFibGVkOiBmYWxzZQotIG5hbWU6IGJyLWV4CiAgdHlwZTogb3ZzLWJyaWRnZQogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogICAgZGhjcDogZmFsc2UKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNlCiAgYnJpZGdlOgogICAgcG9ydDoKICAgIC0gbmFtZTogZW5wMnMwCiAgICAtIG5hbWU6IGJyLWV4Ci0gbmFtZTogYnItZXgKICB0eXBlOiBvdnMtaW50ZXJmYWNlCiAgc3RhdGU6IHVwCiAgY29weS1tYWMtZnJvbTogZW5wMnMwCiAgaXB2NDoKICAgIGVuYWJsZWQ6IHRydWUKICAgIGRoY3A6IHRydWUKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNl
              mode: 0644
              overwrite: true
              path: /etc/nmstate/openshift/master-1.yml
            - contents:
                source: data:text/plain;charset=utf-8;base64,aW50ZXJmYWNlczoKLSBuYW1lOiBlbnAyczAKICB0eXBlOiBldGhlcm5ldAogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogIGlwdjY6CiAgICBlbmFibGVkOiBmYWxzZQotIG5hbWU6IGJyLWV4CiAgdHlwZTogb3ZzLWJyaWRnZQogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogICAgZGhjcDogZmFsc2UKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNlCiAgYnJpZGdlOgogICAgcG9ydDoKICAgIC0gbmFtZTogZW5wMnMwCiAgICAtIG5hbWU6IGJyLWV4Ci0gbmFtZTogYnItZXgKICB0eXBlOiBvdnMtaW50ZXJmYWNlCiAgc3RhdGU6IHVwCiAgY29weS1tYWMtZnJvbTogZW5wMnMwCiAgaXB2NDoKICAgIGVuYWJsZWQ6IHRydWUKICAgIGRoY3A6IHRydWUKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNl
              mode: 0644
              overwrite: true
              path: /etc/nmstate/openshift/master-2.yml
      
      
      [2]nmstate-br-ex.yaml
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: worker
        name: 10-br-ex-worker
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
            - contents:
                source: data:text/plain;charset=utf-8;base64,aW50ZXJmYWNlczoKLSBuYW1lOiBlbnAyczAKICB0eXBlOiBldGhlcm5ldAogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogIGlwdjY6CiAgICBlbmFibGVkOiBmYWxzZQotIG5hbWU6IGJyLWV4CiAgdHlwZTogb3ZzLWJyaWRnZQogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogICAgZGhjcDogZmFsc2UKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNlCiAgYnJpZGdlOgogICAgcG9ydDoKICAgIC0gbmFtZTogZW5wMnMwCiAgICAtIG5hbWU6IGJyLWV4Ci0gbmFtZTogYnItZXgKICB0eXBlOiBvdnMtaW50ZXJmYWNlCiAgc3RhdGU6IHVwCiAgY29weS1tYWMtZnJvbTogZW5wMnMwCiAgaXB2NDoKICAgIGVuYWJsZWQ6IHRydWUKICAgIGRoY3A6IHRydWUKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNl
              mode: 0644
              overwrite: true
              path: /etc/nmstate/openshift/worker-0.yml
            - contents:
                source: data:text/plain;charset=utf-8;base64,aW50ZXJmYWNlczoKLSBuYW1lOiBlbnAyczAKICB0eXBlOiBldGhlcm5ldAogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogIGlwdjY6CiAgICBlbmFibGVkOiBmYWxzZQotIG5hbWU6IGJyLWV4CiAgdHlwZTogb3ZzLWJyaWRnZQogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogICAgZGhjcDogZmFsc2UKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNlCiAgYnJpZGdlOgogICAgcG9ydDoKICAgIC0gbmFtZTogZW5wMnMwCiAgICAtIG5hbWU6IGJyLWV4Ci0gbmFtZTogYnItZXgKICB0eXBlOiBvdnMtaW50ZXJmYWNlCiAgc3RhdGU6IHVwCiAgY29weS1tYWMtZnJvbTogZW5wMnMwCiAgaXB2NDoKICAgIGVuYWJsZWQ6IHRydWUKICAgIGRoY3A6IHRydWUKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNl
              mode: 0644
              overwrite: true
              path: /etc/nmstate/openshift/worker-1.yml
            - contents:
                source: data:text/plain;charset=utf-8;base64,aW50ZXJmYWNlczoKLSBuYW1lOiBlbnAyczAKICB0eXBlOiBldGhlcm5ldAogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogIGlwdjY6CiAgICBlbmFibGVkOiBmYWxzZQotIG5hbWU6IGJyLWV4CiAgdHlwZTogb3ZzLWJyaWRnZQogIHN0YXRlOiB1cAogIGlwdjQ6CiAgICBlbmFibGVkOiBmYWxzZQogICAgZGhjcDogZmFsc2UKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNlCiAgYnJpZGdlOgogICAgcG9ydDoKICAgIC0gbmFtZTogZW5wMnMwCiAgICAtIG5hbWU6IGJyLWV4Ci0gbmFtZTogYnItZXgKICB0eXBlOiBvdnMtaW50ZXJmYWNlCiAgc3RhdGU6IHVwCiAgY29weS1tYWMtZnJvbTogZW5wMnMwCiAgaXB2NDoKICAgIGVuYWJsZWQ6IHRydWUKICAgIGRoY3A6IHRydWUKICBpcHY2OgogICAgZW5hYmxlZDogZmFsc2UKICAgIGRoY3A6IGZhbHNl
              mode: 0644
              overwrite: true
              path: /etc/nmstate/openshift/worker-2.yml
      
      
      NetworkManager & nmstate version on the node:
      [core@worker-0 ~]$ rpm -qa | grep NetworkManager
      NetworkManager-libnm-1.46.0-18.el9_4.x86_64
      NetworkManager-1.46.0-18.el9_4.x86_64
      NetworkManager-team-1.46.0-18.el9_4.x86_64
      NetworkManager-tui-1.46.0-18.el9_4.x86_64
      NetworkManager-cloud-setup-1.46.0-18.el9_4.x86_64
      NetworkManager-ovs-1.46.0-18.el9_4.x86_64
      [core@worker-0 ~]$
      [core@worker-0 ~]$ rpm -qa | grep nmstate
      nmstate-2.2.33-1.el9_4.x86_64

              bnemec@redhat.com Benjamin Nemec
              rh-ee-qiowang Qiong Wang
              Qiong Wang Qiong Wang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: