Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-12207

ovs-configuration fails to start after node reboot when close to 50 transparent vlans are configured via nncp

    XMLWordPrintable

Details

    • Important
    • No
    • SDN Sprint 235, SDN Sprint 236, SDN Sprint 237
    • 3
    • Rejected
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      Customer is trying to configure Transparent VLANs using NNCP policy.
      
      There is however an issue with the nm-dispatcher scripts which causes the resolv.conf to become empty(!).
      
      NetworkManager calls nm-dispatcher for each and every transparent vlan.
      When resolv.conf is empty, the dispatcher waits for a nameserver, and eventually times out after 90 seconds.
      Each vlan thereafter will cause ovs-configuration waiting for an additional 90 seconds, until a timeout is reached starting br-ex.
      
      Eventually ovs-configuration failes with error code and while the system is 'Ready', it's unable to operate normally (eg, MCP update is stuck).
      
      
      node-logs show:
      
      ```
      Mar 21 13:31:57.191951 worker-0.cp-1-cluster.local nm-dispatcher[13358]: NM resolv-prepender: Prepending 'nameserver 192.168.120.46' to /etc/resolv.conf (other nameservers from /var/run/NetworkManager/resolv.conf)
      Mar 21 13:31:57.349128 worker-0.cp-1-cluster.local nm-dispatcher[13513]: NM resolv-prepender: NM resolv.conf still empty of nameserver
      Mar 21 13:31:57.850758 worker-0.cp-1-cluster.local nm-dispatcher[13513]: NM resolv-prepender: NM resolv.conf still empty of nameserver
      ...
      Mar 21 13:32:26.946586 worker-0.cp-1-cluster.local nm-dispatcher[13513]: NM resolv-prepender: NM resolv.conf still empty of nameserver
      Mar 21 13:32:27.486351 worker-0.cp-1-cluster.local nm-dispatcher[13662]: NM resolv-prepender: NM resolv.conf still empty of nameserver
      ...
      Mar 21 13:32:57.086134 worker-0.cp-1-cluster.local nm-dispatcher[13662]: NM resolv-prepender: NM resolv.conf still empty of nameserver
      Mar 21 13:32:57.630174 worker-0.cp-1-cluster.local nm-dispatcher[13811]: NM resolv-prepender: NM resolv.conf still empty of nameserver
      ...
      Mar 21 13:33:04.654262 worker-0.cp-1-cluster.local nm-dispatcher[13811]: NM resolv-prepender: NM resolv.conf still empty of nameserver

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      Customer reproduced on their environment

      Steps to Reproduce:

      1. Use the NNCP to create transparent VLANs
      2. Customer refferd option C of this KBase--> https://access.redhat.com/solutions/6957685
      3. They are using similar to below NNCP. Not full content

       

      apiVersion: nmstate.io/v1
      kind: NodeNetworkConfigurationPolicy
      metadata: {name: 10-worker-network-vlan-data-25g-vlans}
      spec:
        desiredState:
          interfaces:
          - description: data_25g.3003
            name: data_25g.3003
            state: up
            type: vlan
            vlan: {base-iface: data_25g, id: 3003}
          - bridge:
              options:
                stp: {enabled: false}
              port:
              - name: data_25g.3003
                vlan: {}
            description: data_25g_v_3003
            ipv4: {dhcp: false, enabled: false}
            ipv6: {enabled: false}
            mtu: 9000
            name: data_25g_v_3003
            state: up
            type: linux-bridge

       
      Actual results:

      ovs-configuration.service fails. Below are node logs
      
      Mar 21 13:31:57.191951 worker-0.cp-1-cluster.local nm-dispatcher[13358]: NM resolv-prepender: Prepending 'nameserver 192.168.120.46' to /etc/resolv.conf (other nameservers from /var/run/NetworkManager/resolv.conf) Mar 21 13:31:57.349128 worker-0.cp-1-cluster.local nm-dispatcher[13513]: NM resolv-prepender: NM resolv.conf still empty of nameserver Mar 21 13:31:57.850758 worker-0.cp-1-cluster.local nm-dispatcher[13513]: NM resolv-prepender: NM resolv.conf still empty of nameserver ... Mar 21 13:32:26.946586 worker-0.cp-1-cluster.local nm-dispatcher[13513]: NM resolv-prepender: NM resolv.conf still empty of nameserver Mar 21 13:32:27.486351 worker-0.cp-1-cluster.local nm-dispatcher[13662]: NM resolv-prepender: NM resolv.conf still empty of nameserver ... Mar 21 13:32:57.086134 worker-0.cp-1-cluster.local nm-dispatcher[13662]: NM resolv-prepender: NM resolv.conf still empty of nameserver Mar 21 13:32:57.630174 worker-0.cp-1-cluster.local nm-dispatcher[13811]: NM resolv-prepender: NM resolv.conf still empty of nameserver ... Mar 21 13:33:04.654262 worker-0.cp-1-cluster.local nm-dispatcher[13811]: NM resolv-prepender: NM resolv.conf still empty of nameserver

      Expected results:

      ovs-configuration.service running & node getting resolv.conf populated

      Additional info:

       

      Attachments

        Activity

          People

            bnemec@redhat.com Benjamin Nemec
            rhn-support-adubey Akash Dubey
            Anurag Saxena Anurag Saxena
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: