Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32792

IPI-openstack installation fail because nameserver of /etc/resolv.conf on bootstrap node is not set properly

XMLWordPrintable

    • Important
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      My customer is trying to install OCP 4.15 IPv4/v6 dual stack with IPv6 primary using IPI-OpenStack (platform: openstack) on OSP 17.1.
      However, it fails with the following error
      
      ~~~
      $ ./openshift-install create cluster --dir ./
         :
      ERROR: Bootstrap failed to complete: Get "https://api.openshift.example.com:6443/version": dial tcp [2001:db8::5]:6443: i/o timeout
      ~~~
      
      On the bootstrap node, the VIP "2001:db8::5" is not set.
      
      ~~~
      $ ip addr
            :
      2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
          link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
          inet 10.0.0.3/24 brd 10.0.0.254 scope global dynamic noprefixroute enp3s0
             valid_lft 40000sec preferred_lft 40000sec
          inet6 2001:db8::3/128 scope global noprefixroute 
             valid_lft forever preferred_lft forever
          inet6 fe80::aaaa:aaff:feaa:aaaa/64 scope link noprefixroute 
             valid_lft forever preferred_lft forever
      ~~~
      
      As far as I investigated, the reason why VIP is not set is that "nameserver" is not properly set on /etc/resolv.conf.
      Because of this, name resolution doesn't work on the bootstrap node.
      
      ~~~
      $ cat /etc/resolv.conf
      # Generated by NetworkManager
      nameserver 127.0.0.1
      search openshift.example.com
      ~~~
      ==> There should be a nameserver entry which is advertised by DHCPv6 or DHCPv4. However, there is only 127.0.0.1
      
      /var/run/NetworkManager/resolv.conf has a proper "nameserver" entry which is advertised by DHCPv6:
      
      ~~~
      # cat /var/run/NetworkManager/resolv.conf
      # Generated by NetworkManager
      search openshift.example.com
      nameserver 2001:db8::8888
      ~~~
      
      In IPI-openstack installation, /etc/resolv.conf is generated from /var/run/NetworkManager/resolv.conf by the following script:
      
      https://github.com/openshift/installer/blob/9938156e81b5c0085774b2ec56a4be075413fd2d/data/data/bootstrap/openstack/files/etc/NetworkManager/dispatcher.d/30-local-dns-prepender
      
      I'm wondering if the above script doesn't work well due to timing issue, race condition or something like that.
      
      And according to the customer, this issue depends on DNS setting.
      
      - When DNS server info is advertised only by IPv4 DHCP: The issue occurs
      - When DNS server info is advertised only by IPv6 DHCP: The issue occurs
      - When DNS server info is advertised by both IPv4 and IPv6 DHCP: The issue does NOT occurs

      Version-Release number of selected component (if applicable):

          OCP 4.15 IPI-OpenStack

      How reproducible:

      Steps to Reproduce:

          1. Create a provider network on OSP 17.1
          2. Create IPv4 subnet and IPv6 subnet on the provider network
          3. Create set dns-nameserver setting using "openstack subnet set --dns-nameserver" command only on either of IPv4 subnet or IPv6 subnet 
          4. Run IPI-OpenStack installation on the provider network

      Actual results:

          IPI-openstack installation fails because nameserver of /etc/resolv.conf on bootstrap node is not set properly

      Expected results:

          IPI-openstack installation succeeds and nameserver of /etc/resolv.conf on bootstrap node is set properly

      Additional info:

          

            pprinett@redhat.com Pierre Prinetti
            rhn-support-yatanaka Yamato Tanaka
            Itshak Brown Itshak Brown
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: