Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-224

Missing $SEARCH domain in /etc/resolve.conf for OCP v4.9.31 cluster

    XMLWordPrintable

Details

    • Moderate
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:
      OCP v4.9.31 cluster didn't have the $search domain in /etc/resolv.conf, which was there in the v4.8.29 OCP cluster. This was observed in all the nodes of the v4.9.31 cluster.
      ~~~
      OpenShift 4.9.31
      sh-4.4# cat /etc/resolv.conf

      1. Generated by KNI resolv prepender NM dispatcher script
        nameserver 172.xx.xx.xx
        nameserver 10.xx.xx.xx
        nameserver 10.xx.xx.xx
      2. nameserver 10.xx.xx.xx

      OpenShift 4.8.29

      1. Generated by KNI resolv prepender NM dispatcher script
        search sepia.lab.iad2.dc.paas.redhat.com
        nameserver 172.xx.xx.xx
        nameserver 10.xx.xx.xx
        nameserver 10.xx.xx.xx
      2. nameserver 10.xx.xx.xx
        ~~~

      ENV: OpenStack IAD2, IPI installation. Connected cluster.

      Version-Release number of selected component (if applicable):
      OCP v4.9.31

      How reproducible:
      Always

      Steps to Reproduce:
      1. Install IPI cluster on OpenStack IAD2 platform having cluster version 4.9.31
      2. Debug to any of the node(master/worker)
      3. Check and confirm the missing search domain on all nodes of the cluster.

      Actual results:
      The search domain was missing when checked in `/etc/resolv.conf` file on all nodes of the cluster causing serious issues in the cluster.

      Expected results:
      The installer should embed the search domain in /etc/resolv.conf file on all nodes of the cluster.

      Additional info:

      • Cu was trying to deploy secure Kerberos on the CoreOS nodes and it failed when the IPA-client install command failed. This is when the customer noticed this unusual behavior. They did not manually update the resolv.conf file to include the $search domain. They instead added the script below to /etc/NetworkManager/dispatcher.d/ and restarted NetworkManager on the node to fix this issue and installation was successful.
        ~~~
        #!/bin/bash

      set -eo pipefail

      DISPATCHER_FILE="/etc/NetworkManager/dispatcher.d/30-resolv-prepender"
      DOMAINS="$(grep -E '\s*DOMAINS=.*iad2.dc.paas.redhat.com' $DISPATCHER_FILE \

      grep -oE '[a-z0-9]*.dev.iad2.dc.paas.redhat.com' \
      tr '\n' ' ')"

      >&2 echo "IT-PaaS: overwriting search domains in /etc/resolv.conf with: $DOMAINS"

      sed -e "/^search/d" \
      -e "/Generated by/c# Generated by KNI resolv prepender NM dispatcher script \nsearch $DOMAINS" \
      /etc/resolv.conf > /etc/resolv.tmp

      mv /etc/resolv.tmp /etc/resolv.conf
      ~~~

      • Cu confirms that the $search domain was missing since the cluster was freshly installed/ They even confirmed this with a fresh new cluster as well that it was missing.
      • The fresh cluster was initially installed at v4.9.31 but was updated afterward to v4.9.43 (the latest z-stream) to see if the updates fixed anything but it didn't make any difference. The cluster is currently running v4.9.43 and shows the $search domain missing in the /etc/resolv.conf file on all nodes.

      Attachments

        Issue Links

          Activity

            People

              bnemec@redhat.com Benjamin Nemec
              rhn-support-mmarkand Mridul Markandey
              Zhanqi Zhao Zhanqi Zhao
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: