Description of problem:

Internal cluster service lookups (e.g., <service>.<namespace>.svc.cluster.local) are failing cluster-wide for all pods. The failure is due to the pod's DNS resolver incorrectly appending an external search domain having a wild card record, which is then resolved to an external incorrect IP. This results in connection refused errors for internal service traffic. This behavior represents a regression from prior OCP version v4.17.24.

Version-Release number of selected component (if applicable):

OpenShift v4.18.27

How reproducible:

• One of the search domain in the "/etc/resolv.conf" file within the pod has a wildcard DNS record for it in the external DNS.
   
      

Actual results:

• While performing an nslookup to one of the failing internal cluster service url, it was found that the CoreDNS appended one of the search domain "ikredit360.com" to it and thereby causing it to resolve to the external LB's IP.

Expected results:

• The nslookup of the internal cluster services urls should NOT be appended by any of the external search domains.

    
Additional info:

• The reason why the internal domain (svc.cluster.local) got mapped with the external domain's (ikredit360.com) wild card record is the default "options ndots:5" configuration in the "/etc/resolv.conf" file of the pod.

• When we manually tried to run the query by appending one more dot to one of the affected internal cluster services url  and thereby making it 5 dot query, the resolve considered it as a qualified FQDN and resulted in correct IP of the service.