OpenShift installation on Azure fails with proxy not able populate private internal DNS
When installing OCP the machine-config cluster operator is not successful and complains with following message:
Error:
~~~ 
Message: Node master-0 is reporting: "unexpected on-disk state validating against rendered-master-xx: content mismatch for file \"/etc/mco/proxy.env\"", Node master-1 is reporting: "unexpected on-disk state validating against rendered-master-xx: content mismatch for file \"/etc/mco/proxy.env\"", Node master-2 is reporting: "unexpected on-disk state validating against rendered-master-xx: content mismatch for file \"/etc/mco/proxy.env\"" ~~~

Environment:- 
OCP version 4.12.46- UPI install with a proxy, 
- existing private DNS, 
- existing network.

It is observed that the rendered-master-xxx mc wanted to configure the following

function urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; } ;  urldecode "$(oc get mc rendered-master-xxx -o yaml | grep -B5 "path: /etc/mco/proxy.env" | grep source | tail -n 1 | cut -d, -f2)"

# Proxy environment variables will be populated in this file. Properly
# url encoded passwords with special characters will use '%<HEX><HEX>'.
# Systemd requires that any % used in a password be represented as
# %% in a unit file since % is a prefix for macros; this restriction does not
# apply for environment files. Templates that need the proxy set should use
# 'EnvironmentFile=/etc/mco/proxy.env'.
HTTP_PROXY=http://10.0.0.4:3128
HTTPS_PROXY=http://10.0.0.4:3128
NO_PROXY=.cluster.local,.svc,10.0.0.0/24,10.128.0.0/14,127.0.0.1,168.63.xx.xx,169.254.xx.xx,172.30.0.0/16,api-int.test.example.com,localhost

value observed on the machine was :
But the value in the master machine was:
sh-4.4# cat /etc/mco/proxy.env
# Proxy environment variables will be populated in this file. Properly
# url encoded passwords with special characters will use '%<HEX><HEX>'.
# Systemd requires that any % used in a password be represented as
# %% in a unit file since % is a prefix for macros; this restriction does not
# apply for environment files. Templates that need the proxy set should use
#'EnvironmentFile=/etc/mco/proxy.env'.
HTTP_PROXY=http://10.0.0.4:3128
HTTPS_PROXY=http://10.0.0.4:3128
NO_PROXY=.cluster.local,.svc,10.0.0.0/24,10.128.0.0/14,127.0.0.1,169.254.xx.xx,172.30.0.0/16,api-int.test.example.com,localhost

After manually vi into each master node and add the IP 168.63.xx.xx to each proxy.env resolves the issue. That IP is needed when installing with a proxy and a private azure DNS.    

    100%

    1. Install UPI cluster with proxy and existing private DNS + network
    2. Observe the installation to fail
    3. check proxy.env on the node
    

 Installation fails   

Installation should be successful    

- This was working fine 4.10 
- Also, The master mcp needs to be created because the first one is somehow deleted or was never created.