machine-config-daemon failed to update the OS for cluster running behind proxy

Cluster installation with proxy enabled in disconnected network failed at bootstrap stage, checking on master machine, machine-config-daemon-firstboot.service failed as below: 

[core@gpei-test108-wj88c-master-0 ~]$ systemctl status machine-config-daemon-firstboot.service 
● machine-config-daemon-firstboot.service - Machine Config Daemon Firstboot
   Loaded: loaded (/etc/systemd/system/machine-config-daemon-firstboot.service; enabled; vendor preset: enabled)
   Active: activating (start) since Sat 2022-10-08 03:50:29 UTC; 2h 8min ago
 Main PID: 2652 (machine-config-)
    Tasks: 11 (limit: 204167)
   Memory: 36.7M
      CPU: 16.984s
   CGroup: /system.slice/machine-config-daemon-firstboot.service
           └─2652 /run/bin/machine-config-daemon firstboot-complete-machineconfig

Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:57:41.263061    2652 rpm-ostree.go:447] Running captured: rpm-ostree --version
Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:57:41.289277    2652 rpm-ostree.go:407] Executing rebase to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661
Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:57:41.289310    2652 update.go:2053] Running: rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661
Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[6336]: Pulling manifest: ostree-unverified-image:docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661
Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495095    2652 update.go:1243] Updating files
Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495290    2652 update.go:1308] Deleting stale data
Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495304    2652 update.go:2098] Removing SIGTERM protection
Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: W1008 05:58:41.495313    2652 firstboot_complete_machineconfig.go:46] error: failed to update OS to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661 : error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661: error: remote error: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 75.101.245.134:443: i/o timeout
Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: : exit status 1
Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495319    2652 firstboot_complete_machineconfig.go:47] Sleeping 1 minute for retry


Proxy environment vars were configured correctly in /etc/mco/proxy.env.
[core@gpei-test108-wj88c-master-0 ~]$ grep -r "etc/mco/proxy.env" /etc/systemd/
/etc/systemd/system/kubelet.service.d/10-mco-default-env.conf:EnvironmentFile=/etc/mco/proxy.env
/etc/systemd/system/crio.service.d/10-mco-default-env.conf:EnvironmentFile=/etc/mco/proxy.env
/etc/systemd/system/machine-config-daemon-firstboot.service:EnvironmentFile=/etc/mco/proxy.env
/etc/systemd/system/machine-config-daemon-pull.service:EnvironmentFile=/etc/mco/proxy.env
/etc/systemd/system/nodeip-configuration.service:EnvironmentFile=/etc/mco/proxy.env
/etc/systemd/system/pivot.service.d/10-mco-default-env.conf:EnvironmentFile=/etc/mco/proxy.env
[core@gpei-test108-wj88c-master-0 ~]$ cat /etc/mco/proxy.env
# Proxy environment variables will be populated in this file. Properly
# url encoded passwords with special characters will use '%<HEX><HEX>'.
# Systemd requires that any % used in a password be represented as
# %% in a unit file since % is a prefix for macros; this restriction does not
# apply for environment files. Templates that need the proxy set should use
# 'EnvironmentFile=/etc/mco/proxy.env'.
HTTP_PROXY=http://xxx:xxx@10.0.99.4:3128
HTTPS_PROXY=http://xxx:xxx@10.0.99.4:3128
NO_PROXY=.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.gpei-test108.qe.azure.devcluster.openshift.com,localhost,test.no-proxy.com

And quay only could be available through the proxy
[core@gpei-test108-wj88c-master-0 ~]$ curl -I --proxy-basic --proxy-user 'xxx:xxx' --proxy "http://10.0.99.4:3128" https://quay.io/
HTTP/1.1 200 Connection established

HTTP/2 200 

4.12.0-0.nightly-2022-10-05-053337

Always

1. Prepare a disconnected(no Internet accessibility) VPC, set global proxy in the install-config.yaml to enable the cluster accessing necessary URLs via the proxy server, and trigger the installation.
2.
3.

Seen from QE CI results, the proxy installation is start failing since nightly 4.12.0-0.nightly-2022-10-05-053337, and works well with 4.12.0-0.nightly-2022-09-28-204419