-
Bug
-
Resolution: Done
-
Critical
-
None
-
4.12
-
Critical
-
None
-
Approved
-
False
-
-
NA
-
Rejected
Description of problem:
Cluster installation with proxy enabled in disconnected network failed at bootstrap stage, checking on master machine, machine-config-daemon-firstboot.service failed as below: [core@gpei-test108-wj88c-master-0 ~]$ systemctl status machine-config-daemon-firstboot.service ● machine-config-daemon-firstboot.service - Machine Config Daemon Firstboot Loaded: loaded (/etc/systemd/system/machine-config-daemon-firstboot.service; enabled; vendor preset: enabled) Active: activating (start) since Sat 2022-10-08 03:50:29 UTC; 2h 8min ago Main PID: 2652 (machine-config-) Tasks: 11 (limit: 204167) Memory: 36.7M CPU: 16.984s CGroup: /system.slice/machine-config-daemon-firstboot.service └─2652 /run/bin/machine-config-daemon firstboot-complete-machineconfig Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:57:41.263061 2652 rpm-ostree.go:447] Running captured: rpm-ostree --version Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:57:41.289277 2652 rpm-ostree.go:407] Executing rebase to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661 Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:57:41.289310 2652 update.go:2053] Running: rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661 Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[6336]: Pulling manifest: ostree-unverified-image:docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661 Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495095 2652 update.go:1243] Updating files Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495290 2652 update.go:1308] Deleting stale data Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495304 2652 update.go:2098] Removing SIGTERM protection Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: W1008 05:58:41.495313 2652 firstboot_complete_machineconfig.go:46] error: failed to update OS to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661 : error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661: error: remote error: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 75.101.245.134:443: i/o timeout Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: : exit status 1 Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495319 2652 firstboot_complete_machineconfig.go:47] Sleeping 1 minute for retry Proxy environment vars were configured correctly in /etc/mco/proxy.env. [core@gpei-test108-wj88c-master-0 ~]$ grep -r "etc/mco/proxy.env" /etc/systemd/ /etc/systemd/system/kubelet.service.d/10-mco-default-env.conf:EnvironmentFile=/etc/mco/proxy.env /etc/systemd/system/crio.service.d/10-mco-default-env.conf:EnvironmentFile=/etc/mco/proxy.env /etc/systemd/system/machine-config-daemon-firstboot.service:EnvironmentFile=/etc/mco/proxy.env /etc/systemd/system/machine-config-daemon-pull.service:EnvironmentFile=/etc/mco/proxy.env /etc/systemd/system/nodeip-configuration.service:EnvironmentFile=/etc/mco/proxy.env /etc/systemd/system/pivot.service.d/10-mco-default-env.conf:EnvironmentFile=/etc/mco/proxy.env [core@gpei-test108-wj88c-master-0 ~]$ cat /etc/mco/proxy.env # Proxy environment variables will be populated in this file. Properly # url encoded passwords with special characters will use '%<HEX><HEX>'. # Systemd requires that any % used in a password be represented as # %% in a unit file since % is a prefix for macros; this restriction does not # apply for environment files. Templates that need the proxy set should use # 'EnvironmentFile=/etc/mco/proxy.env'. HTTP_PROXY=http://xxx:xxx@10.0.99.4:3128 HTTPS_PROXY=http://xxx:xxx@10.0.99.4:3128 NO_PROXY=.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.gpei-test108.qe.azure.devcluster.openshift.com,localhost,test.no-proxy.com And quay only could be available through the proxy [core@gpei-test108-wj88c-master-0 ~]$ curl -I --proxy-basic --proxy-user 'xxx:xxx' --proxy "http://10.0.99.4:3128" https://quay.io/ HTTP/1.1 200 Connection established HTTP/2 200
Version-Release number of selected component (if applicable):
4.12.0-0.nightly-2022-10-05-053337
How reproducible:
Always
Steps to Reproduce:
1. Prepare a disconnected(no Internet accessibility) VPC, set global proxy in the install-config.yaml to enable the cluster accessing necessary URLs via the proxy server, and trigger the installation. 2. 3.
Actual results:
Expected results:
Additional info:
Seen from QE CI results, the proxy installation is start failing since nightly 4.12.0-0.nightly-2022-10-05-053337, and works well with 4.12.0-0.nightly-2022-09-28-204419
- duplicates
-
OCPBUGS-2245 rpm-ostree rebase doesn't get proxy settings from MCD
- Closed
- relates to
-
OCPBUGS-2197 [upgrade 4.11.z to 4.12 nightly] rpm-ostree update via container failed
- Closed
-
TRT-608 aws-ovn-proxy appears to be broken
- Closed
- links to