Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-2122

machine-config-daemon failed to update the OS for cluster running behind proxy

XMLWordPrintable

    • Critical
    • Approved
    • False
    • Hide

      None

      Show
      None
    • NA
    • Rejected

      Description of problem:

      Cluster installation with proxy enabled in disconnected network failed at bootstrap stage, checking on master machine, machine-config-daemon-firstboot.service failed as below: 
      
      [core@gpei-test108-wj88c-master-0 ~]$ systemctl status machine-config-daemon-firstboot.service 
      ● machine-config-daemon-firstboot.service - Machine Config Daemon Firstboot
         Loaded: loaded (/etc/systemd/system/machine-config-daemon-firstboot.service; enabled; vendor preset: enabled)
         Active: activating (start) since Sat 2022-10-08 03:50:29 UTC; 2h 8min ago
       Main PID: 2652 (machine-config-)
          Tasks: 11 (limit: 204167)
         Memory: 36.7M
            CPU: 16.984s
         CGroup: /system.slice/machine-config-daemon-firstboot.service
                 └─2652 /run/bin/machine-config-daemon firstboot-complete-machineconfig
      
      Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:57:41.263061    2652 rpm-ostree.go:447] Running captured: rpm-ostree --version
      Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:57:41.289277    2652 rpm-ostree.go:407] Executing rebase to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661
      Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:57:41.289310    2652 update.go:2053] Running: rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661
      Oct 08 05:57:41 gpei-test108-wj88c-master-0 machine-config-daemon[6336]: Pulling manifest: ostree-unverified-image:docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661
      Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495095    2652 update.go:1243] Updating files
      Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495290    2652 update.go:1308] Deleting stale data
      Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495304    2652 update.go:2098] Removing SIGTERM protection
      Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: W1008 05:58:41.495313    2652 firstboot_complete_machineconfig.go:46] error: failed to update OS to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661 : error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0daf5c4a35424410e88dde102022fc3581302bc8a98e09e2e4748502c59b3661: error: remote error: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 75.101.245.134:443: i/o timeout
      Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: : exit status 1
      Oct 08 05:58:41 gpei-test108-wj88c-master-0 machine-config-daemon[2652]: I1008 05:58:41.495319    2652 firstboot_complete_machineconfig.go:47] Sleeping 1 minute for retry
      
      
      Proxy environment vars were configured correctly in /etc/mco/proxy.env.
      [core@gpei-test108-wj88c-master-0 ~]$ grep -r "etc/mco/proxy.env" /etc/systemd/
      /etc/systemd/system/kubelet.service.d/10-mco-default-env.conf:EnvironmentFile=/etc/mco/proxy.env
      /etc/systemd/system/crio.service.d/10-mco-default-env.conf:EnvironmentFile=/etc/mco/proxy.env
      /etc/systemd/system/machine-config-daemon-firstboot.service:EnvironmentFile=/etc/mco/proxy.env
      /etc/systemd/system/machine-config-daemon-pull.service:EnvironmentFile=/etc/mco/proxy.env
      /etc/systemd/system/nodeip-configuration.service:EnvironmentFile=/etc/mco/proxy.env
      /etc/systemd/system/pivot.service.d/10-mco-default-env.conf:EnvironmentFile=/etc/mco/proxy.env
      [core@gpei-test108-wj88c-master-0 ~]$ cat /etc/mco/proxy.env
      # Proxy environment variables will be populated in this file. Properly
      # url encoded passwords with special characters will use '%<HEX><HEX>'.
      # Systemd requires that any % used in a password be represented as
      # %% in a unit file since % is a prefix for macros; this restriction does not
      # apply for environment files. Templates that need the proxy set should use
      # 'EnvironmentFile=/etc/mco/proxy.env'.
      HTTP_PROXY=http://xxx:xxx@10.0.99.4:3128
      HTTPS_PROXY=http://xxx:xxx@10.0.99.4:3128
      NO_PROXY=.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.gpei-test108.qe.azure.devcluster.openshift.com,localhost,test.no-proxy.com
      
      And quay only could be available through the proxy
      [core@gpei-test108-wj88c-master-0 ~]$ curl -I --proxy-basic --proxy-user 'xxx:xxx' --proxy "http://10.0.99.4:3128" https://quay.io/
      HTTP/1.1 200 Connection established
      
      HTTP/2 200 
      
      

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-10-05-053337
      
      

      How reproducible:

      Always
      

      Steps to Reproduce:

      1. Prepare a disconnected(no Internet accessibility) VPC, set global proxy in the install-config.yaml to enable the cluster accessing necessary URLs via the proxy server, and trigger the installation.
      2.
      3.
      

      Actual results:

      
      

      Expected results:

      
      

      Additional info:

      Seen from QE CI results, the proxy installation is start failing since nightly 4.12.0-0.nightly-2022-10-05-053337, and works well with 4.12.0-0.nightly-2022-09-28-204419
      
      

            team-mco Team MCO
            gpei@redhat.com Gaoyun Pei
            Rio Liu Rio Liu
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: