Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-16923

Agent-based installation fails when using custom proxy certificates

XMLWordPrintable

    • No
    • False
    • Hide

      None

      Show
      None

      We tried to install OCP 4.12.18 on a lab cluster using the agent-based installer. Our install-config.yaml file included a private CA certificate that we wanted to add to the certificate trust bundle. This certificate had to be added to the trusted certificates so that a TLS connection can be opened from the OpenShift node to quay.io through the corporate proxy.

      The install-config.yaml file:

      apiVersion: v1
baseDomain: example.com 
compute: 
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane: 
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata: 
  name: mycluster5
networking: 
  clusterNetwork: 
  - cidr: 10.128.0.0/14 
    hostPrefix: 23 
  machineNetwork: 
  - cidr: 192.168.50.0/25
  networkType: OVNKubernetes
  serviceNetwork: 
  - 172.30.0.0/16
platform: 
#  none: {} 
  baremetal: 
    apiVIPs: 
      - 192.168.50.10
    ingressVIPs: 
      - 192.168.50.11
bootstrapInPlace: 
  installationDisk: /dev/vda
proxy: 
  httpsProxy: http://proxy.lab.example.com:3128
  httpProxy: http://proxy.lab.example.com:3128
  noProxy: .example.com,192.168.0.0/16

additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIF3DCCA8SgAwIBAgIUazypQ1bCJkMdAXyqgW6g6khhjgkwDQYJKoZIhvcNAQEL
  BQAwdzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYDVQQHDAlTYW4gRGll
  Z28xEzARBgNVBAoMCkFsZXMgTm9zZWsxDTALBgNVBAMMBHJvb3QxIzAhBgkqhkiG
  9w0BCQEWFGFsZXMubm9zZWtAZ21haWwuY29tMB4XDTIxMTEwNjE5Mzg1MFoXDTMx
  MTEwNDE5Mzg1MFowdzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYDVQQH
  DAlTYW4gRGllZ28xEzARBgNVBAoMCkFsZXMgTm9zZWsxDTALBgNVBAMMBHJvb3Qx
  IzAhBgkqhkiG9w0BCQEWFGFsZXMubm9zZWtAZ21haWwuY29tMIICIjANBgkqhkiG
  9w0BAQEFAAOCAg8AMIICCgKCAgEAuaFXD6sn1BxeCnAJjbzu3RtYK5tm8anP9tbA
  oeu59PnmsEvMKpz8livryIumIyA1RZUkoIfeq2Bt16hCxFfq+E4ocY+znmyQ+e0B
  Iiem8KTEQ0whk93Elf27C0t+CyMdw0VEHN64TicLwmTksPWDVKj/9Z8+mHdBjZSC
  C8/SVmnRkSR5D8wOPPoyyPKmQuaEdgM8TZVCnfgnsQqydd38X6ThTA52jmHKpLBY
  SzzhUf+h3dcykaOBWhSSa2KIbLT//bus/hVyLw75ADSecUV22cXfaQK3f4nAA/Y+
  8ff4dvUwFEQ9TCOQCmwsOuvt/HEQZsCv+iHXljZjmdNWJvqO+uZ/Q3rOAmEZUD4k
  P8kOTxkbvJEdsGitwWqpIoEQlhzkuVUqBQIA6jhQK+vi9gAdiFmJkyBVEc5h9DRZ
  yt2jGDOARYePXfUNj203otbJPpbzqUsIUNABrevlUZiYS+IS2BLbZMFen88i3GGD
  3sohtJ9hevuiAq8izO6S6wro/Prq1AYeNVMNH2f0VKQL9GCrA4+jNOOdIwnllmAy
  z9aMAJA7NjeFB+WvCTPIn7rlDJcrUouO4DgtADWgICmswBQShR0o0MZvcFygOu9n
  UjW6+T30SVcWC097zclT5VMYuPaTJgGfJAEZYvUyGMwEzYdr2ccQfFySg7IT7iFb
  ZT5l9+8CAwEAAaNgMF4wHQYDVR0OBBYEFJo1I76Tto7GTVsT9NZn5Xu+WxedMB8G
  A1UdIwQYMBaAFJo1I76Tto7GTVsT9NZn5Xu+WxedMA8GA1UdEwEB/wQFMAMBAf8w
  CwYDVR0PBAQDAgGmMA0GCSqGSIb3DQEBCwUAA4ICAQBT7bGU65Jrmj6cE5xfz/Xq
  uQfXw/PgAV24P+RZUwQcVXaZU+M5lxZGT98V+ft0spxjRIl0bfkA5QTvbUjLkzN9
  ipJA78PnTBfUHHkcQGADtBlwazh2ZO8YjbpV22ki0qbEKtekNPZ0w0jpdj/RRYml
  vOkIfn3YN0dbcF1MB+Lqq2O8eZZLwc2ROWGL2u+/nuqTxS2agVU5S9O97ddZSPUo
  8LcU5eFYRLCZaSVPdHraR9yPj/lJgAwF9+s4mUPmA42PjBUMuFVtd+E378OIOIzY
  K9GExMavV4vF/nAAAiY9Hn2TjaWtDDcOq0fMmRipLktc9hMC+byzODpecVbZtyNV
  5WObi0WiII89SkXjdhq3X7r/kQJHH3JdCe/lhBNP5Tmq1EyCZVCs6LnF3WqYUZEn
  7Q3t2ISaOvY7RfWDDI4+DZAYvE64Us5sOXeAvVMXMDdfDsB4RlnJDrpEpTQGGqYs
  nmG0DT16RWysDY81+PTkdUeOpm4JjMIKc/x1zkkitHPLJ5N3eTudtxrGBh4Zu/4Q
  bvJYzkDdGPOrKDZCwEGJBjqzuMSV/N/V6v8TQ/AE2cx+1TiXjkLFbV8zJFCtoZ61
  zzEWFnYD/tuy49aXpRsGiX61YWldwxDsr8wbfoI4IdAB0Q110dTrYJKYUzgEPbv6
  UG3U5wKk91vppQCtg9AV4Q==
  -----END CERTIFICATE-----

pullSecret: REDACTED

sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzLiL31YTM+haec6P9UuqW8Z2CV/EE9QBX2kD7elMKpRIb4jEOx6MCnKu+unIofvwWJ3/doiYh/mqTbr7yU6LTQ0nkL+2QTKD3y2St1OjJcHO9oC2SjXxVZFMkCf4c59SC+zqqRI/pY+QUkiR8XvEFZevNajTY1KML5sd8lCs58Yz94rdBfkCvuD9Y+XHhcxJJ9i6WW3Joz/TlLQUcQfEuZ6V0Hd+fhT51hX5tmJJPBcjkF9hmXE/n/lLHpOO9MjuVAu25kyyiwU/jzFncLjksN984KABYqrk4vgpJ8lDAr06CODfnQG6FE26QmJ019kprwN80lwG+mo6HVyY/34R/ anosek@zihadlo'

      The agent-config.yaml file:

      apiVersion: v1alpha1
kind: AgentConfig
metadata: 
  name: sno-cluster
rendezvousIP: 192.168.50.21
hosts: 
  - hostname: master1
    role: master
    interfaces: 
      - name: ens3
        macAddress: 52:54:00:2c:0e:01
  - hostname: master2
    role: master
    interfaces: 
      - name: ens3
        macAddress: 52:54:00:2c:0e:02
  - hostname: master3
    role: master
    interfaces: 
      - name: ens3
        macAddress: 52:54:00:2c:0e:03
  - hostname: worker1
    role: worker
    interfaces: 
      - name: ens3
        macAddress: 52:54:00:2c:0e:04
  - hostname: worker2
    role: worker
    interfaces: 
      - name: ens3
        macAddress: 52:54:00:2c:0e:05

      We created a CD-ROM image

      $ ./openshift-install --dir clusterconfigs agent create image

      and booted the OpenShift nodes using this image. The nodes were not able to install due to:

      $ journalctl -f
Jul 19 02:58:40 master2 resolv-prepender.sh[14786]: Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:891d162a94044031a
5c30d5986e16311340e89fc0653ad3cac25bdcdf3c2fc26: pinging container registry quay.io: Get "https://quay.io/v2/": x509: certificate signed by unknown authority
Jul 19 02:58:41 master2 bash[14799]: Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:891d162a94044031a5c30d5986e16311340e89fc0653ad3cac25b
dcdf3c2fc26...
Jul 19 02:58:41 master2 bash[14799]: Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:891d162a94044031a5c30d5986e16311
340e89fc0653ad3cac25bdcdf3c2fc26: pinging container registry quay.io: Get "https://quay.io/v2/": x509: certificate signed by unknown authority
Jul 19 02:58:41 master2 resolv-prepender.sh[14812]: Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:891d162a94044031a5c30d5986e16311340e89
fc0653ad3cac25bdcdf3c2fc26...
Jul 19 02:58:41 master2 resolv-prepender.sh[14812]: Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:891d162a94044031a
5c30d5986e16311340e89fc0653ad3cac25bdcdf3c2fc26: pinging container registry quay.io: Get "https://quay.io/v2/": x509: certificate signed by unknown authority
Jul 19 02:58:42 master2 resolv-prepender.sh[14827]: Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:891d162a94044031a5c30d5986e16311340e89
fc0653ad3cac25bdcdf3c2fc26...

      As a workaround, we SSHed into the nodes during the installation and added our private CA certificate to the trust store manually:

      $ vi /etc/pki/ca-trust/source/anchors/root.crt.pem
$ update-ca-trust

      After that, the installation proceeded immediately and was eventually completed successfully.

      We noticed that when the node was stuck trying to connect to quay.io, the file /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt was empty. After the installation was completed, this file was filled with our private CA certificate automatically.

      The OpenShift documentation doesn't tell whether an agent-based installer supports using custom CA certificates or not. Is this a supported feature?

              bfournie@redhat.com Robert Fournier
              anosek@redhat.com Ales Nosek
              Manoj Hans Manoj Hans
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: