Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: omr-v1.0.0
Affects Version/s: omr-v1.0.0
Component/s: OMR
Labels:
- triaged

Blocked:
False
Ready:
False
Git Pull Request:
https://github.com/quay/openshift-mirror-registry/pull/36
Market:

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description:

This is an issue found when using OMR 1.0 to install OCP Cluster, when deploy OMR with default installation, that means OMR installer will generate the TLS Cert, but with the CARoot Cert provided by OMR installer, the OCP installation was failed, the reason is that OMR Quay endpoint didn't return the chain cert. Pls review this issue and make OMR to return the chain cert.

Note: this issue will block OCP Customers to deploy OCP Cluster when they use default OMR installation.

OMR: openshift-openshift-mirror-registry-rhel8:v1.0-4

The following is the OCP installation logs, see OCP Installer bootkube.log

Jan 07 06:36:27 ip-10-0-21-47 bootkube.sh[7241]: Error: unable to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b: unable to pull image: Error initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b: (Mirrors also failed: [quayomr10.qe.devcluster.openshift.com:8443/redhat/ocp46@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b: error pinging docker registry quayomr10.qe.devcluster.openshift.com:8443: Get "https://quayomr10.qe.devcluster.openshift.com:8443/v2/": x509: certificate signed by unknown authority]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b: Error reading manifest sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b in quay.io/openshift-release-dev/ocp-v4.0-art-dev: unauthorized: access to the requested resource is not authorized

OMR Server Cert:

OMR CARoot Cert:

OMR Chain Cert:

Use Openssl to get OMR Cert:

openssl s_client -connect quayomr10.qe.devcluster.openshift.com:8443 -showcerts
CONNECTED(00000005)
depth=0 CN = quay-enterprise
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = quay-enterprise
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=quay-enterprise
   i:/C=US/ST=VA/L=New York/O=Quay/OU=Division/CN=quayomr10.qe.devcluster.openshift.com
-----BEGIN CERTIFICATE-----
MIIDfTCCAmWgAwIBAgIUPn5aGshHQ3HwdQhuV4C1QoAxlOowDQYJKoZIhvcNAQEL
BQAwfzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlZBMREwDwYDVQQHDAhOZXcgWW9y
azENMAsGA1UECgwEUXVheTERMA8GA1UECwwIRGl2aXNpb24xLjAsBgNVBAMMJXF1
YXlvbXIxMC5xZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20wHhcNMjIwMTA3MDYw
NTQ2WhcNMjIxMjI5MDYwNTQ2WjAaMRgwFgYDVQQDDA9xdWF5LWVudGVycHJpc2Uw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDlXucXmumdStR5qi7OjLni
6gRJhhoUiUD72GUpW2BTJx9c7bs+qklrVH4bmBs/g/Tmp6Aqi/Lbnop2R3VOEd8+
eqWNcfNEuJicQGWQyPV6rRe8oIoikQ47kXMhZxPrG5jQqdkKGyk9+zF1jb/qVjRQ
hYYKd5bXWPMSSMPAy7YRBWKEpzMgW3sJYSsaNuelQKK4N4MO7FS3nX9x/IPWn4zo
HSegMn/2VnaNsSKkZYc3ciYbhyk+xr6YfhtyXVkF86O0v2QZ9eGlcQFO2zt0s74v
EvmvsuXLDgwySc5V6MKIpVAZ3P1oZoD9qqCMptQxcJ+5BidUrtmtY6M4tdQrrwjt
AgMBAAGjVjBUMAsGA1UdDwQEAwIC5DATBgNVHSUEDDAKBggrBgEFBQcDATAwBgNV
HREEKTAngiVxdWF5b21yMTAucWUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tMA0G
CSqGSIb3DQEBCwUAA4IBAQBVdUbQpoJBFQlpsZ5UkmoFwBcQdMYK27985ELdoeNj
RwmB4KmsQoTyFNHy7nPFH0WRcKepMRzIm7lIhjNdOgT3UdMos4z24YShapzwkwA0
fZQ4Px19rgUHIJNetJlwcReaVafMr1QEw6umfHy5IkLbLQngwPVF6f/upUMVYtUR
hbIb7wmH/ljUx7a+tkQIyJpH/WbSOl8adQ1fXENS0mWtBGm3dMs1FrWgTqjm9HOH
UGjw/YsQljOTtG2giBUu++zCzcLE/HU+xO9EYl+z29clzkrDNOC/EO4isV5TnPb8
cO3HWbn+o9nRaNfdToweIqciJczJKw9P7ox7vgEwJgl8
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=quay-enterprise
issuer=/C=US/ST=VA/L=New York/O=Quay/OU=Division/CN=quayomr10.qe.devcluster.openshift.com
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 1534 bytes and written 289 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: FCC6BB6ABC136806939927B031DAC3AB36914BE43D5ACA2E1B437AA590868C30
    Session-ID-ctx:
    Master-Key: 54CA4037A7E2864C55DF7C5C9806E6623643803FC659C5D1E0D092DC46DAD5ECCC29419BE1377AA6B527805423C8082D
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 83 15 13 fa 24 12 dd d5-82 66 a8 2c 97 a3 16 9d   ....$....f.,....
    0010 - 5b 4b cf 39 4e ca ab 98-92 22 5a bd 31 67 63 3d   [K.9N...."Z.1gc=
    0020 - 33 db 18 0b 8a 0c 74 4d-3a 17 2d 73 b8 69 32 6d   3.....tM:.-s.i2m
    0030 - 52 6e d9 59 6d e0 af 79-52 60 27 08 b4 7b 67 31   Rn.Ym..yR`'..{g1
    0040 - 04 c4 78 21 cf eb 66 6a-07 9c 46 bc 5a b4 11 38   ..x!..fj..F.Z..8
    0050 - 20 72 a5 bf a3 71 6f 32-19 f9 e4 c9 a5 b9 14 b5    r...qo2........
    0060 - e3 08 3d 84 96 41 b3 18-ac 09 5c b2 dc 7e 37 f2   ..=..A....\..~7.
    0070 - f5 82 2f 27 08 7c b3 10-aa 1b 47 2b c4 74 93 62   ../'.|....G+.t.b
    0080 - 75 52 ba a0 74 db 85 b4-cf 7c 3d 71 18 34 c4 99   uR..t....|=q.4..
    0090 - b4 c2 68 b1 e0 d7 78 e3-7d ce a7 97 98 ba c6 ad   ..h...x.}.......
    00a0 - f6 9c 50 33 52 5a 1f 89-aa e8 02 22 34 e5 5b e3   ..P3RZ....."4.[.    Start Time: 1641535944
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
closed

OCP installer config.yaml:

apiVersion: v1
baseDomain: qe.devcluster.openshift.com 
credentialsMode: Mint 
controlPlane:   
  hyperthreading: Enabled 
  name: master
  platform:
    aws:
      zones:
      - us-west-2a
      - us-west-2b
      rootVolume:
        iops: 4000
        size: 500
        type: io1 
      type: m4.2xlarge
  replicas: 
compute: 
- hyperthreading: Enabled 
  name: worker
  platform:
    aws:
      rootVolume:
        iops: 2000
        size: 500
        type: io1 
      type: m4.2xlarge
      zones:
      - us-west-2c
  replicas: 3
metadata:
  name: quayomr012
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  aws:
    region: us-west-2 
    userTags:
      adminContact: luffy
      costCenter: 7536
    hostedZone: Z3B3KOVA3TRCWP 
fips: false 
sshKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN8nUzLnPHq9o6Crika8brT4i5CL0a0azoHJoHe02BH8/vgDyhgHin+1qDrHA414t6smDIhYRM/L503J0kD2/jUTPVqeFNmbxbzXnEXWv2RaAyKChMzw2PkrKiLntY4CxcukdSN6lqtJa8TH3/Vmy/YUOMJOKWEsYkg6qojDWPYbFHMubm6JWPydiEJJYPYCH7tHPaq4Y3CWNw+jx2sL69Sltnsdc/oj5Icl+u/ClF7lm0LPXkrkUF745ktCg6r06dLju3Ap+A0HJ/doTpCymZrt88eEy0RqW9koDYPJsRm380caT0J4wux3HlZiHP0b1mhx9pp7DB0FuhZHxeQawGs4V3aYDisBE27YMoMBqoCmBOqkVqC7uY47HOYiS15YHpriCXSnflE628e6a7zfFVV+CcrcqtcqPltZlXmbm2PeQY547VphB1nivinALOVM+CcSgOchX1Phmj63nXKt/IbsUJhUnZQicFhh2bJzXWKBtCQkodwTnu90RaKJN2pn8= lizhang@lzha-mac 
pullSecret: '{"auths":{"quayomr10.qe.devcluster.openshift.com:8443": {"auth": "aW5pdDpwYXNzd29yZA==","email": "lzha@redhat.com"}, "registry.ci.openshift.org": {"auth": "******","email": "lzha@redhat.com"}}}'
additionalTrustBundle: |
    -----BEGIN CERTIFICATE-----
    MIID4jCCAsqgAwIBAgIULKwFYj9Ta9pepXT1GKTwxPcryuQwDQYJKoZIhvcNAQEL
    BQAwfzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlZBMREwDwYDVQQHDAhOZXcgWW9y
    azENMAsGA1UECgwEUXVheTERMA8GA1UECwwIRGl2aXNpb24xLjAsBgNVBAMMJXF1
    YXlvbXIxMC5xZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20wHhcNMjIwMTA3MDYw
    NTQ0WhcNMjQxMDI3MDYwNTQ0WjB/MQswCQYDVQQGEwJVUzELMAkGA1UECAwCVkEx
    ETAPBgNVBAcMCE5ldyBZb3JrMQ0wCwYDVQQKDARRdWF5MREwDwYDVQQLDAhEaXZp
    c2lvbjEuMCwGA1UEAwwlcXVheW9tcjEwLnFlLmRldmNsdXN0ZXIub3BlbnNoaWZ0
    LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMeGvtCldAtFsJ0B
    MScfPGOTmzIiwJGL+5A/WKDKUiyfTWXLgWghD347+ot6/qu1b6Bw2ulgIB+s/z8v
    rJz9km47tRhsXfzDX7adzek/mqV5CbT/U2+MePEXX/HLVKOj6/2E9BQwUF70oZ9V
    IhpeJJ7texbTX6jkekpwEp+UxF7U04lXasdBHeYP04ts8Zio/wn52ZpM8lCMscB5
    1AtvGNXCqL9Ey5fhfI3ns7GmEGqfZc9C2Xb+rjcwlv/GdVDRo18XnWJhAIFMMX3Z
    Rdqz3cxJ/3a1mq4vrPykIGY8nM4Vpe+azf7tzeYBaNXE5j0J+6lJOlNFyTEXgY8d
    UHWaiOUCAwEAAaNWMFQwCwYDVR0PBAQDAgLkMBMGA1UdJQQMMAoGCCsGAQUFBwMB
    MDAGA1UdEQQpMCeCJXF1YXlvbXIxMC5xZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5j
    b20wDQYJKoZIhvcNAQELBQADggEBAHIxgPX3Zp+7JsWFio3Y3iVKv27TAFZztHUQ
    qA4npsmJRWnUIT8SJkGXbomGtbg97xEuAozviCXA1Xx4kd2AqycpRV9SUJqC5tU7
    wYPyyeWd6OJWQkCVup5/E5QZS8S8VyfDz3y/6PrNgEL7YJYmN3Gt6OGAs65kDdD3
    u7ynM0sTJI0itV6Gi7EZ7HVXAtIDv9aebAQ6/S2+wY+fxCBs6/m6yA/feiV8Gzqj
    4BZYaO0ZWt2OG8EPPuaaX1r7qGDWoytSYUxr0bOP60hQRZvgCmeF3D2niVevxlkW
    xtomiDKisRFjzjbyM/mqQR+3EBaqvFmsP517h8pZykUiBo2GAoI=
    -----END CERTIFICATE-----
imageContentSources:
- mirrors:
  - quayomr10.qe.devcluster.openshift.com:8443/redhat/ocp46
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - quayomr10.qe.devcluster.openshift.com:8443/redhat/ocp46
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

bootkube.log
152 kB
2022/01/07 8:45 AM

Details

Description

Attachments

Attachments

Easy Agile Planning Poker

Activity

People

Dates