Loading...

Type: Bug
Resolution: Not a Bug
Priority: Critical
Fix Version/s: None
Affects Version/s: quay-v3.9.0
Component/s: quay-builder
Labels:
- triaged

Blocked:
False
Blocked Reason:
None
Ready:
False
Intelligence Requested:
Market:

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

When run build function on a OSD (Red Hat Openshift Dedicated) cluster, the build process will be stuck at "Waiting for available build worker" phase.

Got "certificate verify failed" error in quay pod logs. But these OCP API certificates used in quay can pass curl and openssl verification.

Use the same way to obtain OCP API certificates in a common OCP cluster, the whole build function works well. Just hit such weird problem in OSD cluster now.

Haven't test ROSA, ARO cluster, if hit the same problem in ROSA, ARO cluster, will file new issues.

This is the first time to run build function on OSD cluster, tested quay 3.9.0. Suppose quay 3.8.x has the same problem.

Version-Release number of selected component (if applicable):

 quay-operator-bundle-container-v3.9.0-128
 OSD + ocp 4.13

How reproducible:

always

Steps to Reproduce:

1.set up a OSD cluster on AWS

2.set up build environment in OSD cluster

------------------------------
$ oc login https://api.ci-osd-ccs-99vd.ev09.s1.devshift.org:6443 -u osd-admin -p HTPasswd_7W4byl --insecure-skip-tls-verify=true
WARNING: Using insecure TLS client config. Setting this option is not supported!
Login successful.
You have access to 107 projects, the list has been suppressed. You can list all projects with 'oc projects'
Using project "quay-enterprise39".

------------------------------
$ oc extract cm/kube-root-ca.crt -n openshift-apiserver
ca.crt

------------------------------
$ mv ca.crt build_cluster.crt

------------------------------
$ oc create ns virtual-builds
namespace/virtual-builds created

------------------------------
$ oc create sa quay-builder -n virtual-builds
serviceaccount/quay-builder created

------------------------------
$ oc adm policy add-role-to-user edit system:serviceaccount:virtual-builds:quay-builder -n virtual-builds
clusterrole.rbac.authorization.k8s.io/edit added: "system:serviceaccount:virtual-builds:quay-builder"

------------------------------
$ oc adm policy add-scc-to-user anyuid -z quay-builder -n virtual-builds
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "quay-builder"

------------------------------
$ oc create token quay-builder -n virtual-builds --duration 24h |tee ./quay-builder.token
eyJhbGciOiJSUzI1NiIsImtpZCI6IlBTd2ZXT05mOVQzRzBWVmc1Q2paNHlUVWtaZ1o5U1pJczBiNzZTbDlRLXMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY4ODUzODMwMCwiaWF0IjoxNjg4NDUxOTAwLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6IjBmYzI1NDhjLWJjMmUtNDk2Yi05ZDQ2LTM0ZjExNTliZTRkZiJ9fSwibmJmIjoxNjg4NDUxOTAwLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.Zx-Oxnfa6Z0H7Xmn0R4KDQmAVDiP6DVBiRiy0A-OUihId8GHq1CDuizAgMLrvZjmQUKjtlEXoflvBdYNCl2m5nsmsC5Lf3dFt_Gn366oAgB5QUqg3w5TSRlLgG5-9EB8HGhR9LGxImuBZPhTtFuARZWlXvalxlT_rhqty8-KOd2zP4Gid471hmHAY7bQJHId-p97t1jWj0Y8nUdlhgtArxsxzQUY_EkGaqnC_1mEWb2FLc_52W1jOWDqy4EgosBiR2VYo_FNkPVg54O-mnvwjuvIjPXh-90tipnYKULODY_T0nwpKRbYYF7m-ttnKLevnj3ocOHTCVAIdISiqGyaxHLaiAv_nPQPQNCYIcgMoMDevL1qwcDeY1LydVUJnJDprzXK7tG22zsuXGvOWVpUYPVx-IzDsu9yefNvzLhObHgxBk3_vqzK4oXzjsjNT86IF2EuDzb6p0C92iGpON85rGQknf6dGsd3XIKzkz84G7Xn-0CHSjyGLBMks53EHuaMi6U3qLps1Pdw-k-qmB0eGZheltuI159bygjj4uZg0Aml5Ad_n_fZMjcdU30Bjb6ex5jJO_suJ6nqhVBHtDr9mxTjbWraVoQura5tkvGDVOuPhc80Q_wSp0Y8IYBylcSTWvzuY_sUXBWt7T_zh3IAsfPDJXPMphAT33w5y8UNWzg

------------------------------
$ cat config.yaml
FEATURE_BUILD_SUPPORT: true
FEATURE_GITLAB_BUILD: true
GITLAB_TRIGGER_CONFIG:
    CLIENT_ID: e6...8e 
    CLIENT_SECRET: c....69 
GITLAB_TRIGGER_KIND:
    GITLAB_ENDPOINT: https://gitlab.com/
FEATURE_GITHUB_BUILD: true
GITHUB_TRIGGER_CONFIG:
  API_ENDPOINT: https://api.github.com/
  CLIENT_ID: 2d....d
  CLIENT_SECRET:  3....8
  GITHUB_ENDPOINT: https://github.com/
BUILDMAN_HOSTNAME: quayregistry-quay-builder-quay-enterprise39.apps.ci-osd-ccs-99vd.ev09.s1.devshift.org:443 
BUILD_MANAGER:
- ephemeral
- ALLOWED_WORKER_COUNT: 20 
  ORCHESTRATOR_PREFIX: buildman/production/
  ORCHESTRATOR:
    REDIS_HOST: quayregistry-quay-redis
    REDIS_PASSWORD: ""
    REDIS_SSL: false
    REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
  EXECUTORS:
  - EXECUTOR: kubernetesPodman
    DEBUG: true
    NAME: openshift
    BUILDER_NAMESPACE: virtual-builds 
    SETUP_TIME: 180
    QUAY_USERNAME: '...'
    QUAY_PASSWORD: e......c8 
    BUILDER_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.9.0 
    # Kubernetes resource options
    K8S_API_SERVER: api.ci-osd-ccs-99vd.ev09.s1.devshift.org:6443 
    K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt
    VOLUME_SIZE: 8G
    KUBERNETES_DISTRIBUTION: openshift
    CONTAINER_MEMORY_LIMITS: 1G 
    CONTAINER_CPU_LIMITS: 1000m
    CONTAINER_MEMORY_REQUEST: 1G 
    CONTAINER_CPU_REQUEST: 500m
    NODE_SELECTOR_LABEL_KEY: ""
    NODE_SELECTOR_LABEL_VALUE: ""
    SERVICE_ACCOUNT_NAME: quay-builder 
    SERVICE_ACCOUNT_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6IlBTd2ZXT05mOVQzRzBWVmc1Q2paNHlUVWtaZ1o5U1pJczBiNzZTbDlRLXMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY4ODUzODMwMCwiaWF0IjoxNjg4NDUxOTAwLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6IjBmYzI1NDhjLWJjMmUtNDk2Yi05ZDQ2LTM0ZjExNTliZTRkZiJ9fSwibmJmIjoxNjg4NDUxOTAwLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.Zx-Oxnfa6Z0H7Xmn0R4KDQmAVDiP6DVBiRiy0A-OUihId8GHq1CDuizAgMLrvZjmQUKjtlEXoflvBdYNCl2m5nsmsC5Lf3dFt_Gn366oAgB5QUqg3w5TSRlLgG5-9EB8HGhR9LGxImuBZPhTtFuARZWlXvalxlT_rhqty8-KOd2zP4Gid471hmHAY7bQJHId-p97t1jWj0Y8nUdlhgtArxsxzQUY_EkGaqnC_1mEWb2FLc_52W1jOWDqy4EgosBiR2VYo_FNkPVg54O-mnvwjuvIjPXh-90tipnYKULODY_T0nwpKRbYYF7m-ttnKLevnj3ocOHTCVAIdISiqGyaxHLaiAv_nPQPQNCYIcgMoMDevL1qwcDeY1LydVUJnJDprzXK7tG22zsuXGvOWVpUYPVx-IzDsu9yefNvzLhObHgxBk3_vqzK4oXzjsjNT86IF2EuDzb6p0C92iGpON85rGQknf6dGsd3XIKzkz84G7Xn-0CHSjyGLBMks53EHuaMi6U3qLps1Pdw-k-qmB0eGZheltuI159bygjj4uZg0Aml5Ad_n_fZMjcdU30Bjb6ex5jJO_suJ6nqhVBHtDr9mxTjbWraVoQura5tkvGDVOuPhc80Q_wSp0Y8IYBylcSTWvzuY_sUXBWt7T_zh3IAsfPDJXPMphAT33w5y8UNWzg 
USERFILES_LOCATION: default
USERFILES_PATH: userfiles
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: 
  - default
DISTRIBUTED_STORAGE_PREFERENCE:
  - default
DISTRIBUTED_STORAGE_CONFIG:
  default:
  - S3Storage
  - host: s3.us-east-2.amazonaws.com
    s3_access_key: AK....C
    s3_bucket: quayprowci10644
    s3_secret_key: r.....V
    storage_path: /quay
SUPER_USERS:
  - whuquay
FEATURE_ANONYMOUS_ACCESS: true 
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_PROXY_CACHE: true
BROWSER_API_CALLS_XHR_ONLY: false
CREATE_PRIVATE_REPO_ON_PUSH: true
CREATE_NAMESPACE_ON_PUSH: true
FEATURE_USERNAME_CONFIRMATION: false
FEATURE_USER_INITIALIZE: true 
AUTHENTICATION_TYPE: Database
FEATURE_UI_V2: true
FEATURE_LISTEN_IP_VERSION: IPv4
FEATURE_SUPERUSERS_FULL_ACCESS: true
QUOTA_BACKFILL: true
FEATURE_GARBAGE_COLLECTION: true
GARBAGE_COLLECTION_FREQUENCY: 10
DEFAULT_TAG_EXPIRATION: 2w
TAG_EXPIRATION_OPTIONS:
- 2w
- 10s
FEATURE_SECURITY_SCANNER_NOTIFY_ON_NEW_INDEX: True
FEATURE_GENERAL_OCI_SUPPORT: true
DEFAULT_SYSTEM_REJECT_QUOTA_BYTES: 102400000000

3. Create a github build trigger in UI

create a gitlab build trigger

4. Trigger a build process in UI

Actual results:

The build process will be stuck at "Waiting for available build worker" phase.

the build process will be stuck

Got warning message in quay pod log

builder stdout | 2023-07-04 07:45:03,109 [55] [DEBUG] [buildman.manager.executor] Kubernetes request: GET https://api.ci-osd-ccs-99vd.ev09.s1.devshift.org:6443/apis/batch/v1/namespaces/virtual-builds/jobs: {'params': {'labelSelector': 'build,time,manager,quay-sha'}, 'verify': '/conf/stack/extra_ca_certs/build_cluster.crt', 'timeout': 20, 'headers': {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IlBTd2ZXT05mOVQzRzBWVmc1Q2paNHlUVWtaZ1o5U1pJczBiNzZTbDlRLXMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY4ODUzODMwMCwiaWF0IjoxNjg4NDUxOTAwLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6IjBmYzI1NDhjLWJjMmUtNDk2Yi05ZDQ2LTM0ZjExNTliZTRkZiJ9fSwibmJmIjoxNjg4NDUxOTAwLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.Zx-Oxnfa6Z0H7Xmn0R4KDQmAVDiP6DVBiRiy0A-OUihId8GHq1CDuizAgMLrvZjmQUKjtlEXoflvBdYNCl2m5nsmsC5Lf3dFt_Gn366oAgB5QUqg3w5TSRlLgG5-9EB8HGhR9LGxImuBZPhTtFuARZWlXvalxlT_rhqty8-KOd2zP4Gid471hmHAY7bQJHId-p97t1jWj0Y8nUdlhgtArxsxzQUY_EkGaqnC_1mEWb2FLc_52W1jOWDqy4EgosBiR2VYo_FNkPVg54O-mnvwjuvIjPXh-90tipnYKULODY_T0nwpKRbYYF7m-ttnKLevnj3ocOHTCVAIdISiqGyaxHLaiAv_nPQPQNCYIcgMoMDevL1qwcDeY1LydVUJnJDprzXK7tG22zsuXGvOWVpUYPVx-IzDsu9yefNvzLhObHgxBk3_vqzK4oXzjsjNT86IF2EuDzb6p0C92iGpON85rGQknf6dGsd3XIKzkz84G7Xn-0CHSjyGLBMks53EHuaMi6U3qLps1Pdw-k-qmB0eGZheltuI159bygjj4uZg0Aml5Ad_n_fZMjcdU30Bjb6ex5jJO_suJ6nqhVBHtDr9mxTjbWraVoQura5tkvGDVOuPhc80Q_wSp0Y8IYBylcSTWvzuY_sUXBWt7T_zh3IAsfPDJXPMphAT33w5y8UNWzg'}}
builder stdout | 2023-07-04 07:45:03,112 [55] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): api.ci-osd-ccs-99vd.ev09.s1.devshift.org:6443
builder stdout | 2023-07-04 07:45:03,121 [55] [WARNING] [buildman.manager.ephemeral] Failed to get worker count from executors: HTTPSConnectionPool(host='api.ci-osd-ccs-99vd.ev09.s1.devshift.org', port=6443): Max retries exceeded with url: /apis/batch/v1/namespaces/virtual-builds/jobs?labelSelector=build%2Ctime%2Cmanager%2Cquay-sha (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')))

Expected results:

The build process should finish successfully

Additional info:

In OSD cluster

$ oc get pod
NAME                                               READY   STATUS      RESTARTS        AGE
quay-operator.v3.9.0-5fdd587d7-7ff78               1/1     Running     0               3h53m
quayregistry-clair-app-585c679859-d4zxw            1/1     Running     0               113m
quayregistry-clair-app-585c679859-hmqms            1/1     Running     0               113m
quayregistry-clair-postgres-5845b74855-jb64j       1/1     Running     1 (3h53m ago)   3h53m
quayregistry-quay-app-7b64d749d4-qlsvg             1/1     Running     0               113m
quayregistry-quay-app-7b64d749d4-vvhwb             1/1     Running     0               112m
quayregistry-quay-app-upgrade-psddp                0/1     Completed   2               3h53m
quayregistry-quay-config-editor-5ff6c67755-kkdjv   1/1     Running     0               113m
quayregistry-quay-database-6f8c477f78-pr2ws        1/1     Running     0               3h53m
quayregistry-quay-mirror-554c9477cf-dkspj          1/1     Running     0               112m
quayregistry-quay-mirror-554c9477cf-qdn8l          1/1     Running     0               112m
quayregistry-quay-redis-6f49cd65b9-whq6k           1/1     Running     0               3h8m

$ oc rsh quayregistry-quay-app-7b64d749d4-qlsvg
sh-4.4$ cd /conf/stack/extra_ca_certs/
sh-4.4$ ls
build_cluster.crt  ca-bundle.crt  service-ca.crt
sh-4.4$ curl --cacert ./build_cluster.crt  -X GET -H "Content-Type: application/json" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IlBTd2ZXT05mOVQzRzBWVmc1Q2paNHlUVWtaZ1o5U1pJczBiNzZTbDlRLXMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY4ODUzODMwMCwiaWF0IjoxNjg4NDUxOTAwLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6IjBmYzI1NDhjLWJjMmUtNDk2Yi05ZDQ2LTM0ZjExNTliZTRkZiJ9fSwibmJmIjoxNjg4NDUxOTAwLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.Zx-Oxnfa6Z0H7Xmn0R4KDQmAVDiP6DVBiRiy0A-OUihId8GHq1CDuizAgMLrvZjmQUKjtlEXoflvBdYNCl2m5nsmsC5Lf3dFt_Gn366oAgB5QUqg3w5TSRlLgG5-9EB8HGhR9LGxImuBZPhTtFuARZWlXvalxlT_rhqty8-KOd2zP4Gid471hmHAY7bQJHId-p97t1jWj0Y8nUdlhgtArxsxzQUY_EkGaqnC_1mEWb2FLc_52W1jOWDqy4EgosBiR2VYo_FNkPVg54O-mnvwjuvIjPXh-90tipnYKULODY_T0nwpKRbYYF7m-ttnKLevnj3ocOHTCVAIdISiqGyaxHLaiAv_nPQPQNCYIcgMoMDevL1qwcDeY1LydVUJnJDprzXK7tG22zsuXGvOWVpUYPVx-IzDsu9yefNvzLhObHgxBk3_vqzK4oXzjsjNT86IF2EuDzb6p0C92iGpON85rGQknf6dGsd3XIKzkz84G7Xn-0CHSjyGLBMks53EHuaMi6U3qLps1Pdw-k-qmB0eGZheltuI159bygjj4uZg0Aml5Ad_n_fZMjcdU30Bjb6ex5jJO_suJ6nqhVBHtDr9mxTjbWraVoQura5tkvGDVOuPhc80Q_wSp0Y8IYBylcSTWvzuY_sUXBWt7T_zh3IAsfPDJXPMphAT33w5y8UNWzg" https://api.ci-osd-ccs-99vd.ev09.s1.devshift.org:6443/apis/batch/v1/namespaces/virtual-builds/jobs?labelSelector=build%2Ctime%2Cmanager%2Cquay-sha
{
  "kind": "JobList",
  "apiVersion": "batch/v1",
  "metadata": {
    "resourceVersion": "310289"
  },
  "items": []
}

Attach the whole quay pod log in OSD cluster in Attachment

In a common OCP cluster, with the same test steps. Everything is ok.

the build process in the common OCP cluster

The build logs in quay pod

builder stdout | 2023-07-04 06:21:36,159 [61] [DEBUG] [buildman.manager.executor] Kubernetes request: GET https://api.whu412az44.qe.azure.devcluster.openshift.com:6443/apis/batch/v1/namespaces/virtual-builds/jobs: {'params': {'labelSelector': 'build,time,manager,quay-sha'}, 'verify': '/conf/stack/extra_ca_certs/build_cluster.crt', 'timeout': 20, 'headers': {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Im1DenZZUUh3Ul9HQ2FFN3NTV2NNbDdDbWphRlk1WmpVRjVzaXAycU5icVkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY4ODUzNzYyMSwiaWF0IjoxNjg4NDUxMjIxLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6ImFjZmI3MDFlLTZiMjMtNDUxZS04Mzk4LTgxZmE1Y2RlZTJmMCJ9fSwibmJmIjoxNjg4NDUxMjIxLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.hPEX6Lifi9G2aXWnGRsaSQkfdF2wI1WT2ePYVnG1r0PWe3xwJJ6-98faTdbbtLrq5fwjAsdosGF2mm2m_-2oE0yy8TrEar8YWIGNrghldzoFxUcM2icG8UIEGWzMNEylnXfSgz3xf3fnOBdJyEyjNnSP0BQvv--arEkvqwhXXUeIJKIwOHgwiwYXRrP9fBi7YFVBGAp_cW9xYD23_GfpAlhgHlugLKGgHPwQ46K_sIysphSUZbNbCMZ8rfOuBx7cTQT9h6ho3IvqQs08FQNl4EoCXHqV8MoLSUOh2YL1GXbDhBnssm1pmZysFPp1ebj0pYKJRJZKls49ZpBra_8FPKdpaQ_4mQSNcn_9ktLFhSCO6igj11Vk6EvKHxQFiskPRCs9n0Aqvwe4t4CEgD-knaL7dd0PSfTBZoKNwVQix5XGubTqsnqenV_WMFBI3VZR3D6H1Oh2biJH3mSWmDgS2JbgQzbf2alyfcJTGZfxneMxaBpU7_GOV8gmk7KfFjqAGMfjIPtTUsbfl1lM_J5l2v2PkbP_7wkBvZAjZB173lBl1wtP1hE3noTrfhVXovnziN6Q0wH6a8m603IetNfSPEgQwBOTylFvLuZGK9x3AnUyhcOIyp5TGnUiaEWJBoSzRhoEM5wff5vwv0zulOygviaUrFiO8hPfGor_8chtmEo'}}
builder stdout | 2023-07-04 06:21:36,162 [61] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): api.whu412az44.qe.azure.devcluster.openshift.com:6443
builder stdout | 2023-07-04 06:21:36,178 [61] [DEBUG] [urllib3.connectionpool] https://api.whu412az44.qe.azure.devcluster.openshift.com:6443 "GET /apis/batch/v1/namespaces/virtual-builds/jobs?labelSelector=build%2Ctime%2Cmanager%2Cquay-sha HTTP/1.1" 200 94