Loading...

Details

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: quay-v3.4.0
Affects Version/s: quay-v3.4.0
Component/s: quay-builder
Labels:
- quay-enterprise

Blocked:
False
Ready:
False
Release Note Text:
Undefined

RICE Score:
0

SFDC Cases Links:
SFDC Cases Counter:

Description

Description:

This is an issue found when using Quay to build image, after update quay config.yaml, Quay build Job was failed to start POD, the error message is "Failed to pull image "brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos:v3.4.0-13", but in config.yaml it already contains two parameters "QUAY_USERNAME" and "QUAY_PASSWORD", but Quay build POD didn't use that credentials to pull target build image, see screenshot attached, and quay_app_pod logs.

Docs: https://docs.projectquay.io/use_quay.html#setting-up-builders

Index Image:

brew.registry.redhat.io/rh-osbs/iib:39377

Quay Image:

oc get pod quayawsdemo-quay-app-7d9d6b74df-cghg5 -o json | jq '.spec.containers[0].image'
"registry.redhat.io/quay/quay@sha256:248988b8d20b66572e4123ce85424f6e34eac1e670306a0bad6d30b702f3aba1"

Quay App POD logs:

builder stdout | 2021-01-19 03:29:09,011 [83] [DEBUG] [buildman.manager.executor] Using service account token for Kubernetes authentication
builder stdout | 2021-01-19 03:29:09,011 [83] [DEBUG] [buildman.manager.executor] Executor config: {'EXECUTOR': 'kubernetes', 'DEBUG': True, 'BUILDER_NAMESPACE': 'builder', 'K8S_API_SERVER': 'api.ci-ln-mn4f372-86010.origin-ci-int-aws.dev.rhcloud.com:6443', 'VOLUME_SIZE': '8G', 'KUBERNETES_DISTRIBUTION': 'openshift', 'CONTAINER_MEMORY_LIMITS': '5120Mi', 'CONTAINER_CPU_LIMITS': '1000m', 'CONTAINER_MEMORY_REQUEST': '3968Mi', 'CONTAINER_CPU_REQUEST': '500m', 'NODE_SELECTOR_LABEL_KEY': 'node-role.kubernetes.io/worker', 'NODE_SELECTOR_LABEL_VALUE': '', 'CONTAINER_RUNTIME': 'podman', 'SERVICE_ACCOUNT_NAME': 'quay-builder', 'SERVICE_ACCOUNT_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6ImJYcVg5ay0zekN3NktoUFJqNDhPbDJVNXpqSnc1bjFCMl8yVjFzTURLRncifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJidWlsZGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InF1YXktYnVpbGRlci10b2tlbi14NmtteCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJxdWF5LWJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwODA3ZmNlMC02NjBmLTRkNGUtYTczZi03ZWM3NjQ1MDE1OWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6YnVpbGRlcjpxdWF5LWJ1aWxkZXIifQ.H6WpjqUmFe5BT7mQyfWSTcrg5RciA-l1VYAGvtN_hjb0G22fvKeiYH4g-Qdfu98PmKv80ubPYe575b5bhUKbQx-dCQt9D9jVoLpE0gfun0m79n3KdW6qS_3L5EaRf6NS0fovx6GNCANov7-jV16m5_Cfv33zdkoTbTfo5dy2e76tYFAleC9UT62qDpZTPbvex-O6Oqg_T1zVuMmI4NtoGTpswK6cGZ6EEkXimyr03IjIkr20G-6xlMg0zGJMFwoPYtQ4RMoBo8netMTOzTm4P6yktF5PAzF7LhDKHyl07kPnn1XjHPLFS0VzEQIuk1VIXC6_Z8O7SHFye3Pnvn3Qmw', 'QUAY_USERNAME': '******', 'QUAY_PASSWORD': 'eyJhbGciOiJSUzUxMiJ9.eyJzdWIiOiIzNWE5ZTBlZGUwMjU0YzYyODk1ZjFkODI3M2ExMGQ4ZiJ9.jN7QzvnHEtLl1MhXxy0l66w6P177IfcflCPeK0jIZXuQ5******D3aluFMAgcFwGKrgPoOTcL6AOCBf7gQsvJ2lFhJq-fcqodyBedJLOXhPYr-3xyU-zApxkSc7zO9cLrR_2Hpcw7LYRoiUVnx0G1GzCNpnvGeVNtBnf36A9lNsZkie2MCc132jLQUw7nUj2IwqyLtSN9Aku1SUuuoRfSMXyOw_XEyFXcIRabWTCPa8cdtPcRw7bYxstQJQ8aVniR6kg9V6WBQJojL_OCiLYkshknMPjfaCgPKsd7_rNWYAhIxoXKgygF3Db7Akzy9YBNCImLX6Cw9qp2z2rPWJtyjiHpTr499sT9GynFmPo0QZ4ZM0C68PnzX2NGh-W38NVy1Ir_7EVzX2pUUM7LhGJUxhMcVZ73on0kb5mQ8_wv-YJiiuqrXm0y-uLUyPtaTjENs4abHwfemk2451uho8yzWvuRDxIqrqlkNR8ZjNiDk-pRnmcMByvjyFpqq50GjXIN_7l_tGP6mdArE-NNmUurbsfM2PZdan2MO61hDVhGDZrEzcrVXL2_9OUW2EKk13yjAdseLUMZZCHGSiW-a4ugU0RuwQkgCAyItQv1MmcEYoOJmKt4SKJd0nhJ07y6v_iywRfr3xjHgLQ5Mz-nnQVwxyBetSmYdPzmVTo', 'WORKER_IMAGE': 'brew.registry.redhat.io/rh-osbs/quay-quay-builder', 'WORKER_TAG': 'v3.4.0-12', 'BUILDER_VM_CONTAINER_IMAGE': 'brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos:v3.4.0-13', 'SETUP_TIME': 180, 'MINIMUM_RETRY_THRESHOLD': 0, 'SSH_AUTHORIZED_KEYS': ['ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN8nUzLnPHq9o6Crika8brT4i5CL0a0azoHJoHe02BH8/vgDyhgHin+1qDrHA414t6smDIhYRM/L503J0kD2/jUTPVqeFNmbxbzXnEXWv2RaAyKChMzw2PkrKiLntY4CxcukdSN6lqtJa8TH3/Vmy/YUOMJOKWEsYkg6qojDWPYbFHMubm6JWPydiEJJYPYCH7tHPaq4Y3CWNw+jx2sL69Sltnsdc/oj5Icl+u/ClF7lm0LPXkrkUF745ktCg6r06dLju3Ap+A0HJ/doTpCymZrt88eEy0RqW9koDYPJsRm380caT0J4wux3HlZiHP0b1mhx9pp7DB0FuhZHxeQawGs4V3aYDisBE27YMoMBqoCmBOqkVqC7uY47HOYiS15YHpriCXSnflE628e6a7zfFVV+CcrcqtcqPltZlXmbm2PeQY547VphB1nivinALOVM+CcSgOchX1Phmj63nXKt/IbsUJhUnZQicFhh2bJzXWKBtCQkodwTnu90RaKJN2pn8= lizhang@lzha-mac']}
builder stdout | 2021-01-19 03:29:09,011 [83] [DEBUG] [buildman.manager.executor] Kubernetes request: GET https://api.ci-ln-mn4f372-86010.origin-ci-int-aws.dev.rhcloud.com:6443/apis/batch/v1/namespaces/builder/jobs: {'params': {'labelSelector': 'build,time,manager,quay-sha'}, 'timeout': 20, 'headers': {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImJYcVg5ay0zekN3NktoUFJqNDhPbDJVNXpqSnc1bjFCMl8yVjFzTURLRncifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJidWlsZGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InF1YXktYnVpbGRlci10b2tlbi14NmtteCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJxdWF5LWJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwODA3ZmNlMC02NjBmLTRkNGUtYTczZi03ZWM3NjQ1MDE1OWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6YnVpbGRlcjpxdWF5LWJ1aWxkZXIifQ.H6WpjqUmFe5BT7mQyfWSTcrg5RciA-l1VYAGvtN_hjb0G22fvKeiYH4g-Qdfu98PmKv80ubPYe575b5bhUKbQx-dCQt9D9jVoLpE0gfun0m79n3KdW6qS_3L5EaRf6NS0fovx6GNCANov7-jV16m5_Cfv33zdkoTbTfo5dy2e76tYFAleC9UT62qDpZTPbvex-O6Oqg_T1zVuMmI4NtoGTpswK6cGZ6EEkXimyr03IjIkr20G-6xlMg0zGJMFwoPYtQ4RMoBo8netMTOzTm4P6yktF5PAzF7LhDKHyl07kPnn1XjHPLFS0VzEQIuk1VIXC6_Z8O7SHFye3Pnvn3Qmw'}}
builder stdout | 2021-01-19 03:29:09,290 [83] [WARNING] [buildman.manager.ephemeral] Could not schedule build 520501c4-6383-45dc-997d-2178cffe17db. Number of workers at capacity: 3.
builder stdout | 2021-01-19 03:29:09,299 [83] [DEBUG] [data.database] Disconnecting from database.
builder stdout | 2021-01-19 03:29:09,299 [83] [WARNING] [buildman.manager.ephemeral] Unsuccessful schedule. Build ID: 520501c4-6383-45dc-997d-2178cffe17db. Retry restored.
builder stdout | 2021-01-19 03:29:09,301 [83] [DEBUG] [buildman.orchestrator] publishing event: {"event": 1, "key": "buildman/production/lock/520501c4-6383-45dc-997d-2178cffe17db", "value": ""}
builder stdout | 2021-01-19 03:29:09,302 [83] [DEBUG] [buildman.orchestrator] publishing event: {"event": 3, "key": "buildman/production/building/520501c4-6383-45dc-997d-2178cffe17db", "value": "{\"job_queue_item\": {\"id\": 4, \"body\": \"{\\\"build_uuid\\\": \\\"520501c4-6383-45dc-997d-2178cffe17db\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1611034148, \"last_heartbeat\": null}"}

Config.yaml:

BUILDMAN_HOSTNAME: quayawsdemo-quay-builder-quay34.apps.quay-501.qe.devcluster.openshift.com
BUILD_MANAGER:
- ephemeral
- ALLOWED_WORKER_COUNT: 3
  ORCHESTRATOR_PREFIX: buildman/production/
  ORCHESTRATOR:
    REDIS_HOST: quayawsdemo-quay-redis
  EXECUTORS:
  - EXECUTOR: kubernetes
    DEBUG: true   
    BUILDER_NAMESPACE: builder
    K8S_API_SERVER: api.ci-ln-mn4f372-86010.origin-ci-int-aws.dev.rhcloud.com:6443
    VOLUME_SIZE: 8G
    KUBERNETES_DISTRIBUTION: openshift
    CONTAINER_MEMORY_LIMITS: 5120Mi
    CONTAINER_CPU_LIMITS: 1000m
    CONTAINER_MEMORY_REQUEST: 3968Mi
    CONTAINER_CPU_REQUEST: 500m
    NODE_SELECTOR_LABEL_KEY: node-role.kubernetes.io/worker
    NODE_SELECTOR_LABEL_VALUE: ''
    CONTAINER_RUNTIME: podman
    SERVICE_ACCOUNT_NAME: quay-builder
    SERVICE_ACCOUNT_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6ImJYcVg5ay0zekN3NktoUFJqNDhPbDJVNXpqSnc1bjFCMl8yVjFzTURLRncifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJidWlsZGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InF1YXktYnVpbGRlci10b2tlbi14NmtteCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJxdWF5LWJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwODA3ZmNlMC02NjBmLTRkNGUtYTczZi03ZWM3NjQ1MDE1OWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6YnVpbGRlcjpxdWF5LWJ1aWxkZXIifQ.H6WpjqUmFe5BT7mQyfWSTcrg5RciA-l1VYAGvtN_hjb0G22fvKeiYH4g-Qdfu98PmKv80ubPYe575b5bhUKbQx-dCQt9D9jVoLpE0gfun0m79n3KdW6qS_3L5EaRf6NS0fovx6GNCANov7-jV16m5_Cfv33zdkoTbTfo5dy2e76tYFAleC9UT62qDpZTPbvex-O6Oqg_T1zVuMmI4NtoGTpswK6cGZ6EEkXimyr03IjIkr20G-6xlMg0zGJMFwoPYtQ4RMoBo8netMTOzTm4P6yktF5PAzF7LhDKHyl07kPnn1XjHPLFS0VzEQIuk1VIXC6_Z8O7SHFye3Pnvn3Qmw
    QUAY_USERNAME: *****
    QUAY_PASSWORD: *****
    WORKER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder
    WORKER_TAG: v3.4.0-12
    BUILDER_VM_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos:v3.4.0-13
    SETUP_TIME: 180
    MINIMUM_RETRY_THRESHOLD: 0
    SSH_AUTHORIZED_KEYS:
    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN8nUzLnPHq9o6Crika8brT4i5CL0a0azoHJoHe02BH8/vgDyhgHin+1qDrHA414t6smDIhYRM/L503J0kD2/jUTPVqeFNmbxbzXnEXWv2RaAyKChMzw2PkrKiLntY4CxcukdSN6lqtJa8TH3/Vmy/YUOMJOKWEsYkg6qojDWPYbFHMubm6JWPydiEJJYPYCH7tHPaq4Y3CWNw+jx2sL69Sltnsdc/oj5Icl+u/ClF7lm0LPXkrkUF745ktCg6r06dLju3Ap+A0HJ/doTpCymZrt88eEy0RqW9koDYPJsRm380caT0J4wux3HlZiHP0b1mhx9pp7DB0FuhZHxeQawGs4V3aYDisBE27YMoMBqoCmBOqkVqC7uY47HOYiS15YHpriCXSnflE628e6a7zfFVV+CcrcqtcqPltZlXmbm2PeQY547VphB1nivinALOVM+CcSgOchX1Phmj63nXKt/IbsUJhUnZQicFhh2bJzXWKBtCQkodwTnu90RaKJN2pn8= lizhang@lzha-mac

Steps:

Deploy Quay 3.4 with V3.4 TNG Operator on AWS using AWS S3 object Storage
Create a new OCP baremetal Cluster to build image, and create SA with all required steps mentioned in doc
Upload the OCP API Server CA Cert of Baremetal Cluster via config editor
Update config.yaml, including pull image credentials(QUAY_USERNAME and QUAY_PASSWORD)
Delete Quay App Pod
Start new build on Quay console after new quay app pod is ready
Check new build status

Expected Results:

Build Job POD start successfully.

Actual Results:

Build Job POD was failed start with error message "failed to pull image".

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

quay_app_pod.logs
8.08 MB
2021/01/19 1:15 AM
Quay build POD was failed to start.png
355 kB
2021/01/19 1:14 AM

Quay Build POD was failed pull image after specify pull image credentials in config.yaml

Details

Description

Attachments

Attachments

Activity

People

Dates