Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1440

Quay Build POD was failed pull image after specify pull image credentials in config.yaml

XMLWordPrintable

    • False
    • False
    • Undefined

      Description:

      This is an issue found when using Quay to build image, after update quay config.yaml, Quay build Job was failed to start POD, the error message is "Failed to pull image "brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos:v3.4.0-13", but in config.yaml it already contains two parameters "QUAY_USERNAME" and "QUAY_PASSWORD", but Quay build POD didn't use that credentials to pull target build image, see screenshot attached, and quay_app_pod logs.

      Docs: https://docs.projectquay.io/use_quay.html#setting-up-builders 

      Index Image:

      brew.registry.redhat.io/rh-osbs/iib:39377

      Quay Image:

      oc get pod quayawsdemo-quay-app-7d9d6b74df-cghg5 -o json | jq '.spec.containers[0].image'
"registry.redhat.io/quay/quay@sha256:248988b8d20b66572e4123ce85424f6e34eac1e670306a0bad6d30b702f3aba1"

      Quay App POD logs:

      builder stdout | 2021-01-19 03:29:09,011 [83] [DEBUG] [buildman.manager.executor] Using service account token for Kubernetes authentication
builder stdout | 2021-01-19 03:29:09,011 [83] [DEBUG] [buildman.manager.executor] Executor config: {'EXECUTOR': 'kubernetes', 'DEBUG': True, 'BUILDER_NAMESPACE': 'builder', 'K8S_API_SERVER': 'api.ci-ln-mn4f372-86010.origin-ci-int-aws.dev.rhcloud.com:6443', 'VOLUME_SIZE': '8G', 'KUBERNETES_DISTRIBUTION': 'openshift', 'CONTAINER_MEMORY_LIMITS': '5120Mi', 'CONTAINER_CPU_LIMITS': '1000m', 'CONTAINER_MEMORY_REQUEST': '3968Mi', 'CONTAINER_CPU_REQUEST': '500m', 'NODE_SELECTOR_LABEL_KEY': 'node-role.kubernetes.io/worker', 'NODE_SELECTOR_LABEL_VALUE': '', 'CONTAINER_RUNTIME': 'podman', 'SERVICE_ACCOUNT_NAME': 'quay-builder', 'SERVICE_ACCOUNT_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6ImJYcVg5ay0zekN3NktoUFJqNDhPbDJVNXpqSnc1bjFCMl8yVjFzTURLRncifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJidWlsZGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InF1YXktYnVpbGRlci10b2tlbi14NmtteCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJxdWF5LWJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwODA3ZmNlMC02NjBmLTRkNGUtYTczZi03ZWM3NjQ1MDE1OWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6YnVpbGRlcjpxdWF5LWJ1aWxkZXIifQ.H6WpjqUmFe5BT7mQyfWSTcrg5RciA-l1VYAGvtN_hjb0G22fvKeiYH4g-Qdfu98PmKv80ubPYe575b5bhUKbQx-dCQt9D9jVoLpE0gfun0m79n3KdW6qS_3L5EaRf6NS0fovx6GNCANov7-jV16m5_Cfv33zdkoTbTfo5dy2e76tYFAleC9UT62qDpZTPbvex-O6Oqg_T1zVuMmI4NtoGTpswK6cGZ6EEkXimyr03IjIkr20G-6xlMg0zGJMFwoPYtQ4RMoBo8netMTOzTm4P6yktF5PAzF7LhDKHyl07kPnn1XjHPLFS0VzEQIuk1VIXC6_Z8O7SHFye3Pnvn3Qmw', 'QUAY_USERNAME': '******', 'QUAY_PASSWORD': 'eyJhbGciOiJSUzUxMiJ9.eyJzdWIiOiIzNWE5ZTBlZGUwMjU0YzYyODk1ZjFkODI3M2ExMGQ4ZiJ9.jN7QzvnHEtLl1MhXxy0l66w6P177IfcflCPeK0jIZXuQ5******D3aluFMAgcFwGKrgPoOTcL6AOCBf7gQsvJ2lFhJq-fcqodyBedJLOXhPYr-3xyU-zApxkSc7zO9cLrR_2Hpcw7LYRoiUVnx0G1GzCNpnvGeVNtBnf36A9lNsZkie2MCc132jLQUw7nUj2IwqyLtSN9Aku1SUuuoRfSMXyOw_XEyFXcIRabWTCPa8cdtPcRw7bYxstQJQ8aVniR6kg9V6WBQJojL_OCiLYkshknMPjfaCgPKsd7_rNWYAhIxoXKgygF3Db7Akzy9YBNCImLX6Cw9qp2z2rPWJtyjiHpTr499sT9GynFmPo0QZ4ZM0C68PnzX2NGh-W38NVy1Ir_7EVzX2pUUM7LhGJUxhMcVZ73on0kb5mQ8_wv-YJiiuqrXm0y-uLUyPtaTjENs4abHwfemk2451uho8yzWvuRDxIqrqlkNR8ZjNiDk-pRnmcMByvjyFpqq50GjXIN_7l_tGP6mdArE-NNmUurbsfM2PZdan2MO61hDVhGDZrEzcrVXL2_9OUW2EKk13yjAdseLUMZZCHGSiW-a4ugU0RuwQkgCAyItQv1MmcEYoOJmKt4SKJd0nhJ07y6v_iywRfr3xjHgLQ5Mz-nnQVwxyBetSmYdPzmVTo', 'WORKER_IMAGE': 'brew.registry.redhat.io/rh-osbs/quay-quay-builder', 'WORKER_TAG': 'v3.4.0-12', 'BUILDER_VM_CONTAINER_IMAGE': 'brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos:v3.4.0-13', 'SETUP_TIME': 180, 'MINIMUM_RETRY_THRESHOLD': 0, 'SSH_AUTHORIZED_KEYS': ['ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN8nUzLnPHq9o6Crika8brT4i5CL0a0azoHJoHe02BH8/vgDyhgHin+1qDrHA414t6smDIhYRM/L503J0kD2/jUTPVqeFNmbxbzXnEXWv2RaAyKChMzw2PkrKiLntY4CxcukdSN6lqtJa8TH3/Vmy/YUOMJOKWEsYkg6qojDWPYbFHMubm6JWPydiEJJYPYCH7tHPaq4Y3CWNw+jx2sL69Sltnsdc/oj5Icl+u/ClF7lm0LPXkrkUF745ktCg6r06dLju3Ap+A0HJ/doTpCymZrt88eEy0RqW9koDYPJsRm380caT0J4wux3HlZiHP0b1mhx9pp7DB0FuhZHxeQawGs4V3aYDisBE27YMoMBqoCmBOqkVqC7uY47HOYiS15YHpriCXSnflE628e6a7zfFVV+CcrcqtcqPltZlXmbm2PeQY547VphB1nivinALOVM+CcSgOchX1Phmj63nXKt/IbsUJhUnZQicFhh2bJzXWKBtCQkodwTnu90RaKJN2pn8= lizhang@lzha-mac']}
builder stdout | 2021-01-19 03:29:09,011 [83] [DEBUG] [buildman.manager.executor] Kubernetes request: GET https://api.ci-ln-mn4f372-86010.origin-ci-int-aws.dev.rhcloud.com:6443/apis/batch/v1/namespaces/builder/jobs: {'params': {'labelSelector': 'build,time,manager,quay-sha'}, 'timeout': 20, 'headers': {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImJYcVg5ay0zekN3NktoUFJqNDhPbDJVNXpqSnc1bjFCMl8yVjFzTURLRncifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJidWlsZGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InF1YXktYnVpbGRlci10b2tlbi14NmtteCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJxdWF5LWJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwODA3ZmNlMC02NjBmLTRkNGUtYTczZi03ZWM3NjQ1MDE1OWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6YnVpbGRlcjpxdWF5LWJ1aWxkZXIifQ.H6WpjqUmFe5BT7mQyfWSTcrg5RciA-l1VYAGvtN_hjb0G22fvKeiYH4g-Qdfu98PmKv80ubPYe575b5bhUKbQx-dCQt9D9jVoLpE0gfun0m79n3KdW6qS_3L5EaRf6NS0fovx6GNCANov7-jV16m5_Cfv33zdkoTbTfo5dy2e76tYFAleC9UT62qDpZTPbvex-O6Oqg_T1zVuMmI4NtoGTpswK6cGZ6EEkXimyr03IjIkr20G-6xlMg0zGJMFwoPYtQ4RMoBo8netMTOzTm4P6yktF5PAzF7LhDKHyl07kPnn1XjHPLFS0VzEQIuk1VIXC6_Z8O7SHFye3Pnvn3Qmw'}}
builder stdout | 2021-01-19 03:29:09,290 [83] [WARNING] [buildman.manager.ephemeral] Could not schedule build 520501c4-6383-45dc-997d-2178cffe17db. Number of workers at capacity: 3.
builder stdout | 2021-01-19 03:29:09,299 [83] [DEBUG] [data.database] Disconnecting from database.
builder stdout | 2021-01-19 03:29:09,299 [83] [WARNING] [buildman.manager.ephemeral] Unsuccessful schedule. Build ID: 520501c4-6383-45dc-997d-2178cffe17db. Retry restored.
builder stdout | 2021-01-19 03:29:09,301 [83] [DEBUG] [buildman.orchestrator] publishing event: {"event": 1, "key": "buildman/production/lock/520501c4-6383-45dc-997d-2178cffe17db", "value": ""}
builder stdout | 2021-01-19 03:29:09,302 [83] [DEBUG] [buildman.orchestrator] publishing event: {"event": 3, "key": "buildman/production/building/520501c4-6383-45dc-997d-2178cffe17db", "value": "{\"job_queue_item\": {\"id\": 4, \"body\": \"{\\\"build_uuid\\\": \\\"520501c4-6383-45dc-997d-2178cffe17db\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1611034148, \"last_heartbeat\": null}"}

      Config.yaml:

      BUILDMAN_HOSTNAME: quayawsdemo-quay-builder-quay34.apps.quay-501.qe.devcluster.openshift.com
BUILD_MANAGER:
- ephemeral
- ALLOWED_WORKER_COUNT: 3
  ORCHESTRATOR_PREFIX: buildman/production/
  ORCHESTRATOR:
    REDIS_HOST: quayawsdemo-quay-redis
  EXECUTORS:
  - EXECUTOR: kubernetes
    DEBUG: true   
    BUILDER_NAMESPACE: builder
    K8S_API_SERVER: api.ci-ln-mn4f372-86010.origin-ci-int-aws.dev.rhcloud.com:6443
    VOLUME_SIZE: 8G
    KUBERNETES_DISTRIBUTION: openshift
    CONTAINER_MEMORY_LIMITS: 5120Mi
    CONTAINER_CPU_LIMITS: 1000m
    CONTAINER_MEMORY_REQUEST: 3968Mi
    CONTAINER_CPU_REQUEST: 500m
    NODE_SELECTOR_LABEL_KEY: node-role.kubernetes.io/worker
    NODE_SELECTOR_LABEL_VALUE: ''
    CONTAINER_RUNTIME: podman
    SERVICE_ACCOUNT_NAME: quay-builder
    SERVICE_ACCOUNT_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6ImJYcVg5ay0zekN3NktoUFJqNDhPbDJVNXpqSnc1bjFCMl8yVjFzTURLRncifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJidWlsZGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InF1YXktYnVpbGRlci10b2tlbi14NmtteCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJxdWF5LWJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwODA3ZmNlMC02NjBmLTRkNGUtYTczZi03ZWM3NjQ1MDE1OWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6YnVpbGRlcjpxdWF5LWJ1aWxkZXIifQ.H6WpjqUmFe5BT7mQyfWSTcrg5RciA-l1VYAGvtN_hjb0G22fvKeiYH4g-Qdfu98PmKv80ubPYe575b5bhUKbQx-dCQt9D9jVoLpE0gfun0m79n3KdW6qS_3L5EaRf6NS0fovx6GNCANov7-jV16m5_Cfv33zdkoTbTfo5dy2e76tYFAleC9UT62qDpZTPbvex-O6Oqg_T1zVuMmI4NtoGTpswK6cGZ6EEkXimyr03IjIkr20G-6xlMg0zGJMFwoPYtQ4RMoBo8netMTOzTm4P6yktF5PAzF7LhDKHyl07kPnn1XjHPLFS0VzEQIuk1VIXC6_Z8O7SHFye3Pnvn3Qmw
    QUAY_USERNAME: *****
    QUAY_PASSWORD: *****
    WORKER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder
    WORKER_TAG: v3.4.0-12
    BUILDER_VM_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos:v3.4.0-13
    SETUP_TIME: 180
    MINIMUM_RETRY_THRESHOLD: 0
    SSH_AUTHORIZED_KEYS:
    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN8nUzLnPHq9o6Crika8brT4i5CL0a0azoHJoHe02BH8/vgDyhgHin+1qDrHA414t6smDIhYRM/L503J0kD2/jUTPVqeFNmbxbzXnEXWv2RaAyKChMzw2PkrKiLntY4CxcukdSN6lqtJa8TH3/Vmy/YUOMJOKWEsYkg6qojDWPYbFHMubm6JWPydiEJJYPYCH7tHPaq4Y3CWNw+jx2sL69Sltnsdc/oj5Icl+u/ClF7lm0LPXkrkUF745ktCg6r06dLju3Ap+A0HJ/doTpCymZrt88eEy0RqW9koDYPJsRm380caT0J4wux3HlZiHP0b1mhx9pp7DB0FuhZHxeQawGs4V3aYDisBE27YMoMBqoCmBOqkVqC7uY47HOYiS15YHpriCXSnflE628e6a7zfFVV+CcrcqtcqPltZlXmbm2PeQY547VphB1nivinALOVM+CcSgOchX1Phmj63nXKt/IbsUJhUnZQicFhh2bJzXWKBtCQkodwTnu90RaKJN2pn8= lizhang@lzha-mac

      Steps:

      1. Deploy Quay 3.4 with V3.4 TNG Operator on AWS using AWS S3 object Storage
      2. Create a new OCP baremetal Cluster to build image, and create SA with all required steps mentioned in doc
      3. Upload the OCP API Server CA Cert of Baremetal Cluster via config editor
      4. Update config.yaml, including pull image credentials(QUAY_USERNAME and QUAY_PASSWORD)
      5. Delete Quay App Pod
      6. Start new build on Quay console after new quay app pod is ready
      7. Check new build status

      Expected Results:

      Build Job POD start successfully.

      Actual Results:

      Build Job POD was failed start with error message "failed to pull image". 

        1. quay_app_pod.logs
          8.08 MB
        2. Quay build POD was failed to start.png
          Quay build POD was failed to start.png
          355 kB

              rhn-coreos-amerdler Alec Merdler (Inactive)
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: