Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1803

Podman pull image from quay hit 400 error code

XMLWordPrintable

      Description:

      This is an issue found when use podman to pull image from quay 3.5,  after push image to Quay 3.5 registry, to use podman to pull image, hit error "Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)", see quay app pod logs attached.

      Note: This quay was deployed on OCP with FIPS enabled.

      podman --version
podman version 2.0.5

podman login --tls-verify=false quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com
Username: quay
Password: 
Login Succeeded!

podman pull --tls-verify=false quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo:latest
Trying to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo:latest...
  Error fetching blob: invalid status code from registry 400 (Bad Request)
Error: unable to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo:latest: Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)

podman pull --tls-verify=false quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo@sha256:926270b9ef2150159dbd691a76bb11960934bf4a15b69899fc13f8e058d511e7
Trying to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo@sha256:926270b9ef2150159dbd691a76bb11960934bf4a15b69899fc13f8e058d511e7...
  Error fetching blob: invalid status code from registry 400 (Bad Request)
Error: unable to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo@sha256:926270b9ef2150159dbd691a76bb11960934bf4a15b69899fc13f8e058d511e7: Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)

      Quay Version:

      oc get pod
NAME                                               READY   STATUS      RESTARTS   AGE
quay-operator.v3.5.0-7489b8c4f-r4r6j               1/1     Running     0          47m
quayregistry-clair-app-77c4cb85bc-9frx4            1/1     Running     0          29m
quayregistry-clair-postgres-6866688489-jt78f       1/1     Running     1          38m
quayregistry-quay-app-8c9f7894-chn94               1/1     Running     0          92s
quayregistry-quay-config-editor-86cfb5ff5b-4h852   1/1     Running     0          29m
quayregistry-quay-database-5659885fdc-l5k2q        1/1     Running     0          29m
quayregistry-quay-mirror-86864576f8-pt25x          1/1     Running     0          29m
quayregistry-quay-postgres-init-2ndq5              0/1     Completed   0          29m
quayregistry-quay-redis-76d5659fd8-smjtp           1/1     Running     0          38m

oc get pod quayregistry-quay-app-8c9f7894-chn94 -o json | jq '.spec.containers[0].image'
"registry.redhat.io/quay/quay-rhel8@sha256:0444c7b452a14e0c87ee56f9aa72c54484333c38b0a95de9a4f11f6177273f26"

      Quay App Pod logs:

      gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=HeadBucket) with params: {'url_path': '/quayaws141218', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.17.21 Python/3.8.3 Linux/4.18.0-240.15.1.el8_3.x86_64 Botocore/1.20.21'}, 'body': b'', 'url': 'https://s3.us-east-2.amazonaws.com/quayaws141218', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f10678b4130>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'quayaws141218'}}}
gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f1067961ee0>>
gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f1069057f70>>
gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler <function set_operation_specific_signer at 0x7f106b16c550>
gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.hooks] Event before-sign.s3.HeadBucket: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f10678b4d60>>
gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.utils] Using S3 path style addressing.
gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] Calculating signature using v4 auth.
gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] CanonicalRequest:
gunicorn-registry stdout | HEAD
gunicorn-registry stdout | /quayaws141218
gunicorn-registry stdout | host:s3.us-east-2.amazonaws.com
gunicorn-registry stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
gunicorn-registry stdout | x-amz-date:20210329T021534Z
gunicorn-registry stdout | host;x-amz-content-sha256;x-amz-date
gunicorn-registry stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] StringToSign:
gunicorn-registry stdout | AWS4-HMAC-SHA256
gunicorn-registry stdout | 20210329T021534Z
gunicorn-registry stdout | 20210329/us-east-1/s3/aws4_request
gunicorn-registry stdout | 1908611727337defb71bd7ae158ff8c2388cf2713b25e4d686e2bebcd954d140
gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] Signature:
gunicorn-registry stdout | 8c99622d9ebd4d17b70410c1f6015fcfa04eb7049ac925e19ab5d814f60f8f24
gunicorn-registry stdout | 2021-03-29 02:15:34,091 [251] [DEBUG] [botocore.endpoint] Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://s3.us-east-2.amazonaws.com/quayaws141218, headers={'User-Agent': b'Boto3/1.17.21 Python/3.8.3 Linux/4.18.0-240.15.1.el8_3.x86_64 Botocore/1.20.21', 'X-Amz-Date': b'20210329T021534Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAUMQAHCJON275SXFZ/20210329/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8c99622d9ebd4d17b70410c1f6015fcfa04eb7049ac925e19ab5d814f60f8f24'}>
gunicorn-registry stdout | 2021-03-29 02:15:34,091 [251] [DEBUG] [botocore.httpsession] Certificate path: /usr/local/lib/python3.8/site-packages/certifi/cacert.pem
gunicorn-registry stdout | 2021-03-29 02:15:34,092 [251] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): s3.us-east-2.amazonaws.com:443
gunicorn-registry stdout | 2021-03-29 02:15:34,172 [251] [DEBUG] [urllib3.connectionpool] https://s3.us-east-2.amazonaws.com:443 "HEAD /quayaws141218 HTTP/1.1" 400 0

      Steps:

      1. Deploy Quay 3.5.0 Operator to all OCP namespace
      2. Deploy quay with quay 3.5 Operator with using AWS S3 as backend registry storage
      3. Create new Quay Org and image repository
      4. Push image to new image repository
      5. Pull image from quay with podman

      Expected Results:

      Pull image with podman from quay complete successfully.

      Actual Results:

      Pull image with podman from quay was failed with 400 error code.

        1. quay_app_pod_350.logs
          197 kB

              tomckay@redhat.com Thomas Mckay (Inactive)
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: