Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1803

Podman pull image from quay hit 400 error code

XMLWordPrintable

      Description:

      This is an issue found when use podman to pull image from quay 3.5,  after push image to Quay 3.5 registry, to use podman to pull image, hit error "Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)", see quay app pod logs attached.

      Note: This quay was deployed on OCP with FIPS enabled.

      podman --version
      podman version 2.0.5
      
      podman login --tls-verify=false quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com
      Username: quay
      Password: 
      Login Succeeded!
      
      podman pull --tls-verify=false quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo:latest
      Trying to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo:latest...
        Error fetching blob: invalid status code from registry 400 (Bad Request)
      Error: unable to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo:latest: Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)
      
      podman pull --tls-verify=false quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo@sha256:926270b9ef2150159dbd691a76bb11960934bf4a15b69899fc13f8e058d511e7
      Trying to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo@sha256:926270b9ef2150159dbd691a76bb11960934bf4a15b69899fc13f8e058d511e7...
        Error fetching blob: invalid status code from registry 400 (Bad Request)
      Error: unable to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo@sha256:926270b9ef2150159dbd691a76bb11960934bf4a15b69899fc13f8e058d511e7: Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)
      

      Quay Version:

      oc get pod
      NAME                                               READY   STATUS      RESTARTS   AGE
      quay-operator.v3.5.0-7489b8c4f-r4r6j               1/1     Running     0          47m
      quayregistry-clair-app-77c4cb85bc-9frx4            1/1     Running     0          29m
      quayregistry-clair-postgres-6866688489-jt78f       1/1     Running     1          38m
      quayregistry-quay-app-8c9f7894-chn94               1/1     Running     0          92s
      quayregistry-quay-config-editor-86cfb5ff5b-4h852   1/1     Running     0          29m
      quayregistry-quay-database-5659885fdc-l5k2q        1/1     Running     0          29m
      quayregistry-quay-mirror-86864576f8-pt25x          1/1     Running     0          29m
      quayregistry-quay-postgres-init-2ndq5              0/1     Completed   0          29m
      quayregistry-quay-redis-76d5659fd8-smjtp           1/1     Running     0          38m
      
      oc get pod quayregistry-quay-app-8c9f7894-chn94 -o json | jq '.spec.containers[0].image'
      "registry.redhat.io/quay/quay-rhel8@sha256:0444c7b452a14e0c87ee56f9aa72c54484333c38b0a95de9a4f11f6177273f26"
      

      Quay App Pod logs:

      gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=HeadBucket) with params: {'url_path': '/quayaws141218', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.17.21 Python/3.8.3 Linux/4.18.0-240.15.1.el8_3.x86_64 Botocore/1.20.21'}, 'body': b'', 'url': 'https://s3.us-east-2.amazonaws.com/quayaws141218', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f10678b4130>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'quayaws141218'}}}
      gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f1067961ee0>>
      gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f1069057f70>>
      gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler <function set_operation_specific_signer at 0x7f106b16c550>
      gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.hooks] Event before-sign.s3.HeadBucket: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f10678b4d60>>
      gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.utils] Using S3 path style addressing.
      gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] Calculating signature using v4 auth.
      gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] CanonicalRequest:
      gunicorn-registry stdout | HEAD
      gunicorn-registry stdout | /quayaws141218
      gunicorn-registry stdout | host:s3.us-east-2.amazonaws.com
      gunicorn-registry stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      gunicorn-registry stdout | x-amz-date:20210329T021534Z
      gunicorn-registry stdout | host;x-amz-content-sha256;x-amz-date
      gunicorn-registry stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] StringToSign:
      gunicorn-registry stdout | AWS4-HMAC-SHA256
      gunicorn-registry stdout | 20210329T021534Z
      gunicorn-registry stdout | 20210329/us-east-1/s3/aws4_request
      gunicorn-registry stdout | 1908611727337defb71bd7ae158ff8c2388cf2713b25e4d686e2bebcd954d140
      gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] Signature:
      gunicorn-registry stdout | 8c99622d9ebd4d17b70410c1f6015fcfa04eb7049ac925e19ab5d814f60f8f24
      gunicorn-registry stdout | 2021-03-29 02:15:34,091 [251] [DEBUG] [botocore.endpoint] Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://s3.us-east-2.amazonaws.com/quayaws141218, headers={'User-Agent': b'Boto3/1.17.21 Python/3.8.3 Linux/4.18.0-240.15.1.el8_3.x86_64 Botocore/1.20.21', 'X-Amz-Date': b'20210329T021534Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAUMQAHCJON275SXFZ/20210329/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8c99622d9ebd4d17b70410c1f6015fcfa04eb7049ac925e19ab5d814f60f8f24'}>
      gunicorn-registry stdout | 2021-03-29 02:15:34,091 [251] [DEBUG] [botocore.httpsession] Certificate path: /usr/local/lib/python3.8/site-packages/certifi/cacert.pem
      gunicorn-registry stdout | 2021-03-29 02:15:34,092 [251] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): s3.us-east-2.amazonaws.com:443
      gunicorn-registry stdout | 2021-03-29 02:15:34,172 [251] [DEBUG] [urllib3.connectionpool] https://s3.us-east-2.amazonaws.com:443 "HEAD /quayaws141218 HTTP/1.1" 400 0
      

      Steps:

      1. Deploy Quay 3.5.0 Operator to all OCP namespace
      2. Deploy quay with quay 3.5 Operator with using AWS S3 as backend registry storage
      3. Create new Quay Org and image repository
      4. Push image to new image repository
      5. Pull image from quay with podman

      Expected Results:

      Pull image with podman from quay complete successfully.

      Actual Results:

      Pull image with podman from quay was failed with 400 error code.

            tomckay@redhat.com Thomas Mckay (Inactive)
            lzha1981 luffy zhang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: