Details
-
Bug
-
Resolution: Done
-
Critical
-
quay-v3.5.0
Description
Description:
This is an issue found when use podman to pull image from quay 3.5, after push image to Quay 3.5 registry, to use podman to pull image, hit error "Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)", see quay app pod logs attached.
Note: This quay was deployed on OCP with FIPS enabled.
podman --version podman version 2.0.5 podman login --tls-verify=false quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com Username: quay Password: Login Succeeded! podman pull --tls-verify=false quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo:latest Trying to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo:latest... Error fetching blob: invalid status code from registry 400 (Bad Request) Error: unable to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo:latest: Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request) podman pull --tls-verify=false quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo@sha256:926270b9ef2150159dbd691a76bb11960934bf4a15b69899fc13f8e058d511e7 Trying to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo@sha256:926270b9ef2150159dbd691a76bb11960934bf4a15b69899fc13f8e058d511e7... Error fetching blob: invalid status code from registry 400 (Bad Request) Error: unable to pull quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/qeteam/quaydemo@sha256:926270b9ef2150159dbd691a76bb11960934bf4a15b69899fc13f8e058d511e7: Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)
Quay Version:
oc get pod NAME READY STATUS RESTARTS AGE quay-operator.v3.5.0-7489b8c4f-r4r6j 1/1 Running 0 47m quayregistry-clair-app-77c4cb85bc-9frx4 1/1 Running 0 29m quayregistry-clair-postgres-6866688489-jt78f 1/1 Running 1 38m quayregistry-quay-app-8c9f7894-chn94 1/1 Running 0 92s quayregistry-quay-config-editor-86cfb5ff5b-4h852 1/1 Running 0 29m quayregistry-quay-database-5659885fdc-l5k2q 1/1 Running 0 29m quayregistry-quay-mirror-86864576f8-pt25x 1/1 Running 0 29m quayregistry-quay-postgres-init-2ndq5 0/1 Completed 0 29m quayregistry-quay-redis-76d5659fd8-smjtp 1/1 Running 0 38m oc get pod quayregistry-quay-app-8c9f7894-chn94 -o json | jq '.spec.containers[0].image' "registry.redhat.io/quay/quay-rhel8@sha256:0444c7b452a14e0c87ee56f9aa72c54484333c38b0a95de9a4f11f6177273f26"
Quay App Pod logs:
gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=HeadBucket) with params: {'url_path': '/quayaws141218', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.17.21 Python/3.8.3 Linux/4.18.0-240.15.1.el8_3.x86_64 Botocore/1.20.21'}, 'body': b'', 'url': 'https://s3.us-east-2.amazonaws.com/quayaws141218', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f10678b4130>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'quayaws141218'}}} gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f1067961ee0>> gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f1069057f70>> gunicorn-registry stdout | 2021-03-29 02:15:34,089 [251] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler <function set_operation_specific_signer at 0x7f106b16c550> gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.hooks] Event before-sign.s3.HeadBucket: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f10678b4d60>> gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.utils] Using S3 path style addressing. gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] Calculating signature using v4 auth. gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] CanonicalRequest: gunicorn-registry stdout | HEAD gunicorn-registry stdout | /quayaws141218 gunicorn-registry stdout | host:s3.us-east-2.amazonaws.com gunicorn-registry stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gunicorn-registry stdout | x-amz-date:20210329T021534Z gunicorn-registry stdout | host;x-amz-content-sha256;x-amz-date gunicorn-registry stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] StringToSign: gunicorn-registry stdout | AWS4-HMAC-SHA256 gunicorn-registry stdout | 20210329T021534Z gunicorn-registry stdout | 20210329/us-east-1/s3/aws4_request gunicorn-registry stdout | 1908611727337defb71bd7ae158ff8c2388cf2713b25e4d686e2bebcd954d140 gunicorn-registry stdout | 2021-03-29 02:15:34,090 [251] [DEBUG] [botocore.auth] Signature: gunicorn-registry stdout | 8c99622d9ebd4d17b70410c1f6015fcfa04eb7049ac925e19ab5d814f60f8f24 gunicorn-registry stdout | 2021-03-29 02:15:34,091 [251] [DEBUG] [botocore.endpoint] Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://s3.us-east-2.amazonaws.com/quayaws141218, headers={'User-Agent': b'Boto3/1.17.21 Python/3.8.3 Linux/4.18.0-240.15.1.el8_3.x86_64 Botocore/1.20.21', 'X-Amz-Date': b'20210329T021534Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAUMQAHCJON275SXFZ/20210329/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8c99622d9ebd4d17b70410c1f6015fcfa04eb7049ac925e19ab5d814f60f8f24'}> gunicorn-registry stdout | 2021-03-29 02:15:34,091 [251] [DEBUG] [botocore.httpsession] Certificate path: /usr/local/lib/python3.8/site-packages/certifi/cacert.pem gunicorn-registry stdout | 2021-03-29 02:15:34,092 [251] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): s3.us-east-2.amazonaws.com:443 gunicorn-registry stdout | 2021-03-29 02:15:34,172 [251] [DEBUG] [urllib3.connectionpool] https://s3.us-east-2.amazonaws.com:443 "HEAD /quayaws141218 HTTP/1.1" 400 0
Steps:
- Deploy Quay 3.5.0 Operator to all OCP namespace
- Deploy quay with quay 3.5 Operator with using AWS S3 as backend registry storage
- Create new Quay Org and image repository
- Push image to new image repository
- Pull image from quay with podman
Expected Results:
Pull image with podman from quay complete successfully.
Actual Results:
Pull image with podman from quay was failed with 400 error code.
