-
Bug
-
Resolution: Done
-
Critical
-
None
-
quay-v3.12.0
-
None
-
False
-
None
-
False
-
-
Description:
On Hitachi HCP V9.7 with S3Storage engineer, enable flag FEATURE_PROXY_STORAGE: true, pull image/clair scan failed
Quay: 3.12
Pull image was failed with 403 error code:
$ podman pull quayregistry-quay-quay-enterprise-14839.apps.quaytest-14839.qe.devcluster.openshift.com/user1org/user1repo:zot-linux-amd64 --tls-verify=false Trying to pull quayregistry-quay-quay-enterprise-14839.apps.quaytest-14839.qe.devcluster.openshift.com/user1org/user1repo:zot-linux-amd64... Error: parsing image configuration: fetching blob: StatusCode: 403, <?xml version='1.0' encoding='UTF-8'?> <Error> <...
Quay config.yaml:
FEATURE_PROXY_STORAGE: true
DISTRIBUTED_STORAGE_CONFIG:
local_us:
- S3Storage
- s3_access_key: xxx
s3_bucket: redhat
host: partner.hcpdemo.hitachivantara.com
s3_secret_key: xxx
storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
- local_us
DISTRIBUTED_STORAGE_PREFERENCE:
- local_us
Quay Logs:
nginx stdout | 10.129.2.18 (-) - - [29/Jul/2024:06:10:14 +0000] "GET /_storage_proxy/ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklsODNTVlpVTTBNMVNreDFiRTVsZDFWS1lqQnZYM2t5UVZkbmRXRkZWRU5hZEZsaWNFaFBZVlp6WkRnaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVYSmxaMmx6ZEhKNUxYRjFZWGt0Y1hWaGVTMWxiblJsY25CeWFYTmxMVEUwT0RRd0xtRndjSE11Y1hWaGVYUmxjM1F0TVRRNE5EQXVjV1V1WkdWMlkyeDFjM1JsY2k1dmNHVnVjMmhwWm5RdVkyOXRJaXdpYm1KbUlqb3hOekl5TWpNek5ERTBMQ0pwWVhRaU9qRTNNakl5TXpNME1UUXNJbVY0Y0NJNk1UY3lNakl6TXpRME5Dd2ljM1ZpSWpvaWMzUnZjbUZuWlhCeWIzaDVJaXdpWVdOalpYTnpJanBiZXlKMGVYQmxJam9pYzNSdmNtRm5aWEJ5YjNoNUlpd2lkWEpwSWpvaWNtVmthR0YwTDJSaGRHRnpkRzl5WVdkbEwzSmxaMmx6ZEhKNUwzTm9ZVEkxTmk5aE15OWhNMlZrT1RWallXVmlNREptWm1VMk9HTmtaRGxtWkRnME5EQTJOamd3WVdVNU0yUTJNek5qWWpFMk5ESXlaREF3WlRoaE4yTXlNamsxTldJME5tUTBQMWd0UVcxNkxVRnNaMjl5YVhSb2JUMUJWMU0wTFVoTlFVTXRVMGhCTWpVMkpsZ3RRVzE2TFVOeVpXUmxiblJwWVd3OVkyMVdhMkZIUmpBbE1rWXlNREkwTURjeU9TVXlSblZ6TFdWaGMzUXRNU1V5Um5NekpUSkdZWGR6TkY5eVpYRjFaWE4wSmxndFFXMTZMVVJoZEdVOU1qQXlOREEzTWpsVU1EWXhNREUwV2laWUxVRnRlaTFGZUhCcGNtVnpQVFl3TUNaWUxVRnRlaTFUYVdkdVpXUklaV0ZrWlhKelBXaHZjM1FtV0MxQmJYb3RVMmxuYm1GMGRYSmxQV1E0T1dJd1pXTTVZV0psTm1JNE1XUTNaRGxqTURObU1qWXpZV0poTkRJeU1tVmhaV1JrT0Rsa1lURTBaakF3TTJVeU1qQTFZVEkwTURBMFltVXhOVE1pTENKb2IzTjBJam9pY0dGeWRHNWxjaTVvWTNCa1pXMXZMbWhwZEdGamFHbDJZVzUwWVhKaExtTnZiU0lzSW5OamFHVnRaU0k2SW1oMGRIQnpJbjFkTENKamIyNTBaWGgwSWpwN2ZYMC5vbEZOa1ZMM1lEaWNlU2ZCb2hCemp0RGlNaUVkZE84MVliM095UFNrWklUNHYzR3dISG5qU3k4UThjWEh6YjlmaDcwNU91Mk81azg4M2RZNXVHT1FucGNMd0E3SDZXVTZYVU5md2JVNWUyWlJZTmFnVXp3ZmpkZkpSbFlraTR6ZnRSQXdoNklEc2V6VjV1bEVaVkhpUFZWaHdfNzNzYkpXRzNnWFk1b0kxMDFzOWJ6N2ZCSFIwS3F6MGlLamFLWTBvbzZTM3NlMkZmT3lLREtxdFZPeWMzZXlwdmJkS2dXTllzNHlmWnptaE10cnZkQ1dZWTZ0VEx5MU9GNVEwRkpWbW9IcGdKajZXOF9vLW1zZ3g5WUZ1d2Z2SW5jRGRLZGVvcGJtOFdMNjZSZUZ6ZENCb2dNWVFBbnIwVzhKU2l3d3FodUthZXp1aEc3c2ktQkk1Z21sRUE=/https/partner.hcpdemo.hitachivantara.com/redhat/datastorage/registry/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cmVkaGF0%2F20240729%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240729T061014Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=d89b0ec9abe6b81d7d9c03f263aba4222eaedd89da14f003e2205a24004be153 HTTP/1.1" 403 230 "-" "Go-http-client/1.1" (0.243 2694 0.127)
Clair log:
{"level":"warn","request_id":"1031f208a49c4c44","component":"indexer/controller/Controller.Index","manifest":"sha256:cfdb19c8fdc4b5ff3f5ed39b6958b1ac6aab0d41a39b00b15eafa9f3dab4e405","state":"FetchLayers","error":"fetcher: encountered errors: error realizing layer sha256:f93b7d3396c22354724b502c69276a730003297e4c24bd6d03fc16b0e50090d3: fetcher: unexpected status code: 403 Forbidden (body starts: \"<?xml version='1.0' encoding='UTF-8'?>\\n<Error>\\n <Code>AccessDenied</Code>\\n <Message>Access Denied</Message>\\n <RequestId>1722233414630</RequestId>\\n <HostId>aGNwZGVtby5oaXRhY2hpdmFudGFyYS5jb206MTIw</HostId>\\n</Error>\\n\\n\")","time":"2024-07-29T06:10:14Z","message":"layers fetch failure"}
Note:
With FEATURE_PROXY_STORAGE: true, both S3Storage and RADOS storage engine, pull/scan will be fail, RADOS storage engine error log is attached in PROJQUAY-7535 which is same to known issue PROJQUAY-7491
- duplicates
-
PROJQUAY-7491 presigned S3 request computed by Quay using signature-version v2
- Closed
- relates to
-
PROJQUAY-7491 presigned S3 request computed by Quay using signature-version v2
- Closed
-
PROJQUAY-7561 Quay 3.12 pull image was failed with Hitachi HCP CS Storage
- Closed