Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-7562

pull image failed on Hitachi HCP Storage with FEATURE_PROXY_CACHE=true

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • quay-v3.12.0
    • quay
    • None
    • False
    • None
    • False

      Description:

      On Hitachi HCP V9.7 with S3Storage engineer, enable flag FEATURE_PROXY_STORAGE: true, pull image/clair scan failed

      Quay: 3.12

      Pull image was failed with 403 error code:

      $ podman pull quayregistry-quay-quay-enterprise-14839.apps.quaytest-14839.qe.devcluster.openshift.com/user1org/user1repo:zot-linux-amd64 --tls-verify=false
      Trying to pull quayregistry-quay-quay-enterprise-14839.apps.quaytest-14839.qe.devcluster.openshift.com/user1org/user1repo:zot-linux-amd64...
      Error: parsing image configuration: fetching blob: StatusCode: 403, <?xml version='1.0' encoding='UTF-8'?>
      <Error>
        <...
      

      Quay config.yaml:

      FEATURE_PROXY_STORAGE: true
      DISTRIBUTED_STORAGE_CONFIG:
        local_us:
        - S3Storage
        - s3_access_key: xxx
          s3_bucket: redhat
          host: partner.hcpdemo.hitachivantara.com
          s3_secret_key: xxx
          storage_path: /datastorage/registry
      DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
      - local_us
      DISTRIBUTED_STORAGE_PREFERENCE:
      - local_us

      Quay Logs:

      nginx stdout | 10.129.2.18 (-) - - [29/Jul/2024:06:10:14 +0000] "GET /_storage_proxy/ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklsODNTVlpVTTBNMVNreDFiRTVsZDFWS1lqQnZYM2t5UVZkbmRXRkZWRU5hZEZsaWNFaFBZVlp6WkRnaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVYSmxaMmx6ZEhKNUxYRjFZWGt0Y1hWaGVTMWxiblJsY25CeWFYTmxMVEUwT0RRd0xtRndjSE11Y1hWaGVYUmxjM1F0TVRRNE5EQXVjV1V1WkdWMlkyeDFjM1JsY2k1dmNHVnVjMmhwWm5RdVkyOXRJaXdpYm1KbUlqb3hOekl5TWpNek5ERTBMQ0pwWVhRaU9qRTNNakl5TXpNME1UUXNJbVY0Y0NJNk1UY3lNakl6TXpRME5Dd2ljM1ZpSWpvaWMzUnZjbUZuWlhCeWIzaDVJaXdpWVdOalpYTnpJanBiZXlKMGVYQmxJam9pYzNSdmNtRm5aWEJ5YjNoNUlpd2lkWEpwSWpvaWNtVmthR0YwTDJSaGRHRnpkRzl5WVdkbEwzSmxaMmx6ZEhKNUwzTm9ZVEkxTmk5aE15OWhNMlZrT1RWallXVmlNREptWm1VMk9HTmtaRGxtWkRnME5EQTJOamd3WVdVNU0yUTJNek5qWWpFMk5ESXlaREF3WlRoaE4yTXlNamsxTldJME5tUTBQMWd0UVcxNkxVRnNaMjl5YVhSb2JUMUJWMU0wTFVoTlFVTXRVMGhCTWpVMkpsZ3RRVzE2TFVOeVpXUmxiblJwWVd3OVkyMVdhMkZIUmpBbE1rWXlNREkwTURjeU9TVXlSblZ6TFdWaGMzUXRNU1V5Um5NekpUSkdZWGR6TkY5eVpYRjFaWE4wSmxndFFXMTZMVVJoZEdVOU1qQXlOREEzTWpsVU1EWXhNREUwV2laWUxVRnRlaTFGZUhCcGNtVnpQVFl3TUNaWUxVRnRlaTFUYVdkdVpXUklaV0ZrWlhKelBXaHZjM1FtV0MxQmJYb3RVMmxuYm1GMGRYSmxQV1E0T1dJd1pXTTVZV0psTm1JNE1XUTNaRGxqTURObU1qWXpZV0poTkRJeU1tVmhaV1JrT0Rsa1lURTBaakF3TTJVeU1qQTFZVEkwTURBMFltVXhOVE1pTENKb2IzTjBJam9pY0dGeWRHNWxjaTVvWTNCa1pXMXZMbWhwZEdGamFHbDJZVzUwWVhKaExtTnZiU0lzSW5OamFHVnRaU0k2SW1oMGRIQnpJbjFkTENKamIyNTBaWGgwSWpwN2ZYMC5vbEZOa1ZMM1lEaWNlU2ZCb2hCemp0RGlNaUVkZE84MVliM095UFNrWklUNHYzR3dISG5qU3k4UThjWEh6YjlmaDcwNU91Mk81azg4M2RZNXVHT1FucGNMd0E3SDZXVTZYVU5md2JVNWUyWlJZTmFnVXp3ZmpkZkpSbFlraTR6ZnRSQXdoNklEc2V6VjV1bEVaVkhpUFZWaHdfNzNzYkpXRzNnWFk1b0kxMDFzOWJ6N2ZCSFIwS3F6MGlLamFLWTBvbzZTM3NlMkZmT3lLREtxdFZPeWMzZXlwdmJkS2dXTllzNHlmWnptaE10cnZkQ1dZWTZ0VEx5MU9GNVEwRkpWbW9IcGdKajZXOF9vLW1zZ3g5WUZ1d2Z2SW5jRGRLZGVvcGJtOFdMNjZSZUZ6ZENCb2dNWVFBbnIwVzhKU2l3d3FodUthZXp1aEc3c2ktQkk1Z21sRUE=/https/partner.hcpdemo.hitachivantara.com/redhat/datastorage/registry/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cmVkaGF0%2F20240729%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240729T061014Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=d89b0ec9abe6b81d7d9c03f263aba4222eaedd89da14f003e2205a24004be153 HTTP/1.1" 403 230 "-" "Go-http-client/1.1" (0.243 2694 0.127)

      Clair log:

      {"level":"warn","request_id":"1031f208a49c4c44","component":"indexer/controller/Controller.Index","manifest":"sha256:cfdb19c8fdc4b5ff3f5ed39b6958b1ac6aab0d41a39b00b15eafa9f3dab4e405","state":"FetchLayers","error":"fetcher: encountered errors: error realizing layer sha256:f93b7d3396c22354724b502c69276a730003297e4c24bd6d03fc16b0e50090d3: fetcher: unexpected status code: 403 Forbidden (body starts: \"<?xml version='1.0' encoding='UTF-8'?>\\n<Error>\\n  <Code>AccessDenied</Code>\\n  <Message>Access Denied</Message>\\n  <RequestId>1722233414630</RequestId>\\n  <HostId>aGNwZGVtby5oaXRhY2hpdmFudGFyYS5jb206MTIw</HostId>\\n</Error>\\n\\n\")","time":"2024-07-29T06:10:14Z","message":"layers fetch failure"} 

       

      Note:

      With FEATURE_PROXY_STORAGE: true, both S3Storage and RADOS storage engine, pull/scan will be fail, RADOS storage engine error log is attached in PROJQUAY-7535  which is same to known issue PROJQUAY-7491

       

        1. quaypod2.log
          3.74 MB
        2. quaypod1.log
          8.12 MB
        3. clair.log
          1.53 MB
        4. 312_1_quaypod2.log
          18.75 MB
        5. 312_1_quaypod1.log
          6.10 MB
        6. 312_1_clairpod.log
          2.68 MB

              Unassigned Unassigned
              szhao@redhat.com Sean Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: