Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-8591

pull-through proxy fails to serve from cache while upstream connection is severed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • quay-v3.13.3
    • registry-proxy
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      1. configure Quay as proxy cache with 2 organizations:
          - registry.redhat.io with credentials
          - quay.io without credentials

       

      $ sudo podman exec -it postgresql-quay psql quay -c "select id, upstream_registry, upstream_registry_username, organization_id from proxycacheconfig;"  
       id | upstream_registry  |                                        upstream_registry_username                                        | organization_id 
      ----+--------------------+----------------------------------------------------------------------------------------------------------+-----------------
       39 | registry.redhat.io | v0$$<snip>                                                                                               |               8
       40 | quay.io            |                                                                                                          |               7
      (2 rows)

      2.  pull an image requiring authentication from the proxy:

      $ podman pull --tls-verify=false quay.example.net/redhat/openshift4/network-tools-rhel8:latest
      Trying to pull quay.example.net/redhat/openshift4/network-tools-rhel8:latest...
      Getting image source signatures
      <...>
      Copying config 42d87b0879 done   | 
      Writing manifest to image destination
      42d87b0879d77613aa565c798738b12ce6b4e5473e83e2d6991d198c9a277b17

      3. disconnect the connection to the upstream registry, remove the image locally and retry pulling from the proxy:

      $ sudo podman exec quay curl -vv https://registry.redhat.io/v2/
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
        0     0    0     0    0     0      0      0 --:--:--  0:00:27 --:--:--     0* Could not resolve host: registry.redhat.io
      * Closing connection 0
      curl: (6) Could not resolve host: registry.redhat.io
      
      $ podman rmi quay.example.net/redhat/openshift4/network-tools-rhel8:latest
      Untagged: quay.example.net/redhat/openshift4/network-tools-rhel8:latest
      Deleted: 42d87b0879d77613aa565c798738b12ce6b4e5473e83e2d6991d198c9a277b17
      
      $ podman pull --tls-verify=false quay.example.net/redhat/openshift4/network-tools-rhel8:latest
      Trying to pull quay.example.net/redhat/openshift4/network-tools-rhel8:latest...
      WARN[0028] Failed, retrying in 1s ... (1/3). Error: initializing source docker://quay.example.net/redhat/openshift4/network-tools-rhel8:latest: reading manifest latest in quay.example.net/redhat/openshift4/network-tools-rhel8: unknown: the requested image may not exist in the upstream registry, or the configured Quay organization credentials have insufficient rights to access it (HTTPSConnectionPool(host='registry.redhat.io', port=443): Max retries exceeded with url: /v2/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa5e2c82280>: Failed to establish a new connection: [Errno -2] Name or service not known'))) 
      WARN[0057] Failed, retrying in 1s ... (2/3). Error: initializing source docker://quay.example.net/redhat/openshift4/network-tools-rhel8:latest: reading manifest latest in quay.example.net/redhat/openshift4/network-tools-rhel8: unknown: the requested image may not exist in the upstream registry, or the configured Quay organization credentials have insufficient rights to access it (HTTPSConnectionPool(host='registry.redhat.io', port=443): Max retries exceeded with url: /v2/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa5e2c45760>: Failed to establish a new connection: [Errno -2] Name or service not known'))) 
      WARN[0087] Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.example.net/redhat/openshift4/network-tools-rhel8:latest: reading manifest latest in quay.example.net/redhat/openshift4/network-tools-rhel8: unknown: the requested image may not exist in the upstream registry, or the configured Quay organization credentials have insufficient rights to access it (HTTPSConnectionPool(host='registry.redhat.io', port=443): Max retries exceeded with url: /v2/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa5e2c881c0>: Failed to establish a new connection: [Errno -2] Name or service not known'))) 
      Error: initializing source docker://quay.example.net/redhat/openshift4/network-tools-rhel8:latest: reading manifest latest in quay.example.net/redhat/openshift4/network-tools-rhel8: unknown: the requested image may not exist in the upstream registry, or the configured Quay organization credentials have insufficient rights to access it (HTTPSConnectionPool(host='registry.redhat.io', port=443): Max retries exceeded with url: /v2/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa5e2c879d0>: Failed to establish a new connection: [Errno -2] Name or service not known')))
      

       

      Note: this image requires authentication:

       

      $ podman pull registry.redhat.io/openshift4/network-tools-rhel8:latest
      Trying to pull registry.redhat.io/openshift4/network-tools-rhel8:latest...
      Error: initializing source docker://registry.redhat.io/openshift4/network-tools-rhel8:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication

       

      In comparison, for images not requiring authentication, this works:

      1. restore the connection upstream:

      $ sudo podman exec quay curl -sI https://registry.redhat.io/v2/
      HTTP/2 401 
      date: Wed, 19 Feb 2025 12:31:54 GMT
      content-type: application/json
      content-length: 99
      docker-distribution-api-version: registry/2.0
      registry-proxy-request-id: a8f569af-aeaf-4324-a631-875526ae75ef
      www-authenticate: Bearer realm="https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth",service="docker-registry" 

      2. pull an image from the proxy registry:

      $ podman pull --tls-verify=false docker://quay.example.net/quay/openshift/origin-cli:4.14
      Trying to pull quay.example.net/quay/openshift/origin-cli:4.14...
      Getting image source signatures
      Copying blob 4954dd224f02 skipped: already exists  
      Copying blob d8190195889e skipped: already exists  
      Copying blob cc5ef9451a35 skipped: already exists  
      Copying blob c0497b5720d9 skipped: already exists  
      Copying blob bf94af788703 skipped: already exists  
      Copying blob 97da74cc6d8f skipped: already exists  
      Copying config cd6039be3c done   | 
      Writing manifest to image destination
      cd6039be3c7ad49760aad0d5d4d6cfa494152ea77e2dac4d9a1c7fceb4589571 

      3. disconnect from upstream, remove local image and pull again from proxy:

      $ sudo podman exec quay curl -vv https://quay.io/v2/
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
        0     0    0     0    0     0      0      0 --:--:--  0:00:27 --:--:--     0* Could not resolve host: quay.io
      * Closing connection 0
      curl: (6) Could not resolve host: quay.io
      
      $ podman rmi quay.example.net/quay/openshift/origin-cli:4.14
      Untagged: quay.example.net/quay/openshift/origin-cli:4.14
      Deleted: cd6039be3c7ad49760aad0d5d4d6cfa494152ea77e2dac4d9a1c7fceb4589571
      
      $ podman pull --tls-verify=false docker://quay.example.net/quay/openshift/origin-cli:4.14
      Trying to pull quay.example.net/quay/openshift/origin-cli:4.14...
      Getting image source signatures
      Copying blob 4954dd224f02 skipped: already exists  
      Copying blob d8190195889e skipped: already exists  
      Copying blob cc5ef9451a35 skipped: already exists  
      Copying blob c0497b5720d9 skipped: already exists  
      Copying blob bf94af788703 skipped: already exists  
      Copying blob 97da74cc6d8f skipped: already exists  
      Copying config cd6039be3c done   | 
      Writing manifest to image destination
      cd6039be3c7ad49760aad0d5d4d6cfa494152ea77e2dac4d9a1c7fceb4589571

      The connection upstream is unavailable, so the image must have been served from cache.

       

      Show
      1. configure Quay as proxy cache with 2 organizations:     - registry.redhat.io with credentials     - quay.io without credentials   $ sudo podman exec -it postgresql-quay psql quay -c "select id, upstream_registry, upstream_registry_username, organization_id from proxycacheconfig;"    id | upstream_registry  |                                        upstream_registry_username                                        | organization_id  ----+--------------------+----------------------------------------------------------------------------------------------------------+-----------------  39 | registry.redhat.io | v0$$<snip>                                                                                               |               8  40 | quay.io            |                                                                                                          |               7 (2 rows) 2.  pull an image requiring authentication from the proxy: $ podman pull --tls-verify= false quay.example.net/redhat/openshift4/network-tools-rhel8:latest Trying to pull quay.example.net/redhat/openshift4/network-tools-rhel8:latest... Getting image source signatures <...> Copying config 42d87b0879 done   |  Writing manifest to image destination 42d87b0879d77613aa565c798738b12ce6b4e5473e83e2d6991d198c9a277b17 3. disconnect the connection to the upstream registry, remove the image locally and retry pulling from the proxy: $ sudo podman exec quay curl -vv https: //registry.redhat.io/v2/   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed   0     0    0     0    0     0      0      0 --:--:--  0:00:27 --:--:--     0* Could not resolve host: registry.redhat.io * Closing connection 0 curl: (6) Could not resolve host: registry.redhat.io $ podman rmi quay.example.net/redhat/openshift4/network-tools-rhel8:latest Untagged: quay.example.net/redhat/openshift4/network-tools-rhel8:latest Deleted: 42d87b0879d77613aa565c798738b12ce6b4e5473e83e2d6991d198c9a277b17 $ podman pull --tls-verify= false quay.example.net/redhat/openshift4/network-tools-rhel8:latest Trying to pull quay.example.net/redhat/openshift4/network-tools-rhel8:latest... WARN[0028] Failed, retrying in 1s ... (1/3). Error: initializing source docker: //quay.example.net/redhat/openshift4/network-tools-rhel8:latest: reading manifest latest in quay.example.net/redhat/openshift4/network-tools-rhel8: unknown: the requested image may not exist in the upstream registry, or the configured Quay organization credentials have insufficient rights to access it (HTTPSConnectionPool(host= 'registry.redhat.io' , port=443): Max retries exceeded with url: /v2/ (Caused by NewConnectionError( '<urllib3.connection.HTTPSConnection object at 0x7fa5e2c82280>: Failed to establish a new connection: [Errno -2] Name or service not known' )))  WARN[0057] Failed, retrying in 1s ... (2/3). Error: initializing source docker: //quay.example.net/redhat/openshift4/network-tools-rhel8:latest: reading manifest latest in quay.example.net/redhat/openshift4/network-tools-rhel8: unknown: the requested image may not exist in the upstream registry, or the configured Quay organization credentials have insufficient rights to access it (HTTPSConnectionPool(host= 'registry.redhat.io' , port=443): Max retries exceeded with url: /v2/ (Caused by NewConnectionError( '<urllib3.connection.HTTPSConnection object at 0x7fa5e2c45760>: Failed to establish a new connection: [Errno -2] Name or service not known' )))  WARN[0087] Failed, retrying in 1s ... (3/3). Error: initializing source docker: //quay.example.net/redhat/openshift4/network-tools-rhel8:latest: reading manifest latest in quay.example.net/redhat/openshift4/network-tools-rhel8: unknown: the requested image may not exist in the upstream registry, or the configured Quay organization credentials have insufficient rights to access it (HTTPSConnectionPool(host= 'registry.redhat.io' , port=443): Max retries exceeded with url: /v2/ (Caused by NewConnectionError( '<urllib3.connection.HTTPSConnection object at 0x7fa5e2c881c0>: Failed to establish a new connection: [Errno -2] Name or service not known' )))  Error: initializing source docker: //quay.example.net/redhat/openshift4/network-tools-rhel8:latest: reading manifest latest in quay.example.net/redhat/openshift4/network-tools-rhel8: unknown: the requested image may not exist in the upstream registry, or the configured Quay organization credentials have insufficient rights to access it (HTTPSConnectionPool(host= 'registry.redhat.io' , port=443): Max retries exceeded with url: /v2/ (Caused by NewConnectionError( '<urllib3.connection.HTTPSConnection object at 0x7fa5e2c879d0>: Failed to establish a new connection: [Errno -2] Name or service not known' )))   Note: this image requires authentication:   $ podman pull registry.redhat.io/openshift4/network-tools-rhel8:latest Trying to pull registry.redhat.io/openshift4/network-tools-rhel8:latest... Error: initializing source docker: //registry.redhat.io/openshift4/network-tools-rhel8:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication   In comparison, for images not requiring authentication, this works: 1. restore the connection upstream: $ sudo podman exec quay curl -sI https: //registry.redhat.io/v2/ HTTP/2 401  date: Wed, 19 Feb 2025 12:31:54 GMT content-type: application/json content-length: 99 docker-distribution-api-version: registry/2.0 registry-proxy-request-id: a8f569af-aeaf-4324-a631-875526ae75ef www-authenticate: Bearer realm= "https: //registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth" ,service= "docker-registry" 2. pull an image from the proxy registry: $ podman pull --tls-verify= false docker: //quay.example.net/quay/openshift/origin-cli:4.14 Trying to pull quay.example.net/quay/openshift/origin-cli:4.14... Getting image source signatures Copying blob 4954dd224f02 skipped: already exists   Copying blob d8190195889e skipped: already exists   Copying blob cc5ef9451a35 skipped: already exists   Copying blob c0497b5720d9 skipped: already exists   Copying blob bf94af788703 skipped: already exists   Copying blob 97da74cc6d8f skipped: already exists   Copying config cd6039be3c done   |  Writing manifest to image destination cd6039be3c7ad49760aad0d5d4d6cfa494152ea77e2dac4d9a1c7fceb4589571 3. disconnect from upstream, remove local image and pull again from proxy: $ sudo podman exec quay curl -vv https: //quay.io/v2/   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed   0     0    0     0    0     0      0      0 --:--:--  0:00:27 --:--:--     0* Could not resolve host: quay.io * Closing connection 0 curl: (6) Could not resolve host: quay.io $ podman rmi quay.example.net/quay/openshift/origin-cli:4.14 Untagged: quay.example.net/quay/openshift/origin-cli:4.14 Deleted: cd6039be3c7ad49760aad0d5d4d6cfa494152ea77e2dac4d9a1c7fceb4589571 $ podman pull --tls-verify= false docker: //quay.example.net/quay/openshift/origin-cli:4.14 Trying to pull quay.example.net/quay/openshift/origin-cli:4.14... Getting image source signatures Copying blob 4954dd224f02 skipped: already exists   Copying blob d8190195889e skipped: already exists   Copying blob cc5ef9451a35 skipped: already exists   Copying blob c0497b5720d9 skipped: already exists   Copying blob bf94af788703 skipped: already exists   Copying blob 97da74cc6d8f skipped: already exists   Copying config cd6039be3c done   |  Writing manifest to image destination cd6039be3c7ad49760aad0d5d4d6cfa494152ea77e2dac4d9a1c7fceb4589571 The connection upstream is unavailable, so the image must have been served from cache.  

      Pulling images that require authentication fails when using a proxy/cache while the upstream registry is (temporary) unavailable.

      As a user, I'd expect a cache image to be served if the upstream registry is unavailable regardless of whether the image requires authentication.

              Unassigned Unassigned
              rhn-support-bverschu Bram Verschueren
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: