Improving FEATURE_PROXY_STORAGE to remove failures when having complex infrastructure configuration

The FEATURE_PROXY_STORAGE is utilized most with an Operator deployment within OpenShift. It is considered stable and working within such a Cluster. Various Enterprise customers running Quay in Standalone and or HA mode are required to use the Feature due to rules and regulations preventing direct access to the Storage backend.

The current implementation is utilizing the nginx reverse proxy Feature to do that and provide a unique interface for the registry as well as for accessing blob storage/image layers.

Verifying and trying to reproduce various Scenarios resulting in Errors for customers showed that adding any layer in between the Quay instance using nginx reverse proxy and either the storage or Clair creates issues due to not taking all possible proxy_pass configurations in consideration.

Our current configuration defined for accessing storage blobs

location ~ ^/_storage_proxy/([^/]+)/([^/]+)/([^/]+)/(.+) {
    include resolver.conf;    
    auth_request /_storage_proxy_auth; 
    proxy_pass $2://$3/$4$is_args$args;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $3;
    proxy_set_header Authorization "";
    add_header Host $3;
    proxy_buffering off;
    proxy_request_buffering off;
    proxy_read_timeout 60s;
}

this leaves out configurations where the Storage backend for example is reachable through another proxy or Load Balancing mechanism.

Show casing the scenario by deploying Quay through Operator in a cluster with OpenShift ServiceMesh enabled identified that following configuration options should be added to guarantee proxy mode working in any constellation.

        proxy_http_version 1.1;
        proxy_ssl_name $3;
        proxy_ssl_server_name on;

The options are expected to handle following scenarios:

proxy_http_version 1.1
- will grant upgrading the default set HTTP/1.0 request done by nginx proxy
- will grant forcing Headers like host (as defined in RFC2616 19.6.1.1)
- will improve handling for Multi-homed destinations and conserve IP Addresses (RFC2616 19.6.1.1)
- utilizing and improvement for compatibility with persistent Connections (RFC2616 19.6.2)
- improvement on responses for 403 and 404 (RFC2616 19.6.3)
- proxies should be able to add Content-Length when appropriate (19.6.3)

A verified working configuration can be seen below

   location ~ ^/_storage_proxy/([^/]+)/([^/]+)/([^/]+)/(.+) {
        include resolver.conf;        
        auth_request /_storage_proxy_auth;        
        proxy_pass $2://$3/$4$is_args$args;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;        
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $3;
        proxy_set_header Authorization "";
        proxy_ssl_name $3;
        proxy_ssl_server_name on;        
        add_header Host $3;        
        proxy_buffering off;
        proxy_request_buffering off;       
        proxy_read_timeout 60s;
    }

Show casing examples

current Quay nginx configuration

client response leading to an error of

Error: parsing image configuration: fetching blob: received unexpected HTTP status: 502 Bad Gateway

gunicorn-web stdout | 2023-02-26 10:12:11,842 [256] [DEBUG] [app] Starting request: urn:request:30c515cb-76da-46e1-96c7-d9c59d24e5f0 (/_storage_proxy_auth) {'X-Forwarded-For': '10.91.0.19'}
gunicorn-web stdout | 2023-02-26 10:12:11,842 [256] [DEBUG] [storage.downloadproxy] Got token b'eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImFzRUdOaWtva3NuV3owQllPdHpLRU9XNWk1NGJGbEtXVDJmaFhZb0hhUW8ifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheS5leGFtcGxlLmNvbSIsIm5iZiI6MTY3NzQwNjMzMSwiaWF0IjoxNjc3NDA2MzMxLCJleHAiOjE2Nzc0MDYzNjEsInN1YiI6InN0b3JhZ2Vwcm94eSIsImFjY2VzcyI6W3sidHlwZSI6InN0b3JhZ2Vwcm94eSIsInVyaSI6InF1YXkvZGF0YXN0b3JhZ2UvcmVnaXN0cnkvc2hhMjU2LzZkLzZkM2ZiZmIzZGE2MGFjZjExMjExY2JjYzE0NmVkYjYwNDA0NDk2ZjU5MDE0ZTQ4MjhhNTM2MmZiYmUwYzA4N2I_QVdTQWNjZXNzS2V5SWQ9bWluaW9hZG1pbiZTaWduYXR1cmU9dmElMkJUSHdka0lhYnNuVXhJTVg0dXR5RGJ2akklM0QmRXhwaXJlcz0xNjc3NDA2OTMxIiwiaG9zdCI6Im1pbmlvLnF1YXkuc3ZjOjkwMDAiLCJzY2hlbWUiOiJodHRwIn1dLCJjb250ZXh0Ijp7fX0.SrgZiOYc9HHdJW4uJ5qBfQ5pP7GOXQQtcRSGZsJXAQHiMN8vYM9r00-YrwI3Uu8OvGm-NWWotTgXUIpqnLps1UY5A0GzttW7CZEJMbZeAVOYNDwlrO38XVBoT5FlhsBP8Fn3IzikcsWNpolbZiO_PQEke_Q3czAu0IvzwTBLXkxG4FtKwcuQdY9Lg584Psd5_UtVnkUVIviOWGdWq5S0MjhAgw-eYb0PnezuGZNUYnK2QV3Xz-sdJlMvwfM51faBv4uirjypyQUHRZ4Z3NIXB9x0k9Gp_RPCVmpbf1k3w0NQdpwPMlEILwdA6UMYoxti4maM5BJec3Yf1A7KtjJsqg' for storage proxy auth request /_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJbUZ6UlVkT2FXdHZhM051VjNvd1FsbFBkSHBMUlU5WE5XazFOR0pHYkV0WFZESm1hRmhaYjBoaFVXOGlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qTXpNU3dpYVdGMElqb3hOamMzTkRBMk16TXhMQ0psZUhBaU9qRTJOemMwTURZek5qRXNJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpaa0x6WmtNMlppWm1JelpHRTJNR0ZqWmpFeE1qRXhZMkpqWXpFME5tVmtZall3TkRBME5EazJaalU1TURFMFpUUTRNamhoTlRNMk1tWmlZbVV3WXpBNE4ySV9RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlkbUVsTWtKVVNIZGthMGxoWW5OdVZYaEpUVmcwZFhSNVJHSjJha2tsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEyT1RNeElpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLlNyZ1ppT1ljOUhIZEpXNHVKNXFCZlE1cFA3R09YUVF0Y1JTR1pzSlhBUUhpTU44dllNOXIwMC1ZcndJM1V1OE92R20tTldXb3RUZ1hVSXBxbkxwczFVWTVBMEd6dHRXN0NaRUpNYlplQVZPWU5Ed2xyTzM4WFZCb1Q1Rmxoc0JQOEZuM0l6aWtjc1dOcG9sYlppT19QUUVrZV9RM2N6QXUwSXZ6d1RCTFhreEc0RnRLd2N1UWRZOUxnNTg0UHNkNV9VdFZua1VWSXZpT1dHZFdxNVMwTWpoQWd3LWVZYjBQbmV6dUdaTlVZbksyUVYzWHotc2RKbE12d2ZNNTFmYUJ2NHVpcmp5cHlRVUhSWjRaM05JWEI5eDBrOUdwX1JQQ1ZtcGJmMWszdzBOUWRwd1BNbEVJTHdkQTZVTVlveHRpNG1hTTVCSmVjM1lmMUE3S3RqSnNxZw==/http/minio.quay.svc:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931 with parts ['ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJbUZ6UlVkT2FXdHZhM051VjNvd1FsbFBkSHBMUlU5WE5XazFOR0pHYkV0WFZESm1hRmhaYjBoaFVXOGlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qTXpNU3dpYVdGMElqb3hOamMzTkRBMk16TXhMQ0psZUhBaU9qRTJOemMwTURZek5qRXNJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpaa0x6WmtNMlppWm1JelpHRTJNR0ZqWmpFeE1qRXhZMkpqWXpFME5tVmtZall3TkRBME5EazJaalU1TURFMFpUUTRNamhoTlRNMk1tWmlZbVV3WXpBNE4ySV9RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlkbUVsTWtKVVNIZGthMGxoWW5OdVZYaEpUVmcwZFhSNVJHSjJha2tsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEyT1RNeElpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLlNyZ1ppT1ljOUhIZEpXNHVKNXFCZlE1cFA3R09YUVF0Y1JTR1pzSlhBUUhpTU44dllNOXIwMC1ZcndJM1V1OE92R20tTldXb3RUZ1hVSXBxbkxwczFVWTVBMEd6dHRXN0NaRUpNYlplQVZPWU5Ed2xyTzM4WFZCb1Q1Rmxoc0JQOEZuM0l6aWtjc1dOcG9sYlppT19QUUVrZV9RM2N6QXUwSXZ6d1RCTFhreEc0RnRLd2N1UWRZOUxnNTg0UHNkNV9VdFZua1VWSXZpT1dHZFdxNVMwTWpoQWd3LWVZYjBQbmV6dUdaTlVZbksyUVYzWHotc2RKbE12d2ZNNTFmYUJ2NHVpcmp5cHlRVUhSWjRaM05JWEI5eDBrOUdwX1JQQ1ZtcGJmMWszdzBOUWRwd1BNbEVJTHdkQTZVTVlveHRpNG1hTTVCSmVjM1lmMUE3S3RqSnNxZw==', 'http', 'minio.quay.svc:9000', 'quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931']
gunicorn-web stdout | 2023-02-26 10:12:11,870 [256] [DEBUG] [app] Ending request: urn:request:30c515cb-76da-46e1-96c7-d9c59d24e5f0 (/_storage_proxy_auth) {'endpoint': '_storage_proxy_auth', 'request_id': 'urn:request:30c515cb-76da-46e1-96c7-d9c59d24e5f0', 'remote_addr': '10.91.0.19', 'http_method': 'GET', 'original_url': 'http://web_app_server/_storage_proxy_auth', 'path': '/_storage_proxy_auth', 'parameters': {}, 'json_body': None, 'confsha': 'f613904b', 'user-agent': 'containers/5.24.0 (github.com/containers/image)'}
nginx stdout | 2023/02/26 10:12:21 [error] 128#0: *22 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.6, server: , request: "GET /_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJbUZ6UlVkT2FXdHZhM051VjNvd1FsbFBkSHBMUlU5WE5XazFOR0pHYkV0WFZESm1hRmhaYjBoaFVXOGlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qTXpNU3dpYVdGMElqb3hOamMzTkRBMk16TXhMQ0psZUhBaU9qRTJOemMwTURZek5qRXNJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpaa0x6WmtNMlppWm1JelpHRTJNR0ZqWmpFeE1qRXhZMkpqWXpFME5tVmtZall3TkRBME5EazJaalU1TURFMFpUUTRNamhoTlRNMk1tWmlZbVV3WXpBNE4ySV9RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlkbUVsTWtKVVNIZGthMGxoWW5OdVZYaEpUVmcwZFhSNVJHSjJha2tsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEyT1RNeElpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLlNyZ1ppT1ljOUhIZEpXNHVKNXFCZlE1cFA3R09YUVF0Y1JTR1pzSlhBUUhpTU44dllNOXIwMC1ZcndJM1V1OE92R20tTldXb3RUZ1hVSXBxbkxwczFVWTVBMEd6dHRXN0NaRUpNYlplQVZPWU5Ed2xyTzM4WFZCb1Q1Rmxoc0JQOEZuM0l6aWtjc1dOcG9sYlppT19QUUVrZV9RM2N6QXUwSXZ6d1RCTFhreEc0RnRLd2N1UWRZOUxnNTg0UHNkNV9VdFZua1VWSXZpT1dHZFdxNVMwTWpoQWd3LWVZYjBQbmV6dUdaTlVZbksyUVYzWHotc2RKbE12d2ZNNTFmYUJ2NHVpcmp5cHlRVUhSWjRaM05JWEI5eDBrOUdwX1JQQ1ZtcGJmMWszdzBOUWRwd1BNbEVJTHdkQTZVTVlveHRpNG1hTTVCSmVjM1lmMUE3S3RqSnNxZw==/http/minio.quay.svc:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931 HTTP/1.1", upstream: "http://10.21.101.166:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931", host: "quay.example.com"

as we can see, the configured Storage backend used from config.yaml gets translated into an IP Address which leads to an error

curl -v 'http://10.21.101.166:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931'
*   Trying 10.21.101.166...
* TCP_NODELAY set
* Connected to 10.21.101.166 (10.21.101.166) port 9000 (#0)
> GET /quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931 HTTP/1.1
> Host: 10.21.101.166:9000
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Sun, 26 Feb 2023 10:16:09 GMT
< server: envoy
< 
* Connection #0 to host 10.21.101.166 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure

where as the same URL with the configured Storage backend without resolving it in curl will succeed

curl 'http://minio.quay.svc:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931'
{"architecture":"amd64","config":{"Hostname":"a5bb82701f3a","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container=docker"],"Cmd":null,"Image":"docker.io/kindest/base:v20221214-df207207","Volumes":null,"WorkingDir":"/","Entrypoint":["/usr/local/bin/entrypoint","/sbin/init"],"OnBuild":null,"Labels":{},"StopSignal":"SIGRTMIN+3"},"container":"a5bb82701f3a1762b1c86540327bd4106e41e6772be0910abdea4b4125afc519","container_config":{"Hostname":"a5bb82701f3a","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container=docker"],"Cmd":["infinity"],"Image":"docker.io/kindest/base:v20221214-df207207","Volumes":null,"WorkingDir":"/","Entrypoint":["sleep"],"OnBuild":null,"Labels":{},"StopSignal":"SIGRTMIN+3"},"created":"2022-12-20T03:36:35.176496673Z","docker_version":"20.10.21","history":[{"created":"2022-12-14T22:29:25.547887218Z","created_by":"COPY / / # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2022-12-14T22:29:25.547887218Z","created_by":"ENV container=docker","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-12-14T22:29:25.547887218Z","created_by":"STOPSIGNAL SIGRTMIN+3","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-12-14T22:29:25.547887218Z","created_by":"ENTRYPOINT [\"/usr/local/bin/entrypoint\" \"/sbin/init\"]","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-12-20T03:36:35.176496673Z","created_by":"infinity"}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:ca9f058da292d3509f171c5b74254099c0a16acf0f61937dd878c3316ffb21bb","sha256:250b0034f84363d4d125cbef60ab56b331afcbe818e5de7358251aac4c788f2a"]}}

enhanced Quay nginx configuration

client retrieves requested images as expected

gunicorn-registry stdout | 2023-02-26 10:20:19,277 [258] [DEBUG] [app] Starting request: urn:request:d63c4001-14d7-465a-8d29-61b0c023187b (/v2/milang/kindest/node/blobs/sha256:7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e) {'X-Forwarded-For': '10.91.0.19, 127.0.0.6'}
gunicorn-registry stdout | 2023-02-26 10:20:19,582 [258] [DEBUG] [storage.downloadproxy] Proxying via URL http://quay.example.com/_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJa3hOTkVvMU4yeHVlR1pUZUhCd1gwbzVhek13U0VSbFJuUXpaRk5XUkRjNVpqRjNPVVpwUlRBelRITWlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qZ3hPU3dpYVdGMElqb3hOamMzTkRBMk9ERTVMQ0psZUhBaU9qRTJOemMwTURZNE5Ea3NJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpka0x6ZGtZV1ZsWkRRek5qVTVNbVUxWmpKaFpEZ3lNRGc1WWpWaVpXRTRaak0yWkRRMk1tRm1NVFUzWXpjeFpXSXpaV1EyT0Raa016QmhOemd3T0RObE5tVV9RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlSR293TkVkYVZFVkVkRkJtV0UxeldFVjNibUZLTkNVeVJrTm1ObEVsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEzTkRFNUlpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLkZLNVBTLXM1cDBJTy0xaW95a1JQbzBXSHJ2ei1wX3Z2WFhCQzZVNzBTTUZ6N3ZqdnN3YjJVTW5XYmFqZ3dCTlNFeXU0RlNiVEswWUVYTmdtMkJVWlJKV0FJd3lFS1c5ZmRzT2pCOHEyQXMyUHpFdGZ2VEI2RG82RDd3NnNSS1hNTElabS1hb3c5bnFVOURuN3pwcGltNVpUdkgydFRydVF3N0ljSFdsSkFuaXByUFlQZ1BTUUhnbG5PeTFqdG9kT0tGOGxudTdvU3VvNDUwR3IxTDU2WW1FZ2QtVWRaLTNJWlZhWWVEODY0YkZLZ0ZlSkZ1WWNkbVN4QWFPTmhyV2QwSUdQUWZBcnFLNllaRUc0VS1kMWx3VVkwRVNKcmxKaTFKTFZHaW1sUVV4U00yY3dfYll5STg4MWhXeVZvQ2FOUFVJcDZnLXBHLWh6Z29tdC1UenVoUQ==/http/minio.quay.svc:9000/quay/datastorage/registry/sha256/7d/7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e?AWSAccessKeyId=minioadmin&Signature=Dj04GZTEDtPfXMsXEwnaJ4%2FCf6Q%3D&Expires=1677407419
gunicorn-registry stdout | 2023-02-26 10:20:19,583 [261] [DEBUG] [storage.downloadproxy] Proxying via URL http://quay.example.com/_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJa3hOTkVvMU4yeHVlR1pUZUhCd1gwbzVhek13U0VSbFJuUXpaRk5XUkRjNVpqRjNPVVpwUlRBelRITWlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qZ3hPU3dpYVdGMElqb3hOamMzTkRBMk9ERTVMQ0psZUhBaU9qRTJOemMwTURZNE5Ea3NJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpJd0x6SXdPV0ppWXpnNE16azRNRFppWTJNMVpESTJOemhsT0RRNFptWXdaakE1TlRkaE5XSTVOREkxWmpKaU1XUmpNR0l6T0dRelptVTBaV0ZrTXpBNFlqa19RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlTR05CU1VJMVRtOVpORzk1SlRKQ1FUQktUSHBFV1hSbFlrMWhPRWtsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEzTkRFNUlpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLmdWZGZPa1loc3RXMzVRcjlxWURIUjQ3c2J2MF9OaDZaRGNReXV4M2hrMUZuekI0SzlLRDZtbk9PYnhFWEFJZ3E5Ny1SSldUb1ppaGN6aXdYeU13aVFudC1TSlBoMkowUHZRb0hqLTBncE1WYi10R3VaY2k5bXF5OTJsWm5wNDJ6cTJtNnhlWG12b3BLUjNubGZhY1ZwZUJJeXB4aTc5YXl4UjVfclMyenEwamRMNUZKd0RpeUc3NFk3ZFFkQTJCakNyRmZGX1ZzYWstbWVMbkQ0LWRZNVVLTzBrVkpqbWVTNjgzb1VqWXZlYlg2UVhmTTRVY0wxMDhJY1dfRWRlX09QQ1pTWDVURm5HUHBFbVNmMDE0S1RBZUhnaFhvdDlCcG9fSjIxUlR3Qk9Jd2dzV1lHckZCRDVoMGFGeEpxX2xMWWoyMWdhem4xSlNtQ054aXZxMjNwZw==/http/minio.quay.svc:9000/quay/datastorage/registry/sha256/20/209bbc8839806bcc5d2678e848ff0f0957a5b9425f2b1dc0b38d3fe4ead308b9?AWSAccessKeyId=minioadmin&Signature=HcAIB5NoY4oy%2BA0JLzDYtebMa8I%3D&Expires=1677407419
gunicorn-registry stdout | 2023-02-26 10:20:19,585 [258] [DEBUG] [app] Ending request: urn:request:d63c4001-14d7-465a-8d29-61b0c023187b (/v2/milang/kindest/node/blobs/sha256:7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e) {'endpoint': 'v2.download_blob', 'request_id': 'urn:request:d63c4001-14d7-465a-8d29-61b0c023187b', 'remote_addr': '127.0.0.6', 'http_method': 'GET', 'original_url': 'https://quay.example.com/v2/milang/kindest/node/blobs/sha256:7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e', 'path': '/v2/milang/kindest/node/blobs/sha256:7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e', 'parameters': {}, 'json_body': None, 'confsha': 'ae84ded7', 'user-agent': 'containers/5.24.0 (github.com/containers/image)'}
gunicorn-registry stdout | 2023-02-26 10:20:19,587 [261] [DEBUG] [app] Ending request: urn:request:06369e8a-bddc-4800-a4d8-78ab55be671c (/v2/milang/kindest/node/blobs/sha256:209bbc8839806bcc5d2678e848ff0f0957a5b9425f2b1dc0b38d3fe4ead308b9) {'endpoint': 'v2.download_blob', 'request_id': 'urn:request:06369e8a-bddc-4800-a4d8-78ab55be671c', 'remote_addr': '127.0.0.6', 'http_method': 'GET', 'original_url': 'https://quay.example.com/v2/milang/kindest/node/blobs/sha256:209bbc8839806bcc5d2678e848ff0f0957a5b9425f2b1dc0b38d3fe4ead308b9', 'path': '/v2/milang/kindest/node/blobs/sha256:209bbc8839806bcc5d2678e848ff0f0957a5b9425f2b1dc0b38d3fe4ead308b9', 'parameters': {}, 'json_body': None, 'confsha': 'ae84ded7', 'user-agent': 'containers/5.24.0 (github.com/containers/image)'}

as we can see, the configured Storage backend used from config.yaml will not be resolved and therefor will succeed with SNI and Vhost matching mechanism used to provide HA or restricted Backend availability.