Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-5140

Improving FEATURE_PROXY_STORAGE to remove failures when having complex infrastructure configuration

XMLWordPrintable

    • Improvement
    • False
    • None
    • False
    • Compatibility/Configuration, User Experience

      Improving FEATURE_PROXY_STORAGE to remove failures when having complex infrastructure configuration

       

      The FEATURE_PROXY_STORAGE is utilized most with an Operator deployment within OpenShift. It is considered stable and working within such a Cluster. Various Enterprise customers running Quay in Standalone and or HA mode are required to use the Feature due to rules and regulations preventing direct access to the Storage backend.

       

      The current implementation is utilizing the nginx reverse proxy Feature to do that and provide a unique interface for the registry as well as for accessing blob storage/image layers.

      Verifying and trying to reproduce various Scenarios resulting in Errors for customers showed that adding any layer in between the Quay instance using nginx reverse proxy and either the storage or Clair creates issues due to not taking all possible proxy_pass configurations in consideration.

      Our current configuration defined for accessing storage blobs

      location ~ ^/_storage_proxy/([^/]+)/([^/]+)/([^/]+)/(.+) {
    include resolver.conf;    
    auth_request /_storage_proxy_auth; 
    proxy_pass $2://$3/$4$is_args$args;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $3;
    proxy_set_header Authorization "";
    add_header Host $3;
    proxy_buffering off;
    proxy_request_buffering off;
    proxy_read_timeout 60s;
}

      this leaves out configurations where the Storage backend for example is reachable through another proxy or Load Balancing mechanism.

      Show casing the scenario by deploying Quay through Operator in a cluster with OpenShift ServiceMesh enabled identified that following configuration options should be added to guarantee proxy mode working in any constellation.

              proxy_http_version 1.1;
        proxy_ssl_name $3;
        proxy_ssl_server_name on;

      The options are expected to handle following scenarios:

      • proxy_http_version 1.1 
        • will grant upgrading the default set HTTP/1.0 request done by nginx proxy 
        • will grant forcing Headers like host (as defined in RFC2616 19.6.1.1)
        • will improve handling for Multi-homed destinations and conserve IP Addresses (RFC2616 19.6.1.1)
        • utilizing and improvement for compatibility with persistent Connections (RFC2616 19.6.2)
        • improvement on responses for 403 and 404 (RFC2616 19.6.3)
        • proxies should be able to add Content-Length when appropriate (19.6.3)

       

      A verified working configuration can be seen below

         location ~ ^/_storage_proxy/([^/]+)/([^/]+)/([^/]+)/(.+) {
        include resolver.conf;        
        auth_request /_storage_proxy_auth;        
        proxy_pass $2://$3/$4$is_args$args;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;        
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $3;
        proxy_set_header Authorization "";
        proxy_ssl_name $3;
        proxy_ssl_server_name on;        
        add_header Host $3;        
        proxy_buffering off;
        proxy_request_buffering off;       
        proxy_read_timeout 60s;
    }

       

      Show casing examples

      current Quay nginx configuration

      client response leading to an error of

       

      Error: parsing image configuration: fetching blob: received unexpected HTTP status: 502 Bad Gateway

       

      gunicorn-web stdout | 2023-02-26 10:12:11,842 [256] [DEBUG] [app] Starting request: urn:request:30c515cb-76da-46e1-96c7-d9c59d24e5f0 (/_storage_proxy_auth) {'X-Forwarded-For': '10.91.0.19'}
gunicorn-web stdout | 2023-02-26 10:12:11,842 [256] [DEBUG] [storage.downloadproxy] Got token b'eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImFzRUdOaWtva3NuV3owQllPdHpLRU9XNWk1NGJGbEtXVDJmaFhZb0hhUW8ifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheS5leGFtcGxlLmNvbSIsIm5iZiI6MTY3NzQwNjMzMSwiaWF0IjoxNjc3NDA2MzMxLCJleHAiOjE2Nzc0MDYzNjEsInN1YiI6InN0b3JhZ2Vwcm94eSIsImFjY2VzcyI6W3sidHlwZSI6InN0b3JhZ2Vwcm94eSIsInVyaSI6InF1YXkvZGF0YXN0b3JhZ2UvcmVnaXN0cnkvc2hhMjU2LzZkLzZkM2ZiZmIzZGE2MGFjZjExMjExY2JjYzE0NmVkYjYwNDA0NDk2ZjU5MDE0ZTQ4MjhhNTM2MmZiYmUwYzA4N2I_QVdTQWNjZXNzS2V5SWQ9bWluaW9hZG1pbiZTaWduYXR1cmU9dmElMkJUSHdka0lhYnNuVXhJTVg0dXR5RGJ2akklM0QmRXhwaXJlcz0xNjc3NDA2OTMxIiwiaG9zdCI6Im1pbmlvLnF1YXkuc3ZjOjkwMDAiLCJzY2hlbWUiOiJodHRwIn1dLCJjb250ZXh0Ijp7fX0.SrgZiOYc9HHdJW4uJ5qBfQ5pP7GOXQQtcRSGZsJXAQHiMN8vYM9r00-YrwI3Uu8OvGm-NWWotTgXUIpqnLps1UY5A0GzttW7CZEJMbZeAVOYNDwlrO38XVBoT5FlhsBP8Fn3IzikcsWNpolbZiO_PQEke_Q3czAu0IvzwTBLXkxG4FtKwcuQdY9Lg584Psd5_UtVnkUVIviOWGdWq5S0MjhAgw-eYb0PnezuGZNUYnK2QV3Xz-sdJlMvwfM51faBv4uirjypyQUHRZ4Z3NIXB9x0k9Gp_RPCVmpbf1k3w0NQdpwPMlEILwdA6UMYoxti4maM5BJec3Yf1A7KtjJsqg' for storage proxy auth request /_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJbUZ6UlVkT2FXdHZhM051VjNvd1FsbFBkSHBMUlU5WE5XazFOR0pHYkV0WFZESm1hRmhaYjBoaFVXOGlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qTXpNU3dpYVdGMElqb3hOamMzTkRBMk16TXhMQ0psZUhBaU9qRTJOemMwTURZek5qRXNJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpaa0x6WmtNMlppWm1JelpHRTJNR0ZqWmpFeE1qRXhZMkpqWXpFME5tVmtZall3TkRBME5EazJaalU1TURFMFpUUTRNamhoTlRNMk1tWmlZbVV3WXpBNE4ySV9RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlkbUVsTWtKVVNIZGthMGxoWW5OdVZYaEpUVmcwZFhSNVJHSjJha2tsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEyT1RNeElpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLlNyZ1ppT1ljOUhIZEpXNHVKNXFCZlE1cFA3R09YUVF0Y1JTR1pzSlhBUUhpTU44dllNOXIwMC1ZcndJM1V1OE92R20tTldXb3RUZ1hVSXBxbkxwczFVWTVBMEd6dHRXN0NaRUpNYlplQVZPWU5Ed2xyTzM4WFZCb1Q1Rmxoc0JQOEZuM0l6aWtjc1dOcG9sYlppT19QUUVrZV9RM2N6QXUwSXZ6d1RCTFhreEc0RnRLd2N1UWRZOUxnNTg0UHNkNV9VdFZua1VWSXZpT1dHZFdxNVMwTWpoQWd3LWVZYjBQbmV6dUdaTlVZbksyUVYzWHotc2RKbE12d2ZNNTFmYUJ2NHVpcmp5cHlRVUhSWjRaM05JWEI5eDBrOUdwX1JQQ1ZtcGJmMWszdzBOUWRwd1BNbEVJTHdkQTZVTVlveHRpNG1hTTVCSmVjM1lmMUE3S3RqSnNxZw==/http/minio.quay.svc:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931 with parts ['ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJbUZ6UlVkT2FXdHZhM051VjNvd1FsbFBkSHBMUlU5WE5XazFOR0pHYkV0WFZESm1hRmhaYjBoaFVXOGlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qTXpNU3dpYVdGMElqb3hOamMzTkRBMk16TXhMQ0psZUhBaU9qRTJOemMwTURZek5qRXNJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpaa0x6WmtNMlppWm1JelpHRTJNR0ZqWmpFeE1qRXhZMkpqWXpFME5tVmtZall3TkRBME5EazJaalU1TURFMFpUUTRNamhoTlRNMk1tWmlZbVV3WXpBNE4ySV9RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlkbUVsTWtKVVNIZGthMGxoWW5OdVZYaEpUVmcwZFhSNVJHSjJha2tsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEyT1RNeElpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLlNyZ1ppT1ljOUhIZEpXNHVKNXFCZlE1cFA3R09YUVF0Y1JTR1pzSlhBUUhpTU44dllNOXIwMC1ZcndJM1V1OE92R20tTldXb3RUZ1hVSXBxbkxwczFVWTVBMEd6dHRXN0NaRUpNYlplQVZPWU5Ed2xyTzM4WFZCb1Q1Rmxoc0JQOEZuM0l6aWtjc1dOcG9sYlppT19QUUVrZV9RM2N6QXUwSXZ6d1RCTFhreEc0RnRLd2N1UWRZOUxnNTg0UHNkNV9VdFZua1VWSXZpT1dHZFdxNVMwTWpoQWd3LWVZYjBQbmV6dUdaTlVZbksyUVYzWHotc2RKbE12d2ZNNTFmYUJ2NHVpcmp5cHlRVUhSWjRaM05JWEI5eDBrOUdwX1JQQ1ZtcGJmMWszdzBOUWRwd1BNbEVJTHdkQTZVTVlveHRpNG1hTTVCSmVjM1lmMUE3S3RqSnNxZw==', 'http', 'minio.quay.svc:9000', 'quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931']
gunicorn-web stdout | 2023-02-26 10:12:11,870 [256] [DEBUG] [app] Ending request: urn:request:30c515cb-76da-46e1-96c7-d9c59d24e5f0 (/_storage_proxy_auth) {'endpoint': '_storage_proxy_auth', 'request_id': 'urn:request:30c515cb-76da-46e1-96c7-d9c59d24e5f0', 'remote_addr': '10.91.0.19', 'http_method': 'GET', 'original_url': 'http://web_app_server/_storage_proxy_auth', 'path': '/_storage_proxy_auth', 'parameters': {}, 'json_body': None, 'confsha': 'f613904b', 'user-agent': 'containers/5.24.0 (github.com/containers/image)'}
nginx stdout | 2023/02/26 10:12:21 [error] 128#0: *22 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.6, server: , request: "GET /_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJbUZ6UlVkT2FXdHZhM051VjNvd1FsbFBkSHBMUlU5WE5XazFOR0pHYkV0WFZESm1hRmhaYjBoaFVXOGlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qTXpNU3dpYVdGMElqb3hOamMzTkRBMk16TXhMQ0psZUhBaU9qRTJOemMwTURZek5qRXNJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpaa0x6WmtNMlppWm1JelpHRTJNR0ZqWmpFeE1qRXhZMkpqWXpFME5tVmtZall3TkRBME5EazJaalU1TURFMFpUUTRNamhoTlRNMk1tWmlZbVV3WXpBNE4ySV9RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlkbUVsTWtKVVNIZGthMGxoWW5OdVZYaEpUVmcwZFhSNVJHSjJha2tsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEyT1RNeElpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLlNyZ1ppT1ljOUhIZEpXNHVKNXFCZlE1cFA3R09YUVF0Y1JTR1pzSlhBUUhpTU44dllNOXIwMC1ZcndJM1V1OE92R20tTldXb3RUZ1hVSXBxbkxwczFVWTVBMEd6dHRXN0NaRUpNYlplQVZPWU5Ed2xyTzM4WFZCb1Q1Rmxoc0JQOEZuM0l6aWtjc1dOcG9sYlppT19QUUVrZV9RM2N6QXUwSXZ6d1RCTFhreEc0RnRLd2N1UWRZOUxnNTg0UHNkNV9VdFZua1VWSXZpT1dHZFdxNVMwTWpoQWd3LWVZYjBQbmV6dUdaTlVZbksyUVYzWHotc2RKbE12d2ZNNTFmYUJ2NHVpcmp5cHlRVUhSWjRaM05JWEI5eDBrOUdwX1JQQ1ZtcGJmMWszdzBOUWRwd1BNbEVJTHdkQTZVTVlveHRpNG1hTTVCSmVjM1lmMUE3S3RqSnNxZw==/http/minio.quay.svc:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931 HTTP/1.1", upstream: "http://10.21.101.166:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931", host: "quay.example.com"

      as we can see, the configured Storage backend used from config.yaml gets translated into an IP Address which leads to an error

       

      curl -v 'http://10.21.101.166:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931'
*   Trying 10.21.101.166...
* TCP_NODELAY set
* Connected to 10.21.101.166 (10.21.101.166) port 9000 (#0)
> GET /quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931 HTTP/1.1
> Host: 10.21.101.166:9000
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Sun, 26 Feb 2023 10:16:09 GMT
< server: envoy
< 
* Connection #0 to host 10.21.101.166 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure

      where as the same URL with the configured Storage backend without resolving it in curl will succeed

      curl 'http://minio.quay.svc:9000/quay/datastorage/registry/sha256/6d/6d3fbfb3da60acf11211cbcc146edb60404496f59014e4828a5362fbbe0c087b?AWSAccessKeyId=minioadmin&Signature=va%2BTHwdkIabsnUxIMX4utyDbvjI%3D&Expires=1677406931'
{"architecture":"amd64","config":{"Hostname":"a5bb82701f3a","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container=docker"],"Cmd":null,"Image":"docker.io/kindest/base:v20221214-df207207","Volumes":null,"WorkingDir":"/","Entrypoint":["/usr/local/bin/entrypoint","/sbin/init"],"OnBuild":null,"Labels":{},"StopSignal":"SIGRTMIN+3"},"container":"a5bb82701f3a1762b1c86540327bd4106e41e6772be0910abdea4b4125afc519","container_config":{"Hostname":"a5bb82701f3a","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container=docker"],"Cmd":["infinity"],"Image":"docker.io/kindest/base:v20221214-df207207","Volumes":null,"WorkingDir":"/","Entrypoint":["sleep"],"OnBuild":null,"Labels":{},"StopSignal":"SIGRTMIN+3"},"created":"2022-12-20T03:36:35.176496673Z","docker_version":"20.10.21","history":[{"created":"2022-12-14T22:29:25.547887218Z","created_by":"COPY / / # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2022-12-14T22:29:25.547887218Z","created_by":"ENV container=docker","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-12-14T22:29:25.547887218Z","created_by":"STOPSIGNAL SIGRTMIN+3","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-12-14T22:29:25.547887218Z","created_by":"ENTRYPOINT [\"/usr/local/bin/entrypoint\" \"/sbin/init\"]","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-12-20T03:36:35.176496673Z","created_by":"infinity"}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:ca9f058da292d3509f171c5b74254099c0a16acf0f61937dd878c3316ffb21bb","sha256:250b0034f84363d4d125cbef60ab56b331afcbe818e5de7358251aac4c788f2a"]}}

       

      enhanced Quay nginx configuration

      client retrieves requested images as expected

      gunicorn-registry stdout | 2023-02-26 10:20:19,277 [258] [DEBUG] [app] Starting request: urn:request:d63c4001-14d7-465a-8d29-61b0c023187b (/v2/milang/kindest/node/blobs/sha256:7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e) {'X-Forwarded-For': '10.91.0.19, 127.0.0.6'}
gunicorn-registry stdout | 2023-02-26 10:20:19,582 [258] [DEBUG] [storage.downloadproxy] Proxying via URL http://quay.example.com/_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJa3hOTkVvMU4yeHVlR1pUZUhCd1gwbzVhek13U0VSbFJuUXpaRk5XUkRjNVpqRjNPVVpwUlRBelRITWlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qZ3hPU3dpYVdGMElqb3hOamMzTkRBMk9ERTVMQ0psZUhBaU9qRTJOemMwTURZNE5Ea3NJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpka0x6ZGtZV1ZsWkRRek5qVTVNbVUxWmpKaFpEZ3lNRGc1WWpWaVpXRTRaak0yWkRRMk1tRm1NVFUzWXpjeFpXSXpaV1EyT0Raa016QmhOemd3T0RObE5tVV9RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlSR293TkVkYVZFVkVkRkJtV0UxeldFVjNibUZLTkNVeVJrTm1ObEVsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEzTkRFNUlpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLkZLNVBTLXM1cDBJTy0xaW95a1JQbzBXSHJ2ei1wX3Z2WFhCQzZVNzBTTUZ6N3ZqdnN3YjJVTW5XYmFqZ3dCTlNFeXU0RlNiVEswWUVYTmdtMkJVWlJKV0FJd3lFS1c5ZmRzT2pCOHEyQXMyUHpFdGZ2VEI2RG82RDd3NnNSS1hNTElabS1hb3c5bnFVOURuN3pwcGltNVpUdkgydFRydVF3N0ljSFdsSkFuaXByUFlQZ1BTUUhnbG5PeTFqdG9kT0tGOGxudTdvU3VvNDUwR3IxTDU2WW1FZ2QtVWRaLTNJWlZhWWVEODY0YkZLZ0ZlSkZ1WWNkbVN4QWFPTmhyV2QwSUdQUWZBcnFLNllaRUc0VS1kMWx3VVkwRVNKcmxKaTFKTFZHaW1sUVV4U00yY3dfYll5STg4MWhXeVZvQ2FOUFVJcDZnLXBHLWh6Z29tdC1UenVoUQ==/http/minio.quay.svc:9000/quay/datastorage/registry/sha256/7d/7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e?AWSAccessKeyId=minioadmin&Signature=Dj04GZTEDtPfXMsXEwnaJ4%2FCf6Q%3D&Expires=1677407419
gunicorn-registry stdout | 2023-02-26 10:20:19,583 [261] [DEBUG] [storage.downloadproxy] Proxying via URL http://quay.example.com/_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJa3hOTkVvMU4yeHVlR1pUZUhCd1gwbzVhek13U0VSbFJuUXpaRk5XUkRjNVpqRjNPVVpwUlRBelRITWlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTNWxlR0Z0Y0d4bExtTnZiU0lzSW01aVppSTZNVFkzTnpRd05qZ3hPU3dpYVdGMElqb3hOamMzTkRBMk9ERTVMQ0psZUhBaU9qRTJOemMwTURZNE5Ea3NJbk4xWWlJNkluTjBiM0poWjJWd2NtOTRlU0lzSW1GalkyVnpjeUk2VzNzaWRIbHdaU0k2SW5OMGIzSmhaMlZ3Y205NGVTSXNJblZ5YVNJNkluRjFZWGt2WkdGMFlYTjBiM0poWjJVdmNtVm5hWE4wY25rdmMyaGhNalUyTHpJd0x6SXdPV0ppWXpnNE16azRNRFppWTJNMVpESTJOemhsT0RRNFptWXdaakE1TlRkaE5XSTVOREkxWmpKaU1XUmpNR0l6T0dRelptVTBaV0ZrTXpBNFlqa19RVmRUUVdOalpYTnpTMlY1U1dROWJXbHVhVzloWkcxcGJpWlRhV2R1WVhSMWNtVTlTR05CU1VJMVRtOVpORzk1SlRKQ1FUQktUSHBFV1hSbFlrMWhPRWtsTTBRbVJYaHdhWEpsY3oweE5qYzNOREEzTkRFNUlpd2lhRzl6ZENJNkltMXBibWx2TG5GMVlYa3VjM1pqT2prd01EQWlMQ0p6WTJobGJXVWlPaUpvZEhSd0luMWRMQ0pqYjI1MFpYaDBJanA3ZlgwLmdWZGZPa1loc3RXMzVRcjlxWURIUjQ3c2J2MF9OaDZaRGNReXV4M2hrMUZuekI0SzlLRDZtbk9PYnhFWEFJZ3E5Ny1SSldUb1ppaGN6aXdYeU13aVFudC1TSlBoMkowUHZRb0hqLTBncE1WYi10R3VaY2k5bXF5OTJsWm5wNDJ6cTJtNnhlWG12b3BLUjNubGZhY1ZwZUJJeXB4aTc5YXl4UjVfclMyenEwamRMNUZKd0RpeUc3NFk3ZFFkQTJCakNyRmZGX1ZzYWstbWVMbkQ0LWRZNVVLTzBrVkpqbWVTNjgzb1VqWXZlYlg2UVhmTTRVY0wxMDhJY1dfRWRlX09QQ1pTWDVURm5HUHBFbVNmMDE0S1RBZUhnaFhvdDlCcG9fSjIxUlR3Qk9Jd2dzV1lHckZCRDVoMGFGeEpxX2xMWWoyMWdhem4xSlNtQ054aXZxMjNwZw==/http/minio.quay.svc:9000/quay/datastorage/registry/sha256/20/209bbc8839806bcc5d2678e848ff0f0957a5b9425f2b1dc0b38d3fe4ead308b9?AWSAccessKeyId=minioadmin&Signature=HcAIB5NoY4oy%2BA0JLzDYtebMa8I%3D&Expires=1677407419
gunicorn-registry stdout | 2023-02-26 10:20:19,585 [258] [DEBUG] [app] Ending request: urn:request:d63c4001-14d7-465a-8d29-61b0c023187b (/v2/milang/kindest/node/blobs/sha256:7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e) {'endpoint': 'v2.download_blob', 'request_id': 'urn:request:d63c4001-14d7-465a-8d29-61b0c023187b', 'remote_addr': '127.0.0.6', 'http_method': 'GET', 'original_url': 'https://quay.example.com/v2/milang/kindest/node/blobs/sha256:7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e', 'path': '/v2/milang/kindest/node/blobs/sha256:7daeed436592e5f2ad82089b5bea8f36d462af157c71eb3ed686d30a78083e6e', 'parameters': {}, 'json_body': None, 'confsha': 'ae84ded7', 'user-agent': 'containers/5.24.0 (github.com/containers/image)'}
gunicorn-registry stdout | 2023-02-26 10:20:19,587 [261] [DEBUG] [app] Ending request: urn:request:06369e8a-bddc-4800-a4d8-78ab55be671c (/v2/milang/kindest/node/blobs/sha256:209bbc8839806bcc5d2678e848ff0f0957a5b9425f2b1dc0b38d3fe4ead308b9) {'endpoint': 'v2.download_blob', 'request_id': 'urn:request:06369e8a-bddc-4800-a4d8-78ab55be671c', 'remote_addr': '127.0.0.6', 'http_method': 'GET', 'original_url': 'https://quay.example.com/v2/milang/kindest/node/blobs/sha256:209bbc8839806bcc5d2678e848ff0f0957a5b9425f2b1dc0b38d3fe4ead308b9', 'path': '/v2/milang/kindest/node/blobs/sha256:209bbc8839806bcc5d2678e848ff0f0957a5b9425f2b1dc0b38d3fe4ead308b9', 'parameters': {}, 'json_body': None, 'confsha': 'ae84ded7', 'user-agent': 'containers/5.24.0 (github.com/containers/image)'}

      as we can see, the configured Storage backend used from config.yaml will not be resolved and therefor will succeed with SNI and Vhost matching mechanism used to provide HA or restricted Backend availability.

              rhn-support-ibazulic Ivan Bazulic
              rhn-support-milang Michaela Lang
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: