Major If a push is tried to a repository that is part of an organization where proxy settings are configured, then proxy code is invoked regardless of the fact that we're trying to push instead of pull images through Quay. If the proxy settings are incorrect, push to the repo fails with a 400.
The following example illustrates what happened: ubi9-minimal repository was set to be mirrored from registry.redhat.io inside the org ubi9 which had a partially correct proxy settings for registry.redhat.io (missing username and password).
Request start:
gunicorn-registry stdout | 2023-11-29 14:02:13,551 [166683] [DEBUG] [app] Starting request: urn:request:75bef257-b902-4876-b832-4da9016256c3 (/v2/ubi9/ubi9-minimal/blobs/sha256:a032f50e22ae11b241fcf
38b4a787f0e51009578eedaf9d05894f5f38fd12af5) {'X-Forwarded-For': 'IP ADDRESS'}
Decoded JWT token for this request:
{
"iss": "quay",
"aud": "QUAY_HOSTNAME",
"nbf": 1701266533,
"iat": 1701266533,
"exp": 1701270133,
"sub": "ubi9+ubi9robot",
"access": [
{
"type": "repository",
"name": "ubi9/ubi9-minimal",
"actions": [
"push",
"pull"
]
}
],
"context": {
...
}
The actions clearly specify both push and pull, since this is done by skopeo inside the mirror worker. Proxy cache config check:
gunicorn-registry stdout | 2023-11-29 14:02:13,559 [166683] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."organization_id", "t1"."creation_date", "t1"."upstream_registry", "t1"."upstream_registry_username", "t1"."upstream_registry_password", "t1"."expiration_s", "t1"."insecure" FROM "proxycacheconfig" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") WHERE (("t2"."username" = %s) AND ("t2"."organization" = %s)) LIMIT %s OFFSET %s', ['ubi9', True, 1, 0])
and then we see the invocation of the proxy cache setup:
gunicorn-registry stdout | 2023-11-29 14:02:13,567 [166683] [DEBUG] [data.cache.impl] Checking cache for key upstream_token__ubi9/registry.redhat.io/ubi9-minimal gunicorn-registry stdout | 2023-11-29 14:02:13,568 [166683] [DEBUG] [data.cache.impl] Found no result in cache for key upstream_token__ubi9/registry.redhat.io/ubi9-minimal; calling loader gunicorn-registry stdout | 2023-11-29 14:02:13,568 [166683] [DEBUG] [data.cache.impl] Got loaded result for key upstream_token__ubi9/registry.redhat.io/ubi9-minimal: None gunicorn-registry stdout | 2023-11-29 14:02:13,568 [166683] [DEBUG] [data.cache.impl] Not caching loaded result for key upstream_token__ubi9/registry.redhat.io/ubi9-minimal: None gunicorn-registry stdout | 2023-11-29 14:02:13,569 [166683] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): registry.redhat.io:443 gunicorn-registry stdout | 2023-11-29 14:02:13,595 [166683] [DEBUG] [urllib3.connectionpool] https://registry.redhat.io:443 "GET /v2/ HTTP/1.1" 401 99 gunicorn-registry stdout | 2023-11-29 14:02:13,966 [166683] [DEBUG] [urllib3.connectionpool] https://registry.redhat.io:443 "GET /auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&scope=repository%3Aubi9-minimal%3Apull HTTP/1.1" 401 214 gunicorn-registry stdout | 2023-11-29 14:02:13,967 [166683] [DEBUG] [endpoints.v2] sending response: b'{"errors":[{"code":"INVALID_REQUEST","detail":{},"message":"the requested image may not exist in the upstream registry, or the configured Quay organization credentials have insufficient rights to access it (Failed to get token from: \'https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth\', with status code: 401)"}]}\n'
This results with the initial HEAD request to fail with a 400 because proxy cache config couldn't be verified.
gunicorn-registry stdout | 2023-11-29 14:02:13,969 [166683] [DEBUG] [app] Ending request: urn:request:75bef257-b902-4876-b832-4da9016256c3 (/v2/ubi9/ubi9-minimal/blobs/sha256:a032f50e22ae11b241fcf38b4a787f0e51009578eedaf9d05894f5f38fd12af5) {'endpoint': 'v2.check_blob_exists', 'request_id': 'urn:request:75bef257-b902-4876-b832-4da9016256c3', 'remote_addr': 'IP_ADDR', 'http_method': 'HEAD', 'original_url': 'https://QUAY_HOSTNAME/v2/ubi9/ubi9-minimal/blobs/sha256:a032f50e22ae11b241fcf38b4a787f0e51009578eedaf9d05894f5f38fd12af5', 'path': '/v2/ubi9/ubi9-minimal/blobs/sha256:a032f50e22ae11b241fcf38b4a787f0e51009578eedaf9d05894f5f38fd12af5', 'parameters': {}, 'json_body': None, 'confsha': '1e576e99', 'user-agent': 'skopeo/1.11.2-dev'}
gunicorn-registry stdout | 2023-11-29 14:02:13,970 [166683] [INFO] [gunicorn.access] IP ADDRESS - - [29/Nov/2023:14:02:13 +0000] "HEAD /v2/ubi9/ubi9-minimal/blobs/sha256:a032f50e22ae11b241fcf38b4a787f0e51009578eedaf9d05894f5f38fd12af5 HTTP/1.1" 400 0 "-" "skopeo/1.11.2-dev"
Please check. Thank you!
