-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
quay-3.7
-
False
-
None
-
False
-
Quay Enterprise
Client is having issues with pushing images to nonexisting orgnaizations in Quay. Each time he gets a 405 Method not allowed error. The error consistently happens regardless of how many Quay instances are run (OpenShift deployment):
... DEBU[0000] Looking up image "vdcr-registry-,,,/rhtest/ubi:latest" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "vdcr-registry-.../rhtest/ubi:latest" ... DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@1b7b40f4f1eec6a6dc95c76adffb10174c0f2f4fefb9c7d78c0ffa6a9f7c49c7" DEBU[0000] Found image "vdcr-registry.../rhtest/ubi:latest" as "vdcr-registry-.../rhtest/ubi:latest" in local containers storage Error: trying to reuse blob sha256:db1385a464bbd6ed45b4ca308807b8494c2d5a0239057840d1d5f96a61aef5ea at destination: Requesting bearer token: invalid status code from registry 405 (Method Not Allowed)
When looking at the Quay logs:
gunicorn-registry stdout | 2022-10-20 06:57:19,604 [333] [DEBUG] [endpoints.v2.v2auth] Match: ('rhtest/ubi', 'rhtest/ubi', 'pull,push' ) gunicorn-registry stdout | 2022-10-20 06:57:19,604 [333] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."pass word_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_lo gin_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_n ame", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_access ed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['rhtest', 1, 0]) gunicorn-registry stdout | 2022-10-20 06:57:19,606 [326] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."pass word_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_lo gin_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_n ame", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_access ed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['rhtest', 1, 0]) gunicorn-registry stdout | 2022-10-20 06:57:19,608 [333] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", " t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid ", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."in voice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice _email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."cre ation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['rhtest', 'ubi', 3, 1, 0]) gunicorn-registry stdout | 2022-10-20 06:57:19,608 [326] [DEBUG] [endpoints.v2.v2auth] Creating organization: rhtest/ubi gunicorn-registry stdout | 2022-10-20 06:57:19,609 [326] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."pass word_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_lo gin_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_n ame", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_access ed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) OR ("t1"."email" = %s)) LIMIT %s OFFSET %s', ['rhtest', '646e02a4-35fa-44e1-b873 -4850727efb27', 1, 0]) gunicorn-registry stdout | 2022-10-20 06:57:19,611 [326] [DEBUG] [data.model.user] Existing user with same username or email. gunicorn-registry stdout | 2022-10-20 06:57:19,611 [333] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."pass word_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['rhtest', 1, 0]) gunicorn-registry stdout | 2022-10-20 06:57:19,611 [326] [DEBUG] [endpoints.v2] sending response: b'{"errors":[{"code":"UNSUPPORTED","detail":{},"message":"Cannot create organization"}]}\n'
However, the organization rhtest does not exist and is not created:
postgres=# \c "vdcr-registry-quay-database"; You are now connected to database "vdcr-registry-quay-database" as user "postgres". vdcr-registry-quay-database=# SELECT * FROM "user" WHERE username = 'rhtest'; id | uuid | username | password_hash | email | verified | stripe_id | organization | robot | invoice_email | invalid_login_attempts | last_invalid_login | removed_tag_expiration_s | enabled | invoice_email_addr ess | company | family_name | given_name | location | maximum_queued_builds_count | creation_date | last_accessed ----+------+----------+---------------+-------+----------+-----------+--------------+-------+---------------+------------------------+--------------------+--------------------------+---------+------------------- ----+---------+-------------+------------+----------+-----------------------------+---------------+--------------- (0 rows)
so I'm not sure where the info about the existing username comes from. I tried replicating this locally on both podman and docker. On docker, push goes through successfully without issues. On podman, I also get a 405 on initial push, but it's because of a race condition, there are multiple requests being sent to create an org, the org does get created but push fails. Subsequent pushes succeed without issues.
CREATE_NAMESPACE_ON_PUSH is set to true, Quay version is 3.7.8. Client has already tried the same approach on other registries with the same version and then it worked without issues. Only this one is problematic, for whatever reason. Can you please advise?
