Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: quay-v3.7.12, quay-v3.8.9, quay-v3.9.0
Affects Version/s: quay-v3.7.5, quay-v3.7.7
Component/s: -area/quota, quay
Labels:
- quay
- quota
- triaged

Blocked:
False
Blocked Reason:
None
Ready:
False
Epic Link:
Scalable storage calculation consumption and quota enforcement
Product:
Quay Enterprise
Git Pull Request:
https://github.com/quay/quay/pull/1876

Severity:
Important
Customer Impact:

Customer Escalated

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

When one opens superuser/users endpoint in Quay, the endpoint loads all users from the user table. This also in turn loads quota settings for all users significantly increasing load times for that particular endpoint. Our client has a Quay instance with over 4 thousand users. Loading times without quota management are usually 0.4-0.6 seconds. When quota management is turned on, load times increase to 55+ seconds, sometimes going over 60 seconds and causing a load timeout. The following log is observed for each user in Quay's database:

{"log": "gunicorn-web stdout | 2022-08-11 13:50:36,663 [233] [DEBUG] [peewee] ('SELECT `t1`.`id`, `t1`.`namespace_id`, `t1`.`limit_bytes` FROM `userorganizationquota` AS `t1` INNER JOIN `user` AS `t2` ON (`t1`.`namespace_id` = `t2`.`id`) WHERE (`t2`.`username` = %s)', ['some_username'])\n", "stream": "stdout", "time": "2022-08-11T13:50:36.663922326Z"}

This is the log that shows the length of the load time for the superuser/users API endpoint:

{"log": "gunicorn-web stdout | 2022-08-11 13:51:26,583 [233] [DEBUG] [app] Ending request: urn:request:d7dd372b-549c-474f-b63c-e4ab533ea1c7 (/api/v1/superuser/users/) {'endpoint': 'api.superuserlist', 'request_id': 'urn:request:d7dd372b-549c-474f-b63c-e4ab533ea1c7', 'remote_addr': 'IP_ADDRESS', 'http_method': 'GET', 'original_url': 'https://QUAY_HOSTNAME/api/v1/superuser/users/', 'path': '/api/v1/superuser/users/', 'parameters': {}, 'json_body': None, 'confsha': '65d8bf1b', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36'}\n", "stream": "stdout", "time": "2022-08-11T13:51:26.584568357Z"}
...
{"log": "gunicorn-web stdout | 2022-08-11 13:50:36,176 [233] [DEBUG] [app] Starting request: urn:request:d7dd372b-549c-474f-b63c-e4ab533ea1c7 (/api/v1/superuser/users/) {'X-Forwarded-For': 'IP_ADDRESS'}\n", "stream": "stdout", "time": "2022-08-11T13:50:36.177580686Z"}