-
Bug
-
Resolution: Done
-
Critical
-
quay-v3.6.2
-
3
-
False
-
False
-
Quay Enterprise
-
We've set up mirroring of docker.io/library/python repository with the following regex: latest,3.[789].?,3.[789].,3..?,3..,3-,3.?.?{},3..?,3..??-. The mirroring works fine but the latest tag is not being mirrored. In fact, it is being deleted 1 second after mirroring has been done:
MariaDB [quay]> select * from tag where name = 'latest'; +-------+--------+---------------+-------------+-------------------+-----------------+--------+-----------+-------------+---------------+ | id | name | repository_id | manifest_id | lifetime_start_ms | lifetime_end_ms | hidden | reversion | tag_kind_id | linked_tag_id | +-------+--------+---------------+-------------+-------------------+-----------------+--------+-----------+-------------+---------------+ | 16365 | latest | 2 | 2998 | 1643813610989 | 1643813611088 | 0 | 0 | 1 | NULL | | 20495 | latest | 2 | 2998 | 1643817032981 | 1643817033088 | 0 | 0 | 1 | NULL | | 24628 | latest | 2 | 2998 | 1643820408341 | 1643820408474 | 0 | 0 | 1 | NULL | +-------+--------+---------------+-------------+-------------------+-----------------+--------+-----------+-------------+---------------+ 3 rows in set (0.005 sec)
As can be seen, the lifetime_end_ms is set to 1 second after creation. This is also what the tag history setting shows:
latest was deleted Wed, Feb 2, 2022 5:46 PM latest was created pointing to SHA256 a7a73f894e75 Wed, Feb 2, 2022 5:46 PM
Unfortunately, I'm unable to find the exact cause of the deletion of the tag, I cannot figure out what set the lifetime_end_ms setting. When searching through the logs, the only thing I can find is this:
gunicorn-registry stdout | 2022-02-02 16:46:48,344 [242] [DEBUG] [peewee] ('INSERT INTO `tag` (`name`, `repository_id`, `manifest_id`, `lifetime_start_ms`, `lifetime_end_ms`, `hidden`, `reversion`, `tag_kind_id`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)', ['latest', 2, 2998, 1643820408341, None, False, False, 1]) gunicorn-registry stdout | 2022-02-02 16:46:48,351 [242] [DEBUG] [peewee] ('INSERT INTO `logentry3` (`kind_id`, `account_id`, `performer_id`, `repository_id`, `datetime`, `ip`, `metadata_json`) VALUES (%s, %s, %s, %s, %s, %s, %s)', [42, 3, 4, 2, datetime.datetime(2022, 2, 2, 16, 46, 48, 349642), '54.194.177.148', '{"repo": "python", "namespace": "library", "user-agent": "skopeo/1.4.2-dev", "tag": "latest", "username": "library+repomirror", "is_robot": true, "resolved_ip": {"provider": "internet", "service": "IE", "sync_token": null, "country_iso_code": "IE"}}'])
This shows that the insert operation set the lifetime_end_ms to null, so that field should be empty. Same logs can be seen for other tags that are still in the registry, for example:
gunicorn-registry stdout | 2022-02-02 16:37:24,627 [227] [DEBUG] [peewee] ('INSERT INTO `tag` (`name`, `repository_id`, `manifest_id`, `lifetime_start_ms`, `lifetime_end_ms`, `hidden`, `reversion`, `tag_kind_id`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)', ['3.9.1-buster', 2, 5022, 1643819844622, None, False, False, 1]) gunicorn-registry stdout | 2022-02-02 16:37:24,634 [227] [DEBUG] [peewee] ('INSERT INTO `logentry3` (`kind_id`, `account_id`, `performer_id`, `repository_id`, `datetime`, `ip`, `metadata_json`) VALUES (%s, %s, %s, %s, %s, %s, %s)', [42, 3, 4, 2, datetime.datetime(2022, 2, 2, 16, 37, 24, 632179), '54.194.177.148', '{"repo": "python", "namespace": "library", "user-agent": "skopeo/1.4.2-dev", "tag": "3.9.1-buster", "username": "library+repomirror", "is_robot": true, "resolved_ip": {"provider": "internet", "service": "IE", "sync_token": null, "country_iso_code": "IE"}}'])
The tag is not expired, as can be seen in the UI and in the tags table.
MariaDB [quay]> select * from tag where name = '3.9.1-buster'; +-------+--------------+---------------+-------------+-------------------+-----------------+--------+-----------+-------------+---------------+ | id | name | repository_id | manifest_id | lifetime_start_ms | lifetime_end_ms | hidden | reversion | tag_kind_id | linked_tag_id | +-------+--------------+---------------+-------------+-------------------+-----------------+--------+-----------+-------------+---------------+ | 15671 | 3.9.1-buster | 2 | 5022 | 1643811927174 | 1643816465686 | 0 | 0 | 1 | NULL | | 19719 | 3.9.1-buster | 2 | 5022 | 1643816465686 | 1643819844622 | 0 | 0 | 1 | NULL | | 23852 | 3.9.1-buster | 2 | 5022 | 1643819844622 | NULL | 0 | 0 | 1 | NULL | +-------+--------------+---------------+-------------+-------------------+-----------------+--------+-----------+-------------+---------------+ 3 rows in set (0.005 sec)
Can you please check the logs and see what's causing this issue? We have a customer that is impacted by this error and cannot mirror images properly.
Thanks!
Acceptance Criteria
- The mirror rollback operation will not erroneously delete tags instead of reverting them to a previous version
QE How to test
The issue is when the amount of tags exceed the number of tags retrieved during the rollback operation, which was 100.
Steps on how to test:
- Set REPO_MIRROR_TAG_ROLLBACK_PAGE_SIZE to a low number, ex. 2. The issue presents when the amount of tags is greater than the page size, since we don't want to test pulling down 100 tags, this field reproduces the issue without using as much storage.
- Mirror the "docker.io/library/python" image with the following tags: "latest,3.8,3.9"
- This will pull the latest, 3.8, and 3.9 tags successfully
- Mirror the "docker.io/library/python" image again with the following tags: "latest,3.4.9-alpine,3.8,3.9"
- The "3.4.9-alpine" tag will fail due to not complying with the docker spec
- Confirm that the latest, 3.8, and 3.9 tags are still present
- account is impacted by
-
-
- Closed
-
- is cloned by
-
-
- Closed
-
