-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
quay-v3.15.3, quay-v3.16.2
-
False
-
-
False
-
-
-
Important
-
Customer Escalated, Customer Facing, Customer Reported
This is a bug affecting Quay 3.15.3/3.16.2
The repositorygcworker fails to delete the physical objects from the storage with the following error:
repositorygcworker stdout | 2026-03-10 11:19:07,112 [74] [DEBUG] [data.model.storage] Removing sha256/f5/f5eed114ccc56fddaf974fcfe987df9c0005f1f26c69131e12d9f32b7aeba19a from local_eu repositorygcworker stdout | 2026-03-10 11:19:07,112 [74] [DEBUG] [util.locking] Releasing lock LARGE_GARBAGE_COLLECTION repositorygcworker stdout | 2026-03-10 11:19:07,112 [74] [DEBUG] [redis_lock.release] Releasing 'lock:LARGE_GARBAGE_COLLECTION'. repositorygcworker stdout | 2026-03-10 11:19:07,112 [74] [DEBUG] [util.locking] Released lock LARGE_GARBAGE_COLLECTION repositorygcworker stdout | 2026-03-10 11:19:07,113 [74] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2026-03-10 11:19:07,113 [74] [ERROR] [workers.worker] Operation raised exception repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/workers/worker.py", line 86, in _operation_func repositorygcworker stdout | return operation_func() repositorygcworker stdout | ^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/quay-registry/workers/queueworker.py", line 134, in poll_queue repositorygcworker stdout | self.process_queue_item(job_details) repositorygcworker stdout | File "/quay-registry/workers/repositorygcworker.py", line 30, in process_queue_item repositorygcworker stdout | self._perform_gc(job_details) repositorygcworker stdout | File "/quay-registry/workers/repositorygcworker.py", line 46, in _perform_gc repositorygcworker stdout | if not model.gc.purge_repository(marker.repository): repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/quay-registry/data/model/gc.py", line 98, in purge_repository repositorygcworker stdout | _purge_repository_contents(repo) repositorygcworker stdout | File "/quay-registry/data/model/gc.py", line 244, in _purge_repository_contents repositorygcworker stdout | _run_garbage_collection(context) repositorygcworker stdout | File "/quay-registry/data/model/gc.py", line 314, in _run_garbage_collection repositorygcworker stdout | storage_ids_removed = set(storage.garbage_collect_storage(context.blob_ids)) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/quay-registry/data/model/storage.py", line 237, in garbage_collect_storage repositorygcworker stdout | config.store.remove({location_name}, image_path) repositorygcworker stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper repositorygcworker stdout | storage = self._storages[random.sample(locations, 1)[0]] repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample repositorygcworker stdout | raise TypeError("Population must be a sequence. " repositorygcworker stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d).
This leads to orphaned files in the defined storage backend and wasted storage space
This bug was likely introduced with the upgrade to Python 3.12.
Starting from Python 3.11, random.sample() strictly requires a sequence, automatic conversion to lists is no longer supported.
Documentation
The issue can be always reproduced
Steps to reproduce the issue:
- Deploy Quay 3.15.3 with storage replication feature enabled:
// config.yaml DISTRIBUTED_STORAGE_CONFIG: local_us: - RHOCSStorage - access_key: bucket_name: quay-datastore-976ec783-9433-4d1d-bdf2-2799e0059781 hostname: s3.openshift-storage.svc.cluster.local is_secure: true port: 443 secret_key: storage_path: /datastorage/registry local_eu: - RHOCSStorage - access_key: bucket_name: datastore-replica-5a033188-dc1c-43cc-8974-f70ef8a97345 hostname: s3.openshift-storage.svc.cluster.local is_secure: true port: 443 secret_key: storage_path: /datastorage/registry DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: - local_us - local_eu DISTRIBUTED_STORAGE_PREFERENCE: - local_us - local_eu FEATURE_STORAGE_REPLICATION: true
// QuayRegistry CR apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: reproducer namespace: openshift-operators spec: components: - kind: clair managed: true overrides: replicas: 1 resources: {} - kind: postgres managed: true - kind: objectstorage managed: false - kind: redis managed: true - kind: horizontalpodautoscaler managed: false - kind: route managed: true - kind: mirror managed: true overrides: env: - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE value: local_us replicas: 1 resources: {} - kind: monitoring managed: true - kind: tls managed: true - kind: quay managed: true overrides: env: - name: DEBUGLOG value: "true" - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE value: local_us replicas: 1 resources: {} - kind: clairpostgres managed: true configBundleSecret: config-bundle
- Push an image to the registry:
$ podman push quay.example.com/admin/golang:1.24-alpine
- Confirm blobs are replicated in both defined storage locations
- Delete the repository containing the image to trigger repositorygcworker immediately
- is related to
-
-
- Review
-
