-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
quay-3.15.0
-
Quality / Stability / Reliability
-
False
-
-
False
-
-
ERROR:
2025-11-05T06:54:05.705697278Z logrotateworker stdout | 2025-11-05 06:54:05,705 [84] [DEBUG] [util.locking] Released lock ACTION_LOG_ROTATION 2025-11-05T06:54:05.706318758Z logrotateworker stdout | 2025-11-05 06:54:05,705 [84] [ERROR] [workers.worker] Operation raised exception 2025-11-05T06:54:05.706318758Z logrotateworker stdout | Traceback (most recent call last): 2025-11-05T06:54:05.706318758Z logrotateworker stdout | File "/quay-registry/workers/worker.py", line 86, in _operation_func 2025-11-05T06:54:05.706318758Z logrotateworker stdout | return operation_func() 2025-11-05T06:54:05.706318758Z logrotateworker stdout | ^^^^^^^^^^^^^^^^ 2025-11-05T06:54:05.706318758Z logrotateworker stdout | File "/quay-registry/workers/logrotateworker.py", line 47, in _archive_logs 2025-11-05T06:54:05.706318758Z logrotateworker stdout | self._perform_archiving(cutoff_date) 2025-11-05T06:54:05.706318758Z logrotateworker stdout | File "/quay-registry/workers/logrotateworker.py", line 70, in _perform_archiving 2025-11-05T06:54:05.706318758Z logrotateworker stdout | _write_logs(filename, formatted_logs, log_archive) 2025-11-05T06:54:05.706318758Z logrotateworker stdout | File "/quay-registry/workers/logrotateworker.py", line 130, in _write_logs 2025-11-05T06:54:05.706318758Z logrotateworker stdout | log_archive.store_file(tempfile, JSON_MIMETYPE, content_encoding="gzip", file_id=filename) 2025-11-05T06:54:05.706318758Z logrotateworker stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file 2025-11-05T06:54:05.706318758Z logrotateworker stdout | self._storage.stream_write( 2025-11-05T06:54:05.706318758Z logrotateworker stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper 2025-11-05T06:54:05.706318758Z logrotateworker stdout | storage = self._storages[random.sample(locations, 1)[0]] 2025-11-05T06:54:05.706318758Z logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-11-05T06:54:05.706318758Z logrotateworker stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample 2025-11-05T06:54:05.706318758Z logrotateworker stdout | raise TypeError("Population must be a sequence. " 2025-11-05T06:54:05.706318758Z logrotateworker stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d).
Issue Summary:
LogRotateWorker fails to archive logs due to a `TypeError` in `distributedstorage.py` when attempting to write archived logs to storage. The worker successfully fetches logs and creates the compressed archive (e.g., `766918-771918-logentry3.txt.gz`), but fails during the storage write phase.
Steps to Reproduce:
1. LogRotateWorker is triggered via APScheduler on a 1-hour interval:
2025-11-05T06:54:04.967061769Z logrotateworker stdout | 2025-11-05 06:54:04,966 [84] [INFO] [apscheduler.executors.default] Running job "LogRotateWorker._archive_logs (trigger: interval[1:00:00], next run at: 2025-11-05 07:54:04 GMT)"
2. Logs are retrieved and archive file is generated:
2025-11-05T06:54:05.279606745Z logrotateworker stdout | 2025-11-05 06:54:05,278 [84] [DEBUG] [__main__] Archiving logs rotation 766918-771918-logentry3.txt.gz
3. Failure occurs during `_storage.stream_write` in `distributedstorage.py` at line 23:
https://github.com/quay/quay/blob/6cd9b8535edb49277b0b7bd13830820c71aa8dfd/storage/distributedstorage.py#L23
**
Error:
TypeError: Population must be a sequence. For dicts or sets, use sorted(d).
Root Cause:
- In `logrotateworker.py`, `_perform_archiving` uses `DelegateUserfiles` to manage storage locations:
→ https://github.com/quay/quay/blob/6cd9b8535edb49277b0b7bd13830820c71aa8dfd/workers/logrotateworker.py#L61 - `DelegateUserfiles` stores locations in a {}set{}:
→ `self._locations = {location}`
https://github.com/quay/quay/blob/6cd9b8535edb49277b0b7bd13830820c71aa8dfd/data/userfiles.py#L71 - `DistributedStorage.stream_write` calls:
storage = self._storages[random.sample(locations, 1)[0]]
- `random.sample()` {}does not accept sets{} — it requires a sequence (list/tuple), hence the `TypeError`.
Proposed Solutions:
1. Change storage locations to list in `DelegateUserfiles`:
locations = [location] # instead of {location}
2. Convert set to sorted list before sampling (per Python error suggestion):
storage = self._storages[random.sample(sorted(locations), 1)[0]]
Code References:
- [1] LogRotateWorker._perform_archiving → https://github.com/quay/quay/blob/6cd9b8535edb49277b0b7bd13830820c71aa8dfd/workers/logrotateworker.py#L61
- [2] DelegateUserfiles.{}init{} → https://github.com/quay/quay/blob/6cd9b8535edb49277b0b7bd13830820c71aa8dfd/data/userfiles.py#L71
- [3] DistributedStorage.stream_write → https://github.com/quay/quay/blob/6cd9b8535edb49277b0b7bd13830820c71aa8dfd/storage/distributedstorage.py#L23
Inspect file and config.yaml file attached.
