-
Bug
-
Resolution: Done
-
Critical
-
quay-3.7
A client is reporting that the storage replication is erroring out and shutting down due to the following error:
storagereplication stdout | 2022-11-28 17:03:27,604 [93] [ERROR] [__main__] Unknown exception when copying path sha256/ad/ad37630f333ad7e09a41b000cd075a2da3c648c8e53bb777c4801385fa1e2ad5 of image storage 5dc41c15-6f31-4f13-ac95-c42873f8deea to loc rtp1 storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/workers/storagereplication.py", line 124, in replicate_storage storagereplication stdout | storage.copy_between(path_to_copy, existing_location, location) storagereplication stdout | File "/quay-registry/storage/distributedstorage.py", line 100, in copy_between storagereplication stdout | source_storage.copy_to(destination_storage, path) storagereplication stdout | File "/quay-registry/storage/cloud.py", line 507, in copy_to storagereplication stdout | destination.stream_write(path, fp) storagereplication stdout | File "/quay-registry/storage/cloud.py", line 314, in stream_write storagereplication stdout | _, write_error = self._stream_write_internal(path, fp, content_type, content_encoding) storagereplication stdout | File "/quay-registry/storage/cloud.py", line 357, in _stream_write_internal storagereplication stdout | bytes_staged = self.stream_write_to_fp(fp, buf, bytes_to_copy) storagereplication stdout | File "/quay-registry/storage/basestorage.py", line 108, in stream_write_to_fp storagereplication stdout | buf = in_fp.read(size_to_read) storagereplication stdout | File "/quay-registry/storage/cloud.py", line 92, in read storagereplication stdout | resp = self._key.read(amt) storagereplication stdout | File "/usr/local/lib/python3.8/site-packages/botocore/response.py", line 109, in read storagereplication stdout | self._verify_content_length() storagereplication stdout | File "/usr/local/lib/python3.8/site-packages/botocore/response.py", line 161, in _verify_content_length storagereplication stdout | raise IncompleteReadError( storagereplication stdout | botocore.exceptions.IncompleteReadError: 13760500 read, but total bytes expected is 192738946. storagereplication stdout | 2022-11-28 17:03:27,608 [93] [ERROR] [workers.queueworker] The worker has encountered an error via the job and will not take new jobs
We need information what could cause this exception in the code and how to mitigate it. Please check!
- links to
- mentioned on
(9 mentioned on)