Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-4793

Storage replication errors out with a strange exception

XMLWordPrintable

    • False
    • None
    • False
    • Quay Enterprise

      A client is reporting that the storage replication is erroring out and shutting down due to the following error:

      storagereplication stdout | 2022-11-28 17:03:27,604 [93] [ERROR] [__main__] Unknown exception when copying path sha256/ad/ad37630f333ad7e09a41b000cd075a2da3c648c8e53bb777c4801385fa1e2ad5 of image storage 5dc41c15-6f31-4f13-ac95-c42873f8deea to loc rtp1
      storagereplication stdout | Traceback (most recent call last):
      storagereplication stdout |   File "/quay-registry/workers/storagereplication.py", line 124, in replicate_storage
      storagereplication stdout |     storage.copy_between(path_to_copy, existing_location, location)
      storagereplication stdout |   File "/quay-registry/storage/distributedstorage.py", line 100, in copy_between
      storagereplication stdout |     source_storage.copy_to(destination_storage, path)
      storagereplication stdout |   File "/quay-registry/storage/cloud.py", line 507, in copy_to
      storagereplication stdout |     destination.stream_write(path, fp)
      storagereplication stdout |   File "/quay-registry/storage/cloud.py", line 314, in stream_write
      storagereplication stdout |     _, write_error = self._stream_write_internal(path, fp, content_type, content_encoding)
      storagereplication stdout |   File "/quay-registry/storage/cloud.py", line 357, in _stream_write_internal
      storagereplication stdout |     bytes_staged = self.stream_write_to_fp(fp, buf, bytes_to_copy)
      storagereplication stdout |   File "/quay-registry/storage/basestorage.py", line 108, in stream_write_to_fp
      storagereplication stdout |     buf = in_fp.read(size_to_read)
      storagereplication stdout |   File "/quay-registry/storage/cloud.py", line 92, in read
      storagereplication stdout |     resp = self._key.read(amt)
      storagereplication stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/response.py", line 109, in read
      storagereplication stdout |     self._verify_content_length()
      storagereplication stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/response.py", line 161, in _verify_content_length
      storagereplication stdout |     raise IncompleteReadError(
      storagereplication stdout | botocore.exceptions.IncompleteReadError: 13760500 read, but total bytes expected is 192738946.
      storagereplication stdout | 2022-11-28 17:03:27,608 [93] [ERROR] [workers.queueworker] The worker has encountered an error via the job and will not take new jobs
      

      We need information what could cause this exception in the code and how to mitigate it. Please check!

              sleesinc Kenny Lee Sin Cheong
              rhn-support-ibazulic Ivan Bazulic
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: