Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-6586

Big layer upload fails on Ceph/RADOS driver

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • quay-v3.11.0
    • quay-v3.10.0
    • quay
    • False
    • None
    • False
    • Quay Enterprise
    • 0

      When Ceph/RADOS driver is used, uploading of layers over 5 GB in size breaks down due to Quay doing a one-off copy of the assembled layer which usually results in a timeout on Boto side. To address this, we limit the maximum chunk size for the upload to something that is user configurable. If not specified, 32 MB chunk size is used (the same as Docker uses in its own documentation).

      The following code change seems to resolve the problem:

      ...
      class RadosGWStorage(_CloudStorage):
          def __init__(
              self,
              context,
              hostname,
              is_secure,
              storage_path,
              access_key,
              secret_key,
              bucket_name,
              port=None,
              maximum_chunk_size_mb=None,
          ):
              upload_params = {}
              connect_kwargs = {
                  "endpoint_url": _build_endpoint_url(hostname, port=port, is_secure=is_secure),
              }
      
              super(RadosGWStorage, self).__init__(
                  context,
                  boto3.session.Session,
                  connect_kwargs,
                  upload_params,
                  storage_path,
                  bucket_name,
                  access_key,
                  secret_key,
              )
      
              chunk_size = (
                  maximum_chunk_size_mb if maximum_chunk_size_mb is not None else 32
              )  # 32mb default, as used in Docker registry:2
              self.maximum_chunk_size = chunk_size * 1024 * 1024
      ...
      

      The RHOCSStorage class for Noobaa still inherits everything from the Ceph class.

            rhn-support-ibazulic Ivan Bazulic
            rhn-support-ibazulic Ivan Bazulic
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: