Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-713

Quay 3.3 pod using RWO PV does not have write permissions to /datastore/registry

XMLWordPrintable

    • Quay Enterprise

      I'm comparing Quay Enterprise 3.2 and 3.3, both running on OpenShift 4.3 using the Operator.

      3.2 Ecosystem: 

      apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: acme
spec:
  quay:
    imagePullSecretName: redhat-pull-secret
    keepConfigDeployment: true
    database:
      volumeSize: 10Gi
    registryStorage:
      persistentVolumeAccessModes:
        - ReadWriteOnce
      persistentVolumeSize: 10Gi
    hostname: quay32.apps.d1.casl.rht-labs.com
    superuserCredentialsSecretName: superuser-creds
  clair:
    enabled: true
    imagePullSecretName: redhat-pull-secret

      3.3 Ecosystem: 

      apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: acme
spec:
  quay:
    imagePullSecretName: redhat-pull-secret
    #keepConfigDeployment: true
    database:
      volumeSize: 10Gi
    registryStorage:
      persistentVolumeAccessModes:
        - ReadWriteOnce
      persistentVolumeSize: 10Gi
    externalAccess:
      hostname: quay.apps.d1.casl.rht-labs.com
    superuserCredentialsSecretName: superuser-creds
  clair:
    enabled: true
    imagePullSecretName: redhat-pull-secret

      In Quay 3.3, the PV isn't writable by the quay pod. When I push an image, 3.2 works just fine, but 3.3 fails with the following errors in the logs. btw, I have observed this exact same behavior on multiple clusters, backed by either Cinder volumes or vSphere volumes. 

      OSError: [Errno 13] Permission denied: '/datastorage/registry/uploads'
gunicorn-registry stdout | 2020-05-15 19:57:10,203 [476] [ERROR] [gunicorn.error] Socket error processing request.
Traceback (most recent call last):
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/base_async.py", line 66, in handle
    six.reraise(*sys.exc_info())
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
    addr)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/base_async.py", line 129, in handle_request
    six.reraise(*sys.exc_info())
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
    return self.app(environ, start_response)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/quay-registry/endpoints/decorators.py", line 57, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/auth/registry_jwt_auth.py", line 164, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/v2/__init__.py", line 126, in wrapped
    return func(namespace_name, repo_name, *args, **kwargs)
  File "/quay-registry/endpoints/decorators.py", line 120, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/decorators.py", line 145, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/v2/blob.py", line 233, in start_blob_upload
    blob_uploader = create_blob_upload(repository_ref, storage, _upload_settings())
  File "/quay-registry/data/registry_model/blobuploader.py", line 79, in create_blob_upload
    new_upload_uuid, upload_metadata = storage.initiate_chunked_upload(location_name)
  File "/quay-registry/storage/distributedstorage.py", line 27, in wrapper
    return storage_func(*args, **kwargs)
  File "/quay-registry/storage/local.py", line 91, in initiate_chunked_upload
    with open(self._init_path(self._rel_upload_path(new_uuid), create=True), "w"):
  File "/quay-registry/storage/local.py", line 26, in _init_path
    os.makedirs(dirname)
  File "/opt/rh/python27/root/usr/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/datastorage/registry/uploads'
gunicorn-registry stdout | 2020-05-15 19:57:10,205 [462] [ERROR] [gunicorn.error] Socket error processing request.
Traceback (most recent call last):
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/base_async.py", line 66, in handle
    six.reraise(*sys.exc_info())
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
    addr)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/base_async.py", line 129, in handle_request
    six.reraise(*sys.exc_info())
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
    return self.app(environ, start_response)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/quay-registry/endpoints/decorators.py", line 57, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/auth/registry_jwt_auth.py", line 164, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/v2/__init__.py", line 126, in wrapped
    return func(namespace_name, repo_name, *args, **kwargs)
  File "/quay-registry/endpoints/decorators.py", line 120, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/decorators.py", line 145, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/v2/blob.py", line 233, in start_blob_upload
    blob_uploader = create_blob_upload(repository_ref, storage, _upload_settings())
  File "/quay-registry/data/registry_model/blobuploader.py", line 79, in create_blob_upload
    new_upload_uuid, upload_metadata = storage.initiate_chunked_upload(location_name)
  File "/quay-registry/storage/distributedstorage.py", line 27, in wrapper
    return storage_func(*args, **kwargs)
  File "/quay-registry/storage/local.py", line 91, in initiate_chunked_upload
    with open(self._init_path(self._rel_upload_path(new_uuid), create=True), "w"):
  File "/quay-registry/storage/local.py", line 26, in _init_path
    os.makedirs(dirname)
  File "/opt/rh/python27/root/usr/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/datastorage/registry/uploads'

              rhn-coreos-amerdler Alec Merdler (Inactive)
              esauer@redhat.com Eric Sauer (Inactive)
              Dongbo Yan Dongbo Yan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: