Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-8140

Failed to push image when OMR 2.0.2 installed on RHEL9

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • omr-v2.0.2
    • OMR
    • Critical

      Description

      After installed OMR 2.0.2 on RHEL9, failed to push image

      Build

      registry-proxy.engineering.redhat.com/rh-osbs/openshift-mirror-registry-rhel8:v2.0.2-5

      ENV

      $ podman version
      Client:       Podman Engine
      Version:      4.6.1
      API Version:  4.6.1
      Go Version:   go1.20.6
      Built:        Fri Aug 25 07:08:17 2023
      OS/Arch:      linux/amd64

      $ uname -a 
      Linux quay-omr-rhel-60 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 3 11:12:36 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

      $ cat /etc/redhat-release 
      Red Hat Enterprise Linux release 9.3 (Plow)

      Reproduced steps

      1. Using OMR 2.0 setup quay

      ./mirror-registry install --initUser quay --initPassword password --quayRoot /home/cloud-user/omrtest/config --sqliteStorage /home/cloud-user/omrtest/database --quayStorage /home/cloud-user/omrtest/storage --quayHostname 10.0.0.0 --sslCert ssl.cert --sslKey ssl.key -v 

      2. Push image to quay

      Actual results

      Quay is ready, but push failed with error:

       

      $ podman push --creds=quay:password --tls-verify=false 10.0.0.0:8443/rhel8/redis:6
      Getting image source signatures
      Copying blob 0ee2b26391ef [--------------------------------------] 8.0b / 253.3MiB
      Error: writing blob: initiating layer upload to /v2/rhel8/redis/blobs/uploads/ in 10.0.110.227:8443: received unexpected HTTP status: 502 Bad Gateway

       

      Quay registry logs:

       

      nginx stdout | 2024/10/23 02:15:17 [error] 94#0: *84 upstream prematurely closed connection while reading response header from upstream, client: 10.0.2.100, server: _, request: "POST /v2/quay/httpd/blobs/uploads/ HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/quay/httpd/blobs/uploads/", host: "10.0.110.227:8443"
      nginx stdout | 10.0.2.100 (-) - - [23/Oct/2024:02:15:17 +0000] "POST /v2/quay/httpd/blobs/uploads/ HTTP/1.1" 502 289 "-" "skopeo/1.13.3" (0.011 1190 0.010 : 0.002)
      gunicorn-web stdout | 2024-10-23 02:15:17,370 [226] [INFO] [gunicorn.access] 10.0.2.100 - - [23/Oct/2024:02:15:17 +0000] "GET /quay-registry/static/502.html HTTP/1.0" 308 289 "-" "skopeo/1.13.3"
      gunicorn-registry stdout | 2024-10-23 02:15:17,373 [231] [ERROR] [gunicorn.error] Socket error processing request.
      gunicorn-registry stdout | Traceback (most recent call last):
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 65, in handle
      gunicorn-registry stdout |     util.reraise(*sys.exc_info())
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/util.py", line 641, in reraise
      gunicorn-registry stdout |     raise value
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle
      gunicorn-registry stdout |     self.handle_request(listener_name, req, client, addr)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 128, in handle_request
      gunicorn-registry stdout |     super().handle_request(listener_name, req, sock, addr)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 130, in handle_request
      gunicorn-registry stdout |     util.reraise(*sys.exc_info())
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/util.py", line 641, in reraise
      gunicorn-registry stdout |     raise value
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request
      gunicorn-registry stdout |     respiter = self.wsgi(environ, resp.start_response)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 2213, in __call__
      gunicorn-registry stdout |     return self.wsgi_app(environ, start_response)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 182, in __call__
      gunicorn-registry stdout |     return self.app(environ, start_response)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 2193, in wsgi_app
      gunicorn-registry stdout |     response = self.handle_exception(e)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 2190, in wsgi_app
      gunicorn-registry stdout |     response = self.full_dispatch_request()
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 1486, in full_dispatch_request
      gunicorn-registry stdout |     rv = self.handle_user_exception(e)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request
      gunicorn-registry stdout |     rv = self.dispatch_request()
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request
      gunicorn-registry stdout |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 228, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 91, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/auth/registry_jwt_auth.py", line 175, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/api/__init__.py", line 416, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/__init__.py", line 216, in wrapped
      gunicorn-registry stdout |     return func(namespace_name, repo_name, *args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 164, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 189, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/blob.py", line 277, in start_blob_upload
      gunicorn-registry stdout |     blob_uploader = create_blob_upload(repository_ref, storage, _upload_settings())
      gunicorn-registry stdout |   File "/quay-registry/data/registry_model/blobuploader.py", line 74, in create_blob_upload
      gunicorn-registry stdout |     new_upload_uuid, upload_metadata = storage.initiate_chunked_upload(location_name)
      gunicorn-registry stdout |   File "/quay-registry/storage/distributedstorage.py", line 26, in wrapper
      gunicorn-registry stdout |     return storage_func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/storage/local.py", line 90, in initiate_chunked_upload
      gunicorn-registry stdout |     with open(self._init_path(self._rel_upload_path(new_uuid), create=True), "wb"):
      gunicorn-registry stdout |   File "/quay-registry/storage/local.py", line 25, in _init_path
      gunicorn-registry stdout |     os.makedirs(dirname)
      gunicorn-registry stdout |   File "/usr/lib64/python3.9/os.py", line 225, in makedirs
      gunicorn-registry stdout |     mkdir(name, mode)
      gunicorn-registry stdout | PermissionError: [Errno 13] Permission denied: '/datastorage/uploads'
      nginx stdout | 2024/10/23 02:15:17 [error] 94#0: *86 upstream prematurely closed connection while reading response header from upstream, client: 10.0.2.100, server: _, request: "POST /v2/quay/httpd/blobs/uploads/ HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/quay/httpd/blobs/uploads/", host: "10.0.110.227:8443"
      nginx stdout | 10.0.2.100 (-) - - [23/Oct/2024:02:15:17 +0000] "POST /v2/quay/httpd/blobs/uploads/ HTTP/1.1" 502 289 "-" "skopeo/1.13.3" (0.010 1190 0.008 : 0.001)
      gunicorn-web stdout | 2024-10-23 02:15:17,375 [226] [INFO] [gunicorn.access] 10.0.2.100 - - [23/Oct/2024:02:15:17 +0000] "GET /quay-registry/static/502.html HTTP/1.0" 308 289 "-" "skopeo/1.13.3"
      notificationworker stdout | 2024-10-23 02:15:22,123 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-10-23 02:15:32 UTC)" (scheduled at 2024-10-23 02:15:22.123356+00:00)
      notificationworker stdout | 2024-10-23 02:15:22,129 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-10-23 02:15:32 UTC)" executed successfully
      notificationworker stdout | 2024-10-23 02:15:24,126 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-10-23 02:16:24 UTC)" (scheduled at 2024-10-23 02:15:24.126055+00:00)
      notificationworker stdout | 2024-10-23 02:15:24,126 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-10-23 02:16:24 UTC)" executed successfully
      gcworker stdout | 2024-10-23 02:15:24,320 [57] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2024-10-23 02:15:54 UTC)" (scheduled at 2024-10-23 02:15:24.319990+00:00)
      gcworker stdout | 2024-10-23 02:15:24,324 [57] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2024-10-23 02:15:54 UTC)" executed successfully
      exportactionlogsworker stdout | 2024-10-23 02:15:25,937 [56] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-10-23 02:16:25 UTC)" (scheduled at 2024-10-23 02:15:25.937105+00:00)

       

      Additional info
      1. The custom path for flag --quayStorage does not exist before OMR installation, it is created by OMR actually.
      2. The path status:
        1. $ ls -lrt omrtest/
          total 0
          drwxrwxr-x+ 2 cloud-user cloud-user  6 Oct 22 22:11 storage
          drwxr-xr-x. 3 cloud-user cloud-user 25 Oct 22 22:11 config
          drwxrwxr-x+ 2 cloud-user cloud-user 28 Oct 22 22:20 database
          
          $ ls -al omrtest/
          total 4
          drwxr-xr-x. 5 cloud-user cloud-user   51 Oct 22 22:11 .
          drwx------. 7 cloud-user cloud-user 4096 Oct 22 22:11 ..
          drwxr-xr-x. 3 cloud-user cloud-user   25 Oct 22 22:11 config
          drwxrwxr-x+ 2 cloud-user cloud-user   28 Oct 22 22:57 database
          drwxrwxr-x+ 2 cloud-user cloud-user    6 Oct 22 22:11 storage 

      3.When install OMR on RHEL8, does not meet this issue, pushing image is success

        1. screenshot-1.png
          screenshot-1.png
          142 kB
        2. screenshot-2.png
          screenshot-2.png
          133 kB
        3. screenshot-3.png
          screenshot-3.png
          106 kB
        4. screenshot-4.png
          screenshot-4.png
          30 kB

              hgovinda Harish Govindarajulu
              rhn-support-dyan Dongbo Yan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: