-
Bug
-
Resolution: Done
-
Blocker
-
None
-
omr-v2.0.2
Description
After installed OMR 2.0.2 on RHEL9, failed to push image
Build
registry-proxy.engineering.redhat.com/rh-osbs/openshift-mirror-registry-rhel8:v2.0.2-5
ENV
$ podman version
Client: Podman Engine
Version: 4.6.1
API Version: 4.6.1
Go Version: go1.20.6
Built: Fri Aug 25 07:08:17 2023
OS/Arch: linux/amd64
$ uname -a
Linux quay-omr-rhel-60 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 3 11:12:36 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux release 9.3 (Plow)
Reproduced steps
1. Using OMR 2.0 setup quay
./mirror-registry install --initUser quay --initPassword password --quayRoot /home/cloud-user/omrtest/config --sqliteStorage /home/cloud-user/omrtest/database --quayStorage /home/cloud-user/omrtest/storage --quayHostname 10.0.0.0 --sslCert ssl.cert --sslKey ssl.key -v
2. Push image to quay
Actual results
Quay is ready, but push failed with error:
$ podman push --creds=quay:password --tls-verify=false 10.0.0.0:8443/rhel8/redis:6 Getting image source signatures Copying blob 0ee2b26391ef [--------------------------------------] 8.0b / 253.3MiB Error: writing blob: initiating layer upload to /v2/rhel8/redis/blobs/uploads/ in 10.0.110.227:8443: received unexpected HTTP status: 502 Bad Gateway
Quay registry logs:
nginx stdout | 2024/10/23 02:15:17 [error] 94#0: *84 upstream prematurely closed connection while reading response header from upstream, client: 10.0.2.100, server: _, request: "POST /v2/quay/httpd/blobs/uploads/ HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/quay/httpd/blobs/uploads/", host: "10.0.110.227:8443" nginx stdout | 10.0.2.100 (-) - - [23/Oct/2024:02:15:17 +0000] "POST /v2/quay/httpd/blobs/uploads/ HTTP/1.1" 502 289 "-" "skopeo/1.13.3" (0.011 1190 0.010 : 0.002) gunicorn-web stdout | 2024-10-23 02:15:17,370 [226] [INFO] [gunicorn.access] 10.0.2.100 - - [23/Oct/2024:02:15:17 +0000] "GET /quay-registry/static/502.html HTTP/1.0" 308 289 "-" "skopeo/1.13.3" gunicorn-registry stdout | 2024-10-23 02:15:17,373 [231] [ERROR] [gunicorn.error] Socket error processing request. gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 65, in handle gunicorn-registry stdout | util.reraise(*sys.exc_info()) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/util.py", line 641, in reraise gunicorn-registry stdout | raise value gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle gunicorn-registry stdout | self.handle_request(listener_name, req, client, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 128, in handle_request gunicorn-registry stdout | super().handle_request(listener_name, req, sock, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 130, in handle_request gunicorn-registry stdout | util.reraise(*sys.exc_info()) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/util.py", line 641, in reraise gunicorn-registry stdout | raise value gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request gunicorn-registry stdout | respiter = self.wsgi(environ, resp.start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2213, in __call__ gunicorn-registry stdout | return self.wsgi_app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 182, in __call__ gunicorn-registry stdout | return self.app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2193, in wsgi_app gunicorn-registry stdout | response = self.handle_exception(e) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2190, in wsgi_app gunicorn-registry stdout | response = self.full_dispatch_request() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1486, in full_dispatch_request gunicorn-registry stdout | rv = self.handle_user_exception(e) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request gunicorn-registry stdout | rv = self.dispatch_request() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request gunicorn-registry stdout | return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 228, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 91, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/auth/registry_jwt_auth.py", line 175, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/api/__init__.py", line 416, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/__init__.py", line 216, in wrapped gunicorn-registry stdout | return func(namespace_name, repo_name, *args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 164, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 189, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/blob.py", line 277, in start_blob_upload gunicorn-registry stdout | blob_uploader = create_blob_upload(repository_ref, storage, _upload_settings()) gunicorn-registry stdout | File "/quay-registry/data/registry_model/blobuploader.py", line 74, in create_blob_upload gunicorn-registry stdout | new_upload_uuid, upload_metadata = storage.initiate_chunked_upload(location_name) gunicorn-registry stdout | File "/quay-registry/storage/distributedstorage.py", line 26, in wrapper gunicorn-registry stdout | return storage_func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/storage/local.py", line 90, in initiate_chunked_upload gunicorn-registry stdout | with open(self._init_path(self._rel_upload_path(new_uuid), create=True), "wb"): gunicorn-registry stdout | File "/quay-registry/storage/local.py", line 25, in _init_path gunicorn-registry stdout | os.makedirs(dirname) gunicorn-registry stdout | File "/usr/lib64/python3.9/os.py", line 225, in makedirs gunicorn-registry stdout | mkdir(name, mode) gunicorn-registry stdout | PermissionError: [Errno 13] Permission denied: '/datastorage/uploads' nginx stdout | 2024/10/23 02:15:17 [error] 94#0: *86 upstream prematurely closed connection while reading response header from upstream, client: 10.0.2.100, server: _, request: "POST /v2/quay/httpd/blobs/uploads/ HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/quay/httpd/blobs/uploads/", host: "10.0.110.227:8443" nginx stdout | 10.0.2.100 (-) - - [23/Oct/2024:02:15:17 +0000] "POST /v2/quay/httpd/blobs/uploads/ HTTP/1.1" 502 289 "-" "skopeo/1.13.3" (0.010 1190 0.008 : 0.001) gunicorn-web stdout | 2024-10-23 02:15:17,375 [226] [INFO] [gunicorn.access] 10.0.2.100 - - [23/Oct/2024:02:15:17 +0000] "GET /quay-registry/static/502.html HTTP/1.0" 308 289 "-" "skopeo/1.13.3" notificationworker stdout | 2024-10-23 02:15:22,123 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-10-23 02:15:32 UTC)" (scheduled at 2024-10-23 02:15:22.123356+00:00) notificationworker stdout | 2024-10-23 02:15:22,129 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-10-23 02:15:32 UTC)" executed successfully notificationworker stdout | 2024-10-23 02:15:24,126 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-10-23 02:16:24 UTC)" (scheduled at 2024-10-23 02:15:24.126055+00:00) notificationworker stdout | 2024-10-23 02:15:24,126 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-10-23 02:16:24 UTC)" executed successfully gcworker stdout | 2024-10-23 02:15:24,320 [57] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2024-10-23 02:15:54 UTC)" (scheduled at 2024-10-23 02:15:24.319990+00:00) gcworker stdout | 2024-10-23 02:15:24,324 [57] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2024-10-23 02:15:54 UTC)" executed successfully exportactionlogsworker stdout | 2024-10-23 02:15:25,937 [56] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-10-23 02:16:25 UTC)" (scheduled at 2024-10-23 02:15:25.937105+00:00)
Additional info
- The custom path for flag --quayStorage does not exist before OMR installation, it is created by OMR actually.
- The path status:
$ ls -lrt omrtest/ total 0 drwxrwxr-x+ 2 cloud-user cloud-user 6 Oct 22 22:11 storage drwxr-xr-x. 3 cloud-user cloud-user 25 Oct 22 22:11 config drwxrwxr-x+ 2 cloud-user cloud-user 28 Oct 22 22:20 database $ ls -al omrtest/ total 4 drwxr-xr-x. 5 cloud-user cloud-user 51 Oct 22 22:11 . drwx------. 7 cloud-user cloud-user 4096 Oct 22 22:11 .. drwxr-xr-x. 3 cloud-user cloud-user 25 Oct 22 22:11 config drwxrwxr-x+ 2 cloud-user cloud-user 28 Oct 22 22:57 database drwxrwxr-x+ 2 cloud-user cloud-user 6 Oct 22 22:11 storage
3.When install OMR on RHEL8, does not meet this issue, pushing image is success