-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
quay-v3.12.1, quay-v3.12.2
-
False
-
-
False
-
-
Description:
This is an issue found in Quay 3.12.2, when Quay is deployed on OCP Cluster with FIPS enabled, also choose to use Azure Blob Storage, push image was failed with 500 error code "ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS", pls review this issue, attached logs quay3122_app_pod_azure_blob_storage.logs
Quay: quay-operator-bundle-container-v3.12.2-3
OCP Cluster: 4.16 with FIPS enabled
Azure Blob Storage account with encryption enabled.
Note: this issue is also existed in Quay 3.12
gunicorn-registry stdout | 2024-08-30 04:14:16,402 [282] [ERROR] [gunicorn.error] Error handling request /v2/quayorg134708051329153/imagerepo134708051329153/blobs/uploads/d8a9589c-8fee-4dad-ba33-965d40bdfc5d gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle gunicorn-registry stdout | self.handle_request(listener_name, req, client, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 128, in handle_request gunicorn-registry stdout | super().handle_request(listener_name, req, sock, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request gunicorn-registry stdout | respiter = self.wsgi(environ, resp.start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2213, in __call__ gunicorn-registry stdout | return self.wsgi_app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 182, in __call__ gunicorn-registry stdout | return self.app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2193, in wsgi_app gunicorn-registry stdout | response = self.handle_exception(e) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2190, in wsgi_app gunicorn-registry stdout | response = self.full_dispatch_request() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1486, in full_dispatch_request gunicorn-registry stdout | rv = self.handle_user_exception(e) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request gunicorn-registry stdout | rv = self.dispatch_request() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request gunicorn-registry stdout | return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 228, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 91, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/auth/registry_jwt_auth.py", line 175, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/__init__.py", line 216, in wrapped gunicorn-registry stdout | return func(namespace_name, repo_name, *args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 164, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 189, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/blob.py", line 376, in upload_chunk gunicorn-registry stdout | _upload_chunk(uploader) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/blob.py", line 555, in _upload_chunk gunicorn-registry stdout | blob_uploader.upload_chunk(app.config, input_fp, start_offset, length) gunicorn-registry stdout | File "/quay-registry/data/registry_model/blobuploader.py", line 222, in upload_chunk gunicorn-registry stdout | length_written, new_metadata, upload_error = self.storage.stream_upload_chunk( gunicorn-registry stdout | File "/quay-registry/storage/distributedstorage.py", line 26, in wrapper gunicorn-registry stdout | return storage_func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/storage/azurestorage.py", line 265, in stream_upload_chunk gunicorn-registry stdout | self._blob(upload_blob_path).stage_block(block_id, buf, validate_content=True) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/storage/blob/_blob_client.py", line 1977, in stage_block gunicorn-registry stdout | return self._client.block_blob.stage_block(**options) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/storage/blob/_generated/operations/_block_blob_operations.py", line 344, in stage_block gunicorn-registry stdout | pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 211, in run gunicorn-registry stdout | return first_node.send(pipeline_request) # type: ignore gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 71, in send gunicorn-registry stdout | response = self.next.send(request) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 71, in send gunicorn-registry stdout | response = self.next.send(request) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 71, in send gunicorn-registry stdout | response = self.next.send(request) gunicorn-registry stdout | [Previous line repeated 1 more time] gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 69, in send gunicorn-registry stdout | _await_result(self._policy.on_request, request) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/core/pipeline/_tools.py", line 34, in await_result gunicorn-registry stdout | result = func(*args, **kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/storage/blob/_shared/policies.py", line 349, in on_request gunicorn-registry stdout | computed_md5 = encode_base64(StorageContentValidation.get_content_md5(request.http_request.data)) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/azure/storage/blob/_shared/policies.py", line 325, in get_content_md5 gunicorn-registry stdout | md5 = hashlib.md5() # nosec gunicorn-registry stdout | ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS gunicorn-registry stdout | 2024-08-30 04:14:16,405 [282] [INFO] [gunicorn.access] - - [30/Aug/2024:04:14:16 +0000] "PATCH /v2/quayorg134708051329153/imagerepo134708051329153/blobs/uploads/d8a9589c-8fee-4dad-ba33-965d40bdfc5d HTTP/1.1" 500 0 "-" "-" nginx stdout | 10.131.0.26 (-) - - [30/Aug/2024:04:14:16 +0000] "PATCH /v2/quayorg134708051329153/imagerepo134708051329153/blobs/uploads/d8a9589c-8fee-4dad-ba33-965d40bdfc5d HTTP/1.1" 500 141 "-" "skopeo/1.13.3" (5.213 134412734 5.203)
Quay Config.yaml:
PERMANENTLY_DELETE_TAGS: true RESET_CHILD_MANIFEST_EXPIRATION: true FEATURE_EXTENDED_REPOSITORY_NAMES: true CREATE_NAMESPACE_ON_PUSH: true FEATURE_QUOTA_MANAGEMENT: true FEATURE_PROXY_CACHE: true FEATURE_USER_INITIALIZE: true FEATURE_PROXY_STORAGE: true IGNORE_UNKNOWN_MEDIATYPES: true FEATURE_UI_V2: true FEATURE_SUPERUSERS_FULL_ACCESS: true FEATURE_AUTO_PRUNE: true FEATURE_FIPS: true SUPER_USERS: - quay - admin GLOBAL_READONLY_SUPER_USERS: - superglobalro DISTRIBUTED_STORAGE_CONFIG: default: - AzureStorage - azure_account_key: ****** azure_account_name: quayazure3000 azure_container: quayazure3000 sas_token: ****** storage_path: /quayazuredata/quayregistry DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: - default DISTRIBUTED_STORAGE_PREFERENCE: - default
Azure Blob Storage account:
Azure Blob Storage account with encryption enabled:
