Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-7743

Quay 3.12 push image was failed when using Azure Blob Storage in FIPS Mode

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • quay-v3.12.1, quay-v3.12.2
    • quay
    • False
    • Hide

      None

      Show
      None
    • False

      Description:

      This is an issue found in Quay 3.12.2, when Quay is deployed on OCP Cluster with FIPS enabled, also choose to use Azure Blob Storage, push image was failed with 500 error code "ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS", pls review this issue, attached logs quay3122_app_pod_azure_blob_storage.logs 

      Quay: quay-operator-bundle-container-v3.12.2-3

      OCP Cluster: 4.16 with FIPS enabled

      Azure Blob Storage account with encryption enabled.

      Note: this issue is also existed in Quay 3.12

      gunicorn-registry stdout | 2024-08-30 04:14:16,402 [282] [ERROR] [gunicorn.error] Error handling request /v2/quayorg134708051329153/imagerepo134708051329153/blobs/uploads/d8a9589c-8fee-4dad-ba33-965d40bdfc5d
      gunicorn-registry stdout | Traceback (most recent call last):
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle
      gunicorn-registry stdout |     self.handle_request(listener_name, req, client, addr)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 128, in handle_request
      gunicorn-registry stdout |     super().handle_request(listener_name, req, sock, addr)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request
      gunicorn-registry stdout |     respiter = self.wsgi(environ, resp.start_response)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 2213, in __call__
      gunicorn-registry stdout |     return self.wsgi_app(environ, start_response)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 182, in __call__
      gunicorn-registry stdout |     return self.app(environ, start_response)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 2193, in wsgi_app
      gunicorn-registry stdout |     response = self.handle_exception(e)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 2190, in wsgi_app
      gunicorn-registry stdout |     response = self.full_dispatch_request()
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 1486, in full_dispatch_request
      gunicorn-registry stdout |     rv = self.handle_user_exception(e)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request
      gunicorn-registry stdout |     rv = self.dispatch_request()
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request
      gunicorn-registry stdout |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 228, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 91, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/auth/registry_jwt_auth.py", line 175, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/__init__.py", line 216, in wrapped
      gunicorn-registry stdout |     return func(namespace_name, repo_name, *args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 164, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 189, in wrapper
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/blob.py", line 376, in upload_chunk
      gunicorn-registry stdout |     _upload_chunk(uploader)
      gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/blob.py", line 555, in _upload_chunk
      gunicorn-registry stdout |     blob_uploader.upload_chunk(app.config, input_fp, start_offset, length)
      gunicorn-registry stdout |   File "/quay-registry/data/registry_model/blobuploader.py", line 222, in upload_chunk
      gunicorn-registry stdout |     length_written, new_metadata, upload_error = self.storage.stream_upload_chunk(
      gunicorn-registry stdout |   File "/quay-registry/storage/distributedstorage.py", line 26, in wrapper
      gunicorn-registry stdout |     return storage_func(*args, **kwargs)
      gunicorn-registry stdout |   File "/quay-registry/storage/azurestorage.py", line 265, in stream_upload_chunk
      gunicorn-registry stdout |     self._blob(upload_blob_path).stage_block(block_id, buf, validate_content=True)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
      gunicorn-registry stdout |     return func(*args, **kwargs)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/storage/blob/_blob_client.py", line 1977, in stage_block
      gunicorn-registry stdout |     return self._client.block_blob.stage_block(**options)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/storage/blob/_generated/operations/_block_blob_operations.py", line 344, in stage_block
      gunicorn-registry stdout |     pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 211, in run
      gunicorn-registry stdout |     return first_node.send(pipeline_request)  # type: ignore
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 71, in send
      gunicorn-registry stdout |     response = self.next.send(request)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 71, in send
      gunicorn-registry stdout |     response = self.next.send(request)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 71, in send
      gunicorn-registry stdout |     response = self.next.send(request)
      gunicorn-registry stdout |   [Previous line repeated 1 more time]
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 69, in send
      gunicorn-registry stdout |     _await_result(self._policy.on_request, request)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/core/pipeline/_tools.py", line 34, in await_result
      gunicorn-registry stdout |     result = func(*args, **kwargs)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/storage/blob/_shared/policies.py", line 349, in on_request
      gunicorn-registry stdout |     computed_md5 = encode_base64(StorageContentValidation.get_content_md5(request.http_request.data))
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/azure/storage/blob/_shared/policies.py", line 325, in get_content_md5
      gunicorn-registry stdout |     md5 = hashlib.md5() # nosec
      gunicorn-registry stdout | ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS
      gunicorn-registry stdout | 2024-08-30 04:14:16,405 [282] [INFO] [gunicorn.access]  - - [30/Aug/2024:04:14:16 +0000] "PATCH /v2/quayorg134708051329153/imagerepo134708051329153/blobs/uploads/d8a9589c-8fee-4dad-ba33-965d40bdfc5d HTTP/1.1" 500 0 "-" "-"
      nginx stdout | 10.131.0.26 (-) - - [30/Aug/2024:04:14:16 +0000] "PATCH /v2/quayorg134708051329153/imagerepo134708051329153/blobs/uploads/d8a9589c-8fee-4dad-ba33-965d40bdfc5d HTTP/1.1" 500 141 "-" "skopeo/1.13.3" (5.213 134412734 5.203) 

      Quay Config.yaml:

      PERMANENTLY_DELETE_TAGS: true
      RESET_CHILD_MANIFEST_EXPIRATION: true
      FEATURE_EXTENDED_REPOSITORY_NAMES: true
      CREATE_NAMESPACE_ON_PUSH: true
      FEATURE_QUOTA_MANAGEMENT: true
      FEATURE_PROXY_CACHE: true
      FEATURE_USER_INITIALIZE: true
      FEATURE_PROXY_STORAGE: true
      IGNORE_UNKNOWN_MEDIATYPES: true
      FEATURE_UI_V2: true
      FEATURE_SUPERUSERS_FULL_ACCESS: true
      FEATURE_AUTO_PRUNE: true
      FEATURE_FIPS: true
      SUPER_USERS:
        - quay
        - admin
      GLOBAL_READONLY_SUPER_USERS:
        - superglobalro
      DISTRIBUTED_STORAGE_CONFIG:
          default:
            - AzureStorage
            - azure_account_key: ******
              azure_account_name: quayazure3000
              azure_container: quayazure3000
              sas_token: ******
              storage_path: /quayazuredata/quayregistry
      DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
        - default
      DISTRIBUTED_STORAGE_PREFERENCE:
        - default 

      Azure Blob Storage account:

      Azure Blob Storage account with encryption enabled:

              Unassigned Unassigned
              lzha1981 luffy zhang
              Votes:
              11 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: