Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1806

Quay push image get 500 error code when use azure blob storage

XMLWordPrintable

    • False
    • False
    • Undefined

      Description:

      This is an issue found when deploy quay to use Azure Blob storage as quay backend registry, when push image, get 500 internal error, check quay app pod logs, get error "ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS", see attached quay app pod logs.

      Note: the quay was deployed on OCP with FIPS enable.

      docker push quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/ldapteam/azure
      Using default tag: latest
      The push refers to repository [quayregistry-quay-quay-enterprise.apps.quay-fips-610.qe.devcluster.openshift.com/ldapteam/azure]
      2653d992f4ef: Pushing [==================================================>]  216.5MB
      received unexpected HTTP status: 500 Internal Server Error
      

      Quay App Pod logs:

      nginx stdout | 10.129.2.46 () - - [29/Mar/2021:07:02:10 +0000] "POST /v2/ldapteam/azure/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \x5C(darwin\x5C))" (0.017 1488 0.018)
      gunicorn-registry stdout | 2021-03-29 07:02:10,121 [257] [INFO] [gunicorn.access] 10.129.2.46 - - [29/Mar/2021:07:02:10 +0000] "POST /v2/ldapteam/azure/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \(darwin\))"
      gunicorn-registry stdout | 2021-03-29 07:02:11,424 [263] [ERROR] [gunicorn.error] Error handling request /v2/ldapteam/azure/blobs/uploads/2d0072d1-ddd2-4f52-b7e6-7ebe7ff46f7e
      gunicorn-registry stdout | Traceback (most recent call last):
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
      gunicorn-registry stdout |     self.handle_request(listener_name, req, client, addr)
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
      gunicorn-registry stdout |     super().handle_request(listener_name, req, sock, addr)
      gunicorn-registry stdout |   [Previous line repeated 1 more time]
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
      gunicorn-registry stdout |     _await_result(self._policy.on_request, request)
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_tools.py", line 29, in await_result
      gunicorn-registry stdout |     result = func(*args, **kwargs)
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/azure/storage/blob/_shared/policies.py", line 349, in on_request
      gunicorn-registry stdout |     computed_md5 = encode_base64(StorageContentValidation.get_content_md5(request.http_request.data))
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/azure/storage/blob/_shared/policies.py", line 325, in get_content_md5
      gunicorn-registry stdout |     md5 = hashlib.md5() # nosec
      gunicorn-registry stdout | ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS
      gunicorn-registry stdout | 2021-03-29 07:02:11,428 [263] [INFO] [gunicorn.access]  - - [29/Mar/2021:07:02:11 +0000] "PATCH /v2/ldapteam/azure/blobs/uploads/2d0072d1-ddd2-4f52-b7e6-7ebe7ff46f7e HTTP/1.1" 500 0 "-" "-"
      

       Quay Version:

      oc get pod
      NAME                                               READY   STATUS      RESTARTS   AGE
      quay-operator.v3.5.0-7489b8c4f-r4r6j               1/1     Running     0          94m
      quayregistry-clair-app-574699545c-54zvk            1/1     Running     0          12m
      quayregistry-clair-postgres-64b54bbdd6-zkhfz       1/1     Running     0          11m
      quayregistry-quay-app-696f67485b-wjv9t             1/1     Running     3          12m
      quayregistry-quay-config-editor-6559b644c4-k2nmn   1/1     Running     0          12m
      quayregistry-quay-database-6bb7c58f9b-djwcn        1/1     Running     0          11m
      quayregistry-quay-mirror-74ffcb868b-xbrt7          1/1     Running     1          11m
      quayregistry-quay-postgres-init-n4xsg              0/1     Completed   0          12m
      quayregistry-quay-redis-f5ff84947-rg676            1/1     Running     0          12m
      
      oc get pod quayregistry-quay-app-696f67485b-wjv9t -o json | jq '.spec.containers[0].image'
      "registry.redhat.io/quay/quay-rhel8@sha256:0444c7b452a14e0c87ee56f9aa72c54484333c38b0a95de9a4f11f6177273f26"

      Steps:

      1. Deploy Quay 3.5.0 Operator to all OCP namespace
      2. Deploy quay with quay 3.5 Operator with using Azure Blob Storage as backend registry storage
      3. Create new org and image repository
      4. Push new image to above new image repository

      Expected Results:

      Push image complete successfully.

      Actual Results:

      Push image was failed with 500 error, report "ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS"

              jonathankingfc Jonathan King
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: