Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1284

Quay login was failed with hit 500 error page when FIPS enabled

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • quay-v3.5.0
    • quay-v3.4.0
    • quay
    • 0

    Description

      Description:

      This is an issue found when deploy Quay with TNG Operator on OCP(enabled FIPS), after deployment complete successfully, login quay hit 500 error page, checked quay POD losg, get error message "*ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS", see logs attached.

      Note: This issue block testing quay V3.4.0 on OCP with FIPS enabled.

       

      Check OCP to make sure FIPS was enabled:

      [root@ip-10-0-1-60 centos]# oc get node
      NAME                                         STATUS                     ROLES    AGE     VERSION
      ip-10-0-137-104.us-east-2.compute.internal   Ready                      master   5h54m   v1.19.0+9f84db3
      ip-10-0-140-218.us-east-2.compute.internal   Ready                      worker   5h44m   v1.19.0+9f84db3
      ip-10-0-147-130.us-east-2.compute.internal   Ready                      worker   5h47m   v1.19.0+9f84db3
      ip-10-0-181-88.us-east-2.compute.internal    Ready                      worker   5h44m   v1.19.0+9f84db3
      ip-10-0-190-141.us-east-2.compute.internal   Ready                      master   5h54m   v1.19.0+9f84db3
      ip-10-0-191-137.us-east-2.compute.internal   Ready                      worker   5h44m   v1.19.0+9f84db3
      ip-10-0-192-189.us-east-2.compute.internal   Ready,SchedulingDisabled   master   5h54m   v1.19.0+9f84db3
      ip-10-0-198-131.us-east-2.compute.internal   Ready                      worker   5h45m   v1.19.0+9f84db3
      [root@ip-10-0-1-60 centos]# oc debug no/ip-10-0-140-218.us-east-2.compute.internal
      Starting pod/ip-10-0-140-218us-east-2computeinternal-debug ...
      To use host binaries, run `chroot /host`
      Pod IP: 10.0.140.218
      If you don't see a command prompt, try pressing enter.
      sh-4.4# chroot /host
      sh-4.4# fips-mode-setup --checkfips-mode-setup --check^C
      sh-4.4# fips-mode-setup --check
      FIPS mode is enabled.
      

      Quay Image:

      lizhang@lzha-mac quay3.4 % oc get pod
      NAME                                           READY   STATUS      RESTARTS   AGE
      quayfips-clair-54d55666f9-tb5cw                1/1     Running     0          8m36s
      quayfips-clair-postgres-55cfc4568b-q6dg9       1/1     Running     1          11m
      quayfips-quay-app-79557d4f89-b9qxp             1/1     Running     0          103s
      quayfips-quay-config-editor-549d85d975-jpcwb   1/1     Running     0          8m36s
      quayfips-quay-database-6c5cdf4986-5xmbl        1/1     Running     0          11m
      quayfips-quay-mirror-6757d68f45-kl6pl          1/1     Running     0          8m36s
      quayfips-quay-postgres-init-jbtvz              0/1     Completed   0          10m
      quayfips-quay-redis-568977d465-fd97c           1/1     Running     0          10m
      
      lizhang@lzha-mac quay3.4 % oc get pod quayfips-quay-app-79557d4f89-b9qxp -o json | jq '.spec.containers[0].image'
      "registry.redhat.io/quay/quay@sha256:12434420fa3ca1df4654bf6d08c8980b1209b13fbfe8336fd9b1ef6f04cc4c75"

       

      Quay POD Logs:

      1561 gunicorn-web stdout | 2020-11-13 09:06:14,567 [280] [ERROR] [gunicorn.error] Error handling request /api/v1/user/
      1562 gunicorn-web stdout | Traceback (most recent call last):
      1563 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
      1564 gunicorn-web stdout |     self.handle_request(listener_name, req, client, addr)
      1565 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
      1566 gunicorn-web stdout |     super().handle_request(listener_name, req, sock, addr)
      1567 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request
      1568 gunicorn-web stdout |     respiter = self.wsgi(environ, resp.start_response)
      1569 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2463, in __call__
      1570 gunicorn-web stdout |     return self.wsgi_app(environ, start_response)
      1571 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
      1572 gunicorn-web stdout |     return self.app(environ, start_response)
      1573 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app
      1574 gunicorn-web stdout |     response = self.handle_exception(e)
      1575 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 269, in error_router
      1576 gunicorn-web stdout |     return original_handler(e)
      1577 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1866, in handle_exception
      1578 gunicorn-web stdout |     reraise(exc_type, exc_value, tb)
      1579 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 38, in reraise
      1580 gunicorn-web stdout |     raise value.with_traceback(tb)
      1581 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
      1582 gunicorn-web stdout |     response = self.full_dispatch_request()
      1583 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request
      1584 gunicorn-web stdout |     rv = self.handle_user_exception(e)
      1585 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 269, in error_router
      1586 gunicorn-web stdout |     return original_handler(e)
      1587 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
      1588 gunicorn-web stdout |     reraise(exc_type, exc_value, tb)
      1589 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 38, in reraise
      1590 gunicorn-web stdout |     raise value.with_traceback(tb)
      1591 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
      1592 gunicorn-web stdout |     rv = self.dispatch_request()
      1593 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
      1594 gunicorn-web stdout |     return self.view_functions[rule.endpoint](**req.view_args)
      1595 gunicorn-web stdout |   File "/quay-registry/endpoints/decorators.py", line 197, in wrapper
      1596 gunicorn-web stdout |     return func(*args, **kwargs)
      1597 gunicorn-web stdout |   File "/quay-registry/auth/decorators.py", line 65, in wrapper
      1598 gunicorn-web stdout |     return func(*args, **kwargs)
      1599 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/utils/cors.py", line 35, in wrapped_function
      1600 gunicorn-web stdout |     resp = make_response(f(*args, **kwargs))
      1601 gunicorn-web stdout |   File "/quay-registry/endpoints/csrf.py", line 73, in wrapper
      1602 gunicorn-web stdout |     resp = func(*args, **kwargs)
      1603 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 458, in wrapper
      1604 gunicorn-web stdout |     resp = resource(*args, **kwargs)
      1605 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/views.py", line 89, in view
      1606 gunicorn-web stdout |     return self.dispatch_request(*args, **kwargs)
      1607 gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
      1608 gunicorn-web stdout |     resp = meth(*args, **kwargs)
      1609 gunicorn-web stdout |   File "/quay-registry/endpoints/decorators.py", line 145, in wrapper
      1610 gunicorn-web stdout |     return func(*args, **kwargs)
      1611 gunicorn-web stdout |   File "/quay-registry/endpoints/decorators.py", line 120, in wrapper
      1612 gunicorn-web stdout |     return func(*args, **kwargs)
      1613 gunicorn-web stdout |   File "/quay-registry/endpoints/api/__init__.py", line 416, in wrapped
      1614 gunicorn-web stdout |     return func(self, *args, **kwargs)
      1615 gunicorn-web stdout |   File "/quay-registry/endpoints/api/user.py", line 497, in post
      1616 gunicorn-web stdout |     return user_view(new_user), 200, headers
      1617 gunicorn-web stdout |   File "/quay-registry/endpoints/api/user.py", line 158, in user_view
      1618 gunicorn-web stdout |     "avatar": avatar.get_data_for_user(user),
      1619 gunicorn-web stdout |   File "/quay-registry/avatars/avatars.py", line 85, in get_data_for_user
      1620 gunicorn-web stdout |     return self.get_data(user.username, user.email, "robot" if user.robot else "user")
      1621 gunicorn-web stdout |   File "/quay-registry/avatars/avatars.py", line 110, in get_data
      1622 gunicorn-web stdout |     hash_value = hashlib.md5(username_email_or_id.strip().lower().encode("utf-8")).hexdigest()
      1623 gunicorn-web stdout | ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS
      1624 gunicorn-web stdout | 2020-11-13 09:06:14,570 [280] [INFO] [gunicorn.access]  - - [13/Nov/2020:09:06:14 +0000] "POST /api/v1/user/ HTTP/1.0" 500 0 "-" "-     "
      1625 nginx stdout | 10.131.0.8 () - - [13/Nov/2020:09:06:14 +0000] "POST /api/v1/user/ HTTP/2.0" 500 141 "https://quayfips-quay-quay.apps.lzha1115.qe.devclust     er.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36" (0.348 166     7 0.348)
      

      Steps:

      1. On OCP 4.6 with FIPS enable ,deploy Quay with TNG Operator with using AWS S3 as external registry storage
      2. Create new Quay super user
      3. Login quay with super user

      Expected Results:

      Login Quay successfully with super user.

      Actual Results:

      Login quay was failed and hit quay 500 error page. 

      Attachments

        Activity

          People

            jonathankingfc Jonathan King
            lzha1981 luffy zhang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: