Loading...

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: quay-v3.4.2, quay-v3.5.0
Affects Version/s: quay-v3.4.0
Component/s: quay-operator
Labels:
- triage

Blocked:
False
Ready:
False
Product:
Quay Enterprise
Release Note Text:
Undefined
Git Pull Request:
https://github.com/quay/quay-operator/pull/397
Market:

RICE Score:
0

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Passwords saved in the repomirror configuration cannot be decoded after complete Quay restart (for example, after MCO updates worker nodes). The following stack trace appers in the logs:

gunicorn-web stdout | 2021-02-10 13:51:57,967 [300] [ERROR] [data.encryption] Got exception when trying to decrypt value `iK1I2kkOEesFY/m32sWLS8wVDT0jhVRBYiGhNI0Zr2cCTWLh69w=`
gunicorn-web stdout | Traceback (most recent call last):
gunicorn-web stdout |   File "/quay-registry/data/encryption.py", line 44, in _decrypt_ccm
gunicorn-web stdout |     decrypted = aesccm.decrypt(nonce, ct, None)
gunicorn-web stdout |   File "/usr/local/lib64/python3.8/site-packages/cryptography/hazmat/primitives/ciphers/aead.py", line 120, in decrypt
gunicorn-web stdout |     return aead._decrypt(
gunicorn-web stdout |   File "/usr/local/lib64/python3.8/site-packages/cryptography/hazmat/backends/openssl/aead.py", line 151, in _decrypt
gunicorn-web stdout |     raise InvalidTag
gunicorn-web stdout | cryptography.exceptions.InvalidTag
gunicorn-web stdout | 2021-02-10 13:51:57,969 [300] [ERROR] [gunicorn.error] Error handling request /api/v1/repository/ibazulic/mirror/mirror
gunicorn-web stdout | Traceback (most recent call last):
gunicorn-web stdout |   File "/quay-registry/data/encryption.py", line 44, in _decrypt_ccm
gunicorn-web stdout |     decrypted = aesccm.decrypt(nonce, ct, None)
gunicorn-web stdout |   File "/usr/local/lib64/python3.8/site-packages/cryptography/hazmat/primitives/ciphers/aead.py", line 120, in decrypt
gunicorn-web stdout |     return aead._decrypt(
gunicorn-web stdout |   File "/usr/local/lib64/python3.8/site-packages/cryptography/hazmat/backends/openssl/aead.py", line 151, in _decrypt
gunicorn-web stdout |     raise InvalidTag
gunicorn-web stdout | cryptography.exceptions.InvalidTag
gunicorn-web stdout | During handling of the above exception, another exception occurred:
gunicorn-web stdout | Traceback (most recent call last):
gunicorn-web stdout |   File "/quay-registry/endpoints/api/mirror.py", line 235, in get
gunicorn-web stdout |     username = self._decrypt_username(mirror.external_registry_username)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/mirror.py", line 566, in _decrypt_username
gunicorn-web stdout |     return username.decrypt()
gunicorn-web stdout |   File "/quay-registry/data/fields.py", line 119, in decrypt
gunicorn-web stdout |     return encrypter.decrypt_value(self.encrypted_value)
gunicorn-web stdout |   File "/quay-registry/data/encryption.py", line 97, in decrypt_value
gunicorn-web stdout |     return _VERSIONS[version_prefix].decrypt(self._secret_key, data)
gunicorn-web stdout |   File "/quay-registry/data/encryption.py", line 48, in _decrypt_ccm
gunicorn-web stdout |     raise DecryptionFailureException()
gunicorn-web stdout | data.encryption.DecryptionFailureException
gunicorn-web stdout | During handling of the above exception, another exception occurred:
gunicorn-web stdout | Traceback (most recent call last):
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
gunicorn-web stdout |     self.handle_request(listener_name, req, client, addr)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
gunicorn-web stdout |     super().handle_request(listener_name, req, sock, addr)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request
gunicorn-web stdout |     respiter = self.wsgi(environ, resp.start_response)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2463, in __call__
gunicorn-web stdout |     return self.wsgi_app(environ, start_response)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
gunicorn-web stdout |     return self.app(environ, start_response)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app
gunicorn-web stdout |     response = self.handle_exception(e)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 269, in error_router
gunicorn-web stdout |     return original_handler(e)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1866, in handle_exception
gunicorn-web stdout |     reraise(exc_type, exc_value, tb)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 38, in reraise
gunicorn-web stdout |     raise value.with_traceback(tb)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
gunicorn-web stdout |     response = self.full_dispatch_request()
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request
gunicorn-web stdout |     rv = self.handle_user_exception(e)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 269, in error_router
gunicorn-web stdout |     return original_handler(e)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
gunicorn-web stdout |     reraise(exc_type, exc_value, tb)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 38, in reraise
gunicorn-web stdout |     raise value.with_traceback(tb)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
gunicorn-web stdout |     rv = self.dispatch_request()
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
gunicorn-web stdout |     return self.view_functions[rule.endpoint](**req.view_args)
gunicorn-web stdout |   File "/quay-registry/endpoints/decorators.py", line 197, in wrapper
gunicorn-web stdout |     return func(*args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/auth/decorators.py", line 65, in wrapper
gunicorn-web stdout |     return func(*args, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/utils/cors.py", line 35, in wrapped_function
gunicorn-web stdout |     resp = make_response(f(*args, **kwargs))
gunicorn-web stdout |   File "/quay-registry/endpoints/csrf.py", line 73, in wrapper
gunicorn-web stdout |     resp = func(*args, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 458, in wrapper
gunicorn-web stdout |     resp = resource(*args, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/views.py", line 89, in view
gunicorn-web stdout |     return self.dispatch_request(*args, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
gunicorn-web stdout |     resp = meth(*args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/decorators.py", line 141, in wrapper
gunicorn-web stdout |     return func(*args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/__init__.py", line 241, in wrapper
gunicorn-web stdout |     return func(namespace, repository, *args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/decorators.py", line 120, in wrapper
gunicorn-web stdout |     return func(*args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/__init__.py", line 295, in wrapped
gunicorn-web stdout |     return func(self, namespace, repository, *args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/__init__.py", line 466, in wrapped
gunicorn-web stdout |     resp = func(self, *args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/mirror.py", line 237, in get
gunicorn-web stdout |     logger.warning(
gunicorn-web stdout | NameError: name 'logger' is not defined
gunicorn-web stdout | 2021-02-10 13:51:57,971 [300] [INFO] [gunicorn.access]  - - [10/Feb/2021:13:51:57 +0000] "GET /api/v1/repository/ibazulic/mirror/mirror HTTP/1.0" 500 0 "-" "-"

The workaround is to disable mirror configuration and to delete the config from the db directly. Then access to these repos is once again established:

quay-quay-database=#select * from repository;
 id | namespace_user_id |  name  | visibility_id | description |             badge_token              | kind_id | trust_enabled | state 
----+-------------------+--------+---------------+-------------+--------------------------------------+---------+---------------+-------
  1 |                 1 | mirror |             2 |             | e7c5d7ed-7686-4459-a90d-5d30fa40c655 |       1 | f             |     2
  2 |                 1 | test-2 |             2 |             | fd0bdf59-80ae-4d67-b96f-cc83bcd0a5e8 |       1 | f             |     0
  3 |                 1 | test-3 |             2 |             | 7a212a44-1a2b-420b-aedd-39833f43ce1f |       1 | f             |     0
  4 |                 1 | test-4 |             2 |             | 2496399b-bc42-441b-abbf-02d86aed650a |       1 | f             |     0
  5 |                 3 | test-1 |             2 |             | a9de9269-0873-46d1-8e44-beae0aa7be6a |       1 | f             |     0
  6 |                 3 | test-2 |             2 |             | b24299b8-da9f-48d2-9eb5-6bd2e4a5cac2 |       1 | f             |     0
  7 |                 3 | test-3 |             2 |             | a5b1c292-d3b8-47d1-95a0-0e8b873fddda |       1 | f             |     0
  8 |                 3 | test-4 |             2 |             | 3b216021-c89b-487c-b1e5-7561ad1d901e |       1 | f             |     0
(8 rows)

quay-quay-database=# update repository set state = 0 where id=1;
UPDATE 1
quay-quay-database=# delete from repomirrorconfig where repository_id = 1;
DELETE 1
quay-quay-database=# delete from repomirrorrule where repository_id = 1;
DELETE 1

However, this is not feasible when one has a lot of mirror repositories defined.

Details

Description

Attachments

Activity

People

Dates