Loading...

Type: Bug
Resolution: Done
Priority: Normal
Fix Version/s: None
Affects Version/s: 2.1.1
Component/s: Pulp
Labels:
- automationhub
- customer

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Acceptance Criteria:
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

While deploying Automation Hub in HA with shared filesystem, remote registry is not working properly.

Steps to reproduce:

Install Automation Hub on HA with NFS for `/var/lib/pulp`
Add a new remote registry under Execution Environments > Remote Registries.
Open the second Automation Hub node and try to load the Remote Registries page, it will be forever loading and no output will be shown.
When trying to sync the remote registry you will get the below error:

  File "/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 323, in _perform_task
    result = func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/galaxy_ng/app/tasks/registry_sync.py", line 26, in sync_all_repos_in_registry
    registry = models.ContainerRegistryRemote.objects.get(pk=registry_pk)
  File "/usr/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 68, in __iter__
    for row in compiler.results_iter(results):
  File "/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
    value = converter(value, expression, connection)
  File "/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 104, in from_db_value
    return force_str(self._fernet.decrypt(force_bytes(value)))
  File "/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 75, in decrypt
    return self._decrypt_data(data, timestamp, ttl)
  File "/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 117, in _decrypt_data
    self._verify_signature(data)
  File "/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 106, in _verify_signature
    raise InvalidToken

When checking the logs from second node you can see the following error while trying to check remote registry page:

Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]: pulp [None]: django.request:ERROR: Internal Server Error: /api/galaxy/_ui/v1/execution-environments/registries/
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]: Traceback (most recent call last):
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 104, in _verify_signature
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    h.verify(data[-32:])
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib64/python3.8/site-packages/cryptography/hazmat/primitives/hmac.py", line 66, in verify
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    ctx.verify(signature)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib64/python3.8/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 74, in verify
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    raise InvalidSignature("Signature did not match digest.")
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]: cryptography.exceptions.InvalidSignature: Signature did not match digest.
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]: During handling of the above exception, another exception occurred:
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]: Traceback (most recent call last):
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    response = get_response(request)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    response = wrapped_callback(request, *callback_args, **callback_kwargs)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    return view_func(*args, **kwargs)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    return self.dispatch(request, *args, **kwargs)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    response = self.handle_exception(exc)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    self.raise_uncaught_exception(exc)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    raise exc
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    response = handler(request, *args, **kwargs)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/rest_framework/mixins.py", line 40, in list
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    page = self.paginate_queryset(queryset)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 171, in paginate_queryset
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    return self.paginator.paginate_queryset(queryset, self.request, view=self)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/rest_framework/pagination.py", line 395, in paginate_queryset
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    return list(queryset[self.offset:self.offset + self.limit])
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    self._fetch_all()
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    self._result_cache = list(self._iterable_class(self))
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 68, in __iter__
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    for row in compiler.results_iter(results):
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    value = converter(value, expression, connection)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 104, in from_db_value
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    return force_str(self._fernet.decrypt(force_bytes(value)))
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 75, in decrypt
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    return self._decrypt_data(data, timestamp, ttl)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 117, in _decrypt_data
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    self._verify_signature(data)
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:  File "/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 106, in _verify_signature
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]:    raise InvalidToken
Mar 17 16:03:24 ip-10-0-2-172 gunicorn[1070]: cryptography.fernet.InvalidToken

is duplicated by

AAH-1838 Execution Environment pages in Automation Hub fail with HTTP 500

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates

PagerDuty