-
Bug
-
Resolution: Done
-
Normal
-
6.15.5, 6.16.5.1
-
3
-
False
-
pulp-ansible-0.23.0,pulp-ansible-0.22.2,pulp-ansible-0.21.9,pulp-ansible-0.20.9
-
Moderate
-
Artemis Sprint 154
-
sat-artemis
-
None
-
None
-
None
-
Automated
Description of problem:
Having several CVs promoted to several LEs, each having same ansible collection repo, then an attempt to synchronize the content to Capsule fails with a postgres deadlock.
Affected versions: 6.17 and older (python3.11-pulp-ansible < 0.23).
How reproducible:
almost 100%
Is this issue a regression from an earlier version:
probably not
Steps to Reproduce:
1. Have an ansible collections repo with several collections synced. E.g. in my case, I was syncing from console.redhat.com with Requirements.yaml:
--- collections: - name: ansible.eda - name: check_point.mgmt - name: ansible.platform - name: redhat.satellite
(that is 74 collections)
2. Create 8 CVs with just the one repo, and publish+promote the CV to 5ish Lifecycle Environments.
3. Assign all the 5 LEs to a Capsule.
4. Invoke (Optimised) Capsule Sync.
5. Optionally, delete all repos from the Capsule, run orphan cleanup and repeat step 4.
Actual behavior:
Deadlock happens:
May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: pulp [45c11f6e-0df6-49f5-8f22-0ba00bcc91cf]: pulpcore.tasking.tasks:INFO: Task 01970c0d-0da0-70dc-ab10-c0aa703704e1 failed (deadlock detected May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: DETAIL: Process 15738 waits for ShareLock on transaction 9221793; blocked by process 15695. May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: Process 15695 waits for ShareLock on transaction 9221791; blocked by process 15738. May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: HINT: See server log for query details. May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: CONTEXT: while rechecking updated tuple (2799,4) in relation "ansible_collectionversion") May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: pulp [45c11f6e-0df6-49f5-8f22-0ba00bcc91cf]: pulpcore.tasking.tasks:INFO: File "/usr/lib/python3.11/site-packages/pulpcore/tasking/tasks.py", line 66, in _execute_task May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: result = func(*args, **kwargs) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/pulp_ansible/app/tasks/collections.py", line 192, in sync May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: repo_version = d_version.create() May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/declarative_version.py", line 163, in create May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: loop.run_until_complete(pipeline) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib64/python3.11/asyncio/base_events.py", line 654, in run_until_complete May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: return future.result() May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/api.py", line 220, in create_pipeline May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: await asyncio.gather(*futures) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/api.py", line 41, in __call__ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: await self.run() May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/content_stages.py", line 198, in run May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: await sync_to_async(process_batch)() May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ret = await asyncio.wait_for(future, timeout=None) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib64/python3.11/asyncio/tasks.py", line 452, in wait_for May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: return await fut May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: result = self.fn(*self.args, **self.kwargs) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/asgiref/sync.py", line 490, in thread_handler May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: return func(*args, **kwargs) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/content_stages.py", line 196, in process_batch May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: self._post_save(batch) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/pulp_ansible/app/tasks/collections.py", line 1240, in _post_save May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: _update_highest_version(collection_version) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/pulp_ansible/app/tasks/collections.py", line 443, in _update_highest_version May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: update_qs.update(is_highest=False) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1206, in update May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: rows = query.get_compiler(self.db).execute_sql(CURSOR) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1984, in execute_sql May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: cursor = super().execute_sql(result_type) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: cursor.execute(sql, params) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: return self._execute_with_wrappers( May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: return executor(sql, params, many, context) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: with self.db.wrap_database_errors: May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: raise dj_exc_value.with_traceback(traceback) from exc_value May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: return self.cursor.execute(sql, params) May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: File "/usr/lib/python3.11/site-packages/psycopg/cursor.py", line 97, in execute May 26 12:05:11 pmoravec-caps616 pulpcore-worker-2[15735]: raise ex.with_traceback(None)
and in postgres logs:
2025-05-26 12:05:11 CEST ERROR: deadlock detected
2025-05-26 12:05:11 CEST DETAIL: Process 15738 waits for ShareLock on transaction 9221793; blocked by process 15695.
Process 15695 waits for ShareLock on transaction 9221791; blocked by process 15738.
Process 15738: UPDATE "ansible_collectionversion" SET "is_highest" = false WHERE "ansible_collectionversion"."collection_id" = '01970b72e7547cfab1b3a9d124c711ed'::uuid
Process 15695: UPDATE "ansible_collectionversion" SET "is_highest" = false WHERE "ansible_collectionversion"."collection_id" = '01970b72e7547cfab1b3a9d124c711ed'::uuid
2025-05-26 12:05:11 CEST HINT: See server log for query details.
2025-05-26 12:05:11 CEST CONTEXT: while rechecking updated tuple (2799,4) in relation "ansible_collectionversion"
2025-05-26 12:05:11 CEST STATEMENT: UPDATE "ansible_collectionversion" SET "is_highest" = false WHERE "ansible_collectionversion"."collection_id" = '01970b72e7547cfab1b3a9d124c711ed'::uuid
Expected behavior:
No deadlock
Business Impact / Additional info:
https://github.com/pulp/pulp_ansible/commit/0c5490c248b06bb58107e433b2a8bd23bd2e4ae9 basically fixes this, since it removes the method where concurrent processes stuck in the deadlock.
- is cloned by
-
-
- To Do
-
- links to
