Description of problem:

Import task goes to a paused state and return error complaining about a deadlock when indexing repositories

Version-Release number of selected component (if applicable):

6.12

How reproducible:

Eventually

Steps to Reproduce:

Still unsure. Managed to reproduce once, but couldn't repeat it regularly.

Saved logs from the time it happened on the lab.

Actual results: import task get paused due to deadlock. Resuming the task works fine and it can finish.

Expected results: no deadlock

Additional info:

~~~

1. hammer content-import library --path /var/lib/pulp/imports/Export-Library/1.0/2023-04-05T09-11-07-04-00/ --organization-id 1
   [...................................................................... ] [31%]
   Error: PG::TRDeadlockDetected: ERROR: deadlock detected
   DETAIL: Process 2151861 waits for ShareLock on transaction 12856382; blocked by process 2151215.
   Process 2151215 waits for ShareLock on transaction 12856380; blocked by process 2151861.
   HINT: See server log for query details.
   CONTEXT: while inserting index tuple (1,1) in relation "katello_erratum_bugzillas"
   ~~~

DB log:

~~~
2023-04-05 10:59:42 EDT ERROR: deadlock detected
2023-04-05 10:59:42 EDT DETAIL: Process 2151861 waits for ShareLock on transaction 12856382; blocked by process 2151215.
Process 2151215 waits for ShareLock on transaction 12856380; blocked by process 2151861.
Process 2151861: INSERT INTO "katello_erratum_bugzillas" ("bug_id","href","erratum_id") VALUES ('2131770', 'https://bugzilla.redhat.com/show_bug.cgi?id=2131770', 72684), ('2049850', 'https://bugzilla.redhat.com/show_bug.cgi?id=2049850', 72685), ('1766653', 'https://bugzilla.redhat.com/show_bug.cgi?id=1766653', 72685), ('2124088', 'https://bugzilla.redhat.com/show_bug.cgi?id=2124088', 72686), ('2116280', 'https://bugzilla.redhat.com/show_bug.cgi?id=2116280', 72686) ON CONFLICT ("erratum_id","bug_id","href") DO NOTHING RETURNING "id"
Process 2151215: INSERT INTO "katello_erratum_bugzillas" ("bug_id","href","erratum_id") VALUES ('1766653', 'https://bugzilla.redhat.com/show_bug.cgi?id=1766653', 72685), ('2049850', 'https://bugzilla.redhat.com/show_bug.cgi?id=2049850', 72685), ('2131770', 'https://bugzilla.redhat.com/show_bug.cgi?id=2131770', 72684), ('2124088', 'https://bugzilla.redhat.com/show_bug.cgi?id=2124088', 72686), ('2116280', 'https://bugzilla.redhat.com/show_bug.cgi?id=2116280', 72686) ON CONFLICT ("erratum_id","bug_id","href") DO NOTHING RETURNING "id"
2023-04-05 10:59:42 EDT HINT: See server log for query details.
2023-04-05 10:59:42 EDT CONTEXT: while inserting index tuple (1,1) in relation "katello_erratum_bugzillas"
2023-04-05 10:59:42 EDT STATEMENT: INSERT INTO "katello_erratum_bugzillas" ("bug_id","href","erratum_id") VALUES ('2131770', 'https://bugzilla.redhat.com/show_bug.cgi?id=2131770', 72684), ('2049850', 'https://bugzilla.redhat.com/show_bug.cgi?id=2049850', 72685), ('1766653', 'https://bugzilla.redhat.com/show_bug.cgi?id=1766653', 72685), ('2124088', 'https://bugzilla.redhat.com/show_bug.cgi?id=2124088', 72686), ('2116280', 'https://bugzilla.redhat.com/show_bug.cgi?id=2116280', 72686) ON CONFLICT ("erratum_id","bug_id","href") DO NOTHING RETURNING "id"
2023-04-05 10:59:42 EDT LOG: duration: 1008.287 ms execute <unnamed>: INSERT INTO "katello_erratum_bugzillas" ("bug_id","href","erratum_id") VALUES ('1766653', 'https://bugzilla.redhat.com/show_bug.cgi?id=1766653', 72685), ('2049850', 'https://bugzilla.redhat.com/show_bug.cgi?id=2049850', 72685), ('2131770', 'https://bugzilla.redhat.com/show_bug.cgi?id=2131770', 72684), ('2124088', 'https://bugzilla.redhat.com/show_bug.cgi?id=2124088', 72686), ('2116280', 'https://bugzilla.redhat.com/show_bug.cgi?id=2116280', 72686) ON CONFLICT ("erratum_id","bug_id","href") DO NOTHING RETURNING "id"
~~~

production.log:

~~~
2023-04-05T10:59:42 [I|app|10bb9b2a] CONTENT_INDEX type=Katello::Erratum success=false duration=1166.52
2023-04-05T10:59:42 [E|bac|10bb9b2a] PG::TRDeadlockDetected: ERROR: deadlock detected
10bb9b2a | DETAIL: Process 2151861 waits for ShareLock on transaction 12856382; blocked by process 2151215.
10bb9b2a | Process 2151215 waits for ShareLock on transaction 12856380; blocked by process 2151861.
10bb9b2a | HINT: See server log for query details.
10bb9b2a | CONTEXT: while inserting index tuple (1,1) in relation "katello_erratum_bugzillas"
10bb9b2a | (ActiveRecord::Deadlocked)
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `exec_params'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:728:in `block (2 levels) in log'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:727:in `block in log'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:718:in `log'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:654:in `execute_and_clear'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:153:in `exec_insert_all'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `exec_insert_all'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/insert_all.rb:27:in `execute'
10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/persistence.rb:124:in `insert_all'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/erratum.rb:60:in `insert_child_associations'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/content_unit_indexer.rb:60:in `import_associations'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/content_unit_indexer.rb:50:in `block in import_all'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/pulp_content_unit.rb:97:in `yield'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/pulp_content_unit.rb:97:in `block (2 levels) in pulp_units_batch_for_repo'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/pulp_content_unit.rb:85:in `loop'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/pulp_content_unit.rb:85:in `block in pulp_units_batch_for_repo'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/content_unit_indexer.rb:26:in `each'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/content_unit_indexer.rb:26:in `import_all'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/models/katello/repository.rb:926:in `block (2 levels) in index_content'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/lib/katello/logging.rb:8:in `time'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/models/katello/repository.rb:925:in `block in index_content'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/models/katello/repository.rb:924:in `each'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/models/katello/repository.rb:924:in `index_content'
10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/lib/actions/katello/repository/index_content.rb:22:in `run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/execution_wrapper.rb:91:in `wrap'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:69:in `execute'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
10bb9b2a | [ sidekiq ]
10bb9b2a | [ concurrent-ruby ]
~~~