Loading...

Type: Bug
Resolution: Unresolved
Priority: Normal
Fix Version/s: None
Affects Version/s: stream, 6.18.0
Component/s: Performance, Repositories
Labels:

Story Points:
3
Target Version:

6.18.z, 6.19.0
Blocked:
False
Fixed in Build:
rubygem-katello-4.19.0-0.2.pre.master.20251008032147gitdfc038f.el9sat.noarch
External issue URL:
https://projects.theforeman.org/issues/38789
GitHub Issue:
https://github.com/Katello/katello/pull/11514
Sprint:
Artemis Sprint 156, Artemis Sprint 157
AssignedTeam:
sat-artemis

Release Note Type:
None
Release Note Text:
None
Release Note Status:
None

PX Impact Score:
SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Test Coverage:
None

Market:

Description of problem:

While trying to concurrently synchronize several RPM repos (15 in our case) we've seen random failures:

PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 212202 waits for ShareLock on transaction 837733; blocked by process 211616.
Process 211616 waits for ShareLock on transaction 837755; blocked by process 212202.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (2230,4) in relation "katello_errata"

How reproducible:

Randomly.

Is this issue a regression from an earlier version:

Unknown

Steps to Reproduce:

1. Synchronize RPM repos with enough concurrency.

2.

3.

Actual behavior:
The task fails with the following exception:

ctiveRecord::Deadlocked: PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 212202 waits for ShareLock on transaction 837733; blocked by process 211616.
Process 211616 waits for ShareLock on transaction 837755; blocked by process 212202.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (2230,4) in relation "katello_errata"

And the following backtrace:

/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'
/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'
/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:54:in `exec_query'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract/database_statements.rb:150:in `exec_insert_all'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `exec_insert_all'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/insert_all.rb:41:in `execute'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/persistence.rb:333:in `upsert_all'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/services/katello/content_unit_indexer.rb:50:in `block in import_all'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/services/katello/pulp3/pulp_content_unit.rb:139:in `yield'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/services/katello/pulp3/pulp_content_unit.rb:139:in `block (2 levels) in pulp_units_batch_for_repo'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/services/katello/pulp3/pulp_content_unit.rb:127:in `loop'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/services/katello/pulp3/pulp_content_unit.rb:127:in `block in pulp_units_batch_for_repo'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/services/katello/content_unit_indexer.rb:26:in `each'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/services/katello/content_unit_indexer.rb:26:in `each'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/services/katello/content_unit_indexer.rb:26:in `import_all'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/models/katello/repository.rb:1033:in `block (2 levels) in index_content'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/lib/katello/logging.rb:8:in `time'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/models/katello/repository.rb:1032:in `block in index_content'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/models/katello/repository.rb:1031:in `each'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/models/katello/repository.rb:1031:in `index_content'
/usr/share/gems/gems/katello-4.19.0.pre.master/app/lib/actions/katello/repository/index_content.rb:22:in `run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/execution_wrapper.rb:92:in `wrap'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action/progress.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:33:in `run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:28:in `pass'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware.rb:20:in `pass'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-11.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/stack.rb:24:in `call'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:588:in `catch'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:588:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:491:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:491:in `catch'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:491:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:583:in `execute_run'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/action.rb:304:in `execute'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/director.rb:70:in `execute'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
/usr/share/gems/gems/dynflow-1.9.2/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'
/usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Expected behavior:
All the task should succeed.

Business Impact / Additional info:

is related to

SAT-24910 Possible DB deadlock when publishing a content view with many similar contents

New

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates