-
Bug
-
Resolution: Done-Errata
-
Normal
-
6.13.z, 6.14.z, 6.15.z, 6.16.z, stream
-
Phoenix - Subscriptions
-
5
-
False
-
foreman-3.12.0.7
-
Moderate
-
None
-
None
-
None
-
None
Description of problem:
While testing concurrent registrations in batches we see the following error leading to some hosts (2, 3 usually) with the same OS version failing to register:
PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_operatingsystems_on_title"
DETAIL: Key (title)=(RedHat 9.2) already exists.
After these errors, the rest of registrations work without problems.
How reproducible:
We see it frequently in our test runs (at least since 6.13).
Is this issue a regression from an earlier version:
Analyzing the logs it seems to impact Satellite at least since 6.13.
Steps to Reproduce:
1. Set up concurrent registrations with enough concurrency (in our tests with around 50 concurrent content hosts)
2. Check registration logs
Actual behavior:
The affected hosts fail to register initially but success in the next try.
We can see the following in production.log:
2024-09-09T17:17:40 [I|app|930122f3] Started POST "/rhsm/consumers?owner=Default_Organization&activation_keys=AK_rhel9_Test" for 192.168.53.32 at 2024-09-09 17:17:40 +0000 2024-09-09T17:17:40 [I|app|930122f3] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON 2024-09-09T17:17:40 [I|app|930122f3] Parameters: {"type"=>"system", "name"=>"containerhost-26-1-el9-container086.red.ddns.perf.redhat.com", "facts"=>"[FILTERED]", "installedProducts"=>[{"productId"=>"479", "productName"=>"Red Hat Enterprise Linux for x86_64", "version"=>"9.4", "arch"=>"x86_64"}], "contentTags"=>["rhel-9", "rhel-9-x86_64"], "role"=>"", "addOns"=>[], "usage"=>"", "serviceLevel"=>"", "owner"=>"Default_Organization", "activation_keys"=>"AK_rhel9_Test"} 2024-09-09T17:17:41 [I|aud|930122f3] Nic::Managed (13) create event on mac . . . 2024-09-09T17:17:44 [I|aud|930122f3] Host::Base (12) destroy event on ansible_role_ids [] 2024-09-09T17:17:45 [I|app|930122f3] ForemanWebhooks::EventSubscriber: host_destroyed.event.foreman event received 2024-09-09T17:17:45 [E|kat|930122f3] ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_operatingsystems_on_title" 930122f3 | DETAIL: Key (title)=(RedHat 9.2) already exists. 930122f3 | 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:55:in `exec_query' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:136:in `exec_insert' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:93:in `exec_insert' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:171:in `insert' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `insert' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/persistence.rb:375:in `_insert_record' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/persistence.rb:929:in `_create_record' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/counter_cache.rb:166:in `_create_record' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/locking/optimistic.rb:79:in `_create_record' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/attribute_methods/dirty.rb:201:in `_create_record' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/callbacks.rb:461:in `block in _create_record' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:106:in `run_callbacks' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:824:in `_run_create_callbacks' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/callbacks.rb:461:in `_create_record' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/timestamp.rb:108:in `_create_record' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/persistence.rb:900:in `create_or_update' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/callbacks.rb:457:in `block in create_or_update' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:117:in `block in run_callbacks' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/autosave_association.rb:385:in `around_save_collection_association' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:126:in `block in run_callbacks' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:137:in `run_callbacks' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:824:in `_run_save_callbacks' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/callbacks.rb:457:in `create_or_update' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/timestamp.rb:126:in `create_or_update' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/persistence.rb:474:in `save' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/validations.rb:47:in `save' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:298:in `block in save' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt' 930122f3 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:350:in `with_transaction_returning_status' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:298:in `save' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/suppressor.rb:44:in `save' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/persistence.rb:38:in `create' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:795:in `_create' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:99:in `block in create' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:406:in `block in scoping' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:804:in `_scoping' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:406:in `scoping' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:99:in `create' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation.rb:168:in `find_or_create_by' 930122f3 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/querying.rb:22:in `find_or_create_by' 930122f3 | /usr/share/foreman/app/services/katello/rhsm_fact_parser.rb:76:in `operatingsystem' 930122f3 | /usr/share/foreman/app/models/host/base.rb:169:in `block in set_non_empty_values' 930122f3 | /usr/share/foreman/app/models/host/base.rb:168:in `each' 930122f3 | /usr/share/foreman/app/models/host/base.rb:168:in `set_non_empty_values' 930122f3 | /usr/share/foreman/app/models/host/base.rb:162:in `populate_fields_from_facts' 930122f3 | /usr/share/foreman/app/models/host/managed.rb:472:in `populate_fields_from_facts' 930122f3 | /usr/share/foreman/app/services/host_fact_importer.rb:56:in `block (2 levels) in parse_facts' 930122f3 | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:28:in `telemetry_duration_histogram' 930122f3 | /usr/share/foreman/app/services/host_fact_importer.rb:55:in `block in parse_facts' 930122f3 | /usr/share/foreman/app/models/concerns/orchestration.rb:124:in `without_orchestration' 930122f3 | /usr/share/foreman/app/services/host_fact_importer.rb:95:in `skipping_orchestration' 930122f3 | /usr/share/foreman/app/services/host_fact_importer.rb:51:in `parse_facts' 930122f3 | /usr/share/foreman/app/services/host_fact_importer.rb:34:in `import_facts' . . .
Expected behavior:
The content hosts should register without issue.
Business Impact / Additional info:
It can lead to an increment in registration time because of the need for retrial.
- links to
-
RHSA-2025:147428
Satellite 6.16.4 Async Update
