Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Normal
Fix Version/s: None
Affects Version/s: 6.13.0
Component/s: Fact
Labels:
- team-triaged
- triaged

Story Points:
5
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
PM Score:
0
BZ Keywords:
- Unset
Intelligence Requested:
Market:

Severity:
Moderate

Test Link:
https://github.com/SatelliteQE/robottelo/blob/a54eb8b27ee2c62736378bdf85a3becbb80e8d79/tests/foreman/cli/test_fact.py#L144
Test Coverage:

Automated
Regression:
None

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

PX Priority Data:
PX Impact Score:

Description of problem:
`foreman-rake facts:clean` randomly raises below exception. The cause is non-transactional removal of `leaf_orphaned_facts` (done in batches): when we update a leaf fact during deleting previous orphaned leaf facts, we aim to delete a fact with value and fail with `PG::ForeignKeyViolation`.

How reproducible:
100% in several attempts

Is this issue a regression from an earlier version:
Almost surely not a regression

Steps to Reproduce:

1. Have json files with set and unset fact values (customize the `SOME.HOST.FQDN`):

(echo "{"; echo '    "name": "SOME.HOST.FQDN",'; echo '    "facts": {') > facts_set.json
for i in $(seq 2000); do echo "        \"pmoravec_fact_${i}\": \"some_value\","; done >> facts_set.json
(echo '       "operatingsystem": "RedHat"'; echo '    }'; echo '}') >> facts_set.json
sed "s/some_value//g" facts_set.json > facts_unset.json

Now, `facts_set.json` will set 2k dummy facts about the Host, and `facts_unset.json` will nullify them.

2. Set and unset the facts, just to create leaf orphaned fact records:

curl -u admin:PASSWORD -X POST -H "Accept: application/json" -H "Content-Type: application/json" -d @facts_set.json https://$(hostname -f)/api/v2/hosts/facts 
curl -u admin:PASSWORD -X POST -H "Accept: application/json" -H "Content-Type: application/json" -d @facts_unset.json https://$(hostname -f)/api/v2/hosts/facts

3. Open rake console to be able to promptly call facts cleanup:

foreman-rake console

fc = FactCleaner.new()

4. Now, run almost concurrently (per my experience, run `curl` 1-2 seconds before the `rake console` command):

curl -u admin:PASSWORD -X POST -H "Accept: application/json" -H "Content-Type: application/json" -d @facts_set.json https://$(hostname -f)/api/v2/hosts/facts

and promptly after that in rake console:

fc.clean!

If you are lucky, you get:

Traceback (most recent call last):
        6: from lib/tasks/console.rake:5:in `block in <top (required)>'
        5: from (irb):10
        4: from app/services/fact_cleaner.rb:11:in `clean!'
        3: from app/services/fact_cleaner.rb:34:in `delete_orphaned_facts'
        2: from app/services/fact_cleaner.rb:41:in `delete_leaf_orphaned_facts'
        1: from app/services/fact_cleaner.rb:44:in `block in delete_leaf_orphaned_facts'
ActiveRecord::InvalidForeignKey (PG::ForeignKeyViolation: ERROR:  update or delete on table "fact_names" violates foreign key constraint "fact_values_fact_name_id_fk" on table "fact_values")
DETAIL:  Key (id)=(5146) is still referenced from table "fact_values".

if unlucky and you get:

irb(main):011:0> fc.clean!
=> #<FactCleaner:0x0000565509742c00 @batch_size=500, @deleted_count=0, @live_ancestors=#<Set: {3, 24, 26, 27, 33, 39, 40, 44, 48, 52, 57, 58, 66, 79, 92, 93, 95, 109, 110, 113, 118, 119, 130, 142, 153, 164, 175, 141, 197, 196, 218, 238, 242, 243, 268, 270, 274, 281, 285, 294, 295, 301, 306, 313, 326, 340, 358, 359, 360, 365, 366, 371, 372, 385, 391, 400, 406, 410, 411, 416, 424, 435, 441, 456, 458, 468, 475, 485, 512, 515, 516, 517, 519, 524, 525, 548, 550, 553, 559, 594, 1116, 409}>>
irb(main):012:0>

then let set & unset the facts again, and try the [ concurrent set + cleaner] again.

Actual behavior:
Above backtrace appears. Optionally, if running `foreman-rakce facts:clean` directly, you get backtrace:

[root@satellite bin]# /usr/sbin/foreman-rake facts:clean
Starting orphaned facts clean up
rake aborted!
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  update or delete on table "fact_names" violates foreign key constraint "fact_values_fact_name_id_fk" on table "fact_values"
DETAIL:  Key (id)=(103773) is still referenced from table "fact_values".
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:72:in `exec_delete'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:185:in `delete'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `delete'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation.rb:594:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:42:in `block in delete_leaf_orphaned_facts'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/foreman/app/services/fact_cleaner.rb:41:in `delete_leaf_orphaned_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:34:in `delete_orphaned_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:11:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:15:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'

Expected behavior:
no such backtrace / error

Business Impact / Additional info:

links to

KCS 7085386

Assignee:: Unassigned

Reporter:: Pavel Moravec

Votes:: 1 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2024/09/03 10:13 AM

Updated:: 2024/10/31 8:34 AM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates