Description of problem:

Getting '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0] ERROR while trying to upgrade Satellite 6.13 to 6.14

Version-Release number of selected component (if applicable):

6.13

How reproducible:

100% in customer's env

Steps to Reproduce:
1. Refer the steps to upgrade your satellite from 6.13.z to 6.14

https://access.redhat.com/documentation/en-us/red_hat_satellite/6.14/html/upgrading_red_hat_satellite_to_6.14/upgrading_satellite#upgrading_a_connected_satellite_server_upgrade-guide

2. Upgrade fails at db migrate stage

Actual results:

2023-11-29 08:30:59 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2023-11-29 08:30:59 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2023-11-29 08:31:56 [NOTICE] [configure] 1500 configuration steps out of 1612 steps complete.
2023-11-29 08:31:59 [NOTICE] [configure] System configuration has finished.

Expected results:

Upgrade should completed without any errors.

Additional info:

1. foreman-rake db:migrate --trace --verbose
   • Invoke db:migrate (first_time)
   • Invoke db:load_config (first_time)
   • Invoke environment (first_time)
   • Execute environment
   • Execute db:load_config
   • Invoke plugin:refresh_migrations (first_time)
   • Invoke environment
   • Execute plugin:refresh_migrations
   • Execute db:migrate
     == 20220929204746 AddContentViewEnvironmentContentFacet: migrating ============
   • create_table(:katello_content_view_environment_content_facets)
     -> 0.0069s
     rake aborted!
     StandardError: An error has occurred, this and all later migrations canceled:

undefined method `id' for nil:NilClass
/usr/share/gems/gems/katello-4.9.0.16/db/migrate/20220929204746_add_content_view_environment_content_facet.rb:15:in `block in up'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/katello-4.9.0.16/db/migrate/20220929204746_add_content_view_environment_content_facet.rb:11:in `up'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:870:in `public_send'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:870:in `exec_migration'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/usr/share/ruby/benchmark.rb:293:in `measure'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:850:in `block in migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:849:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1037:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/transactions.rb:209:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1380:in `ddl_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1302:in `each'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1251:in `block in migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1251:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1086:in `up'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1061:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/databases.rake:90:in `each'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/bin/rake:23:in `load'
/usr/bin/rake:23:in `<main>'

Caused by:
NoMethodError: undefined method `id' for nil:NilClass
/usr/share/gems/gems/katello-4.9.0.16/db/migrate/20220929204746_add_content_view_environment_content_facet.rb:15:in `block in up'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/katello-4.9.0.16/db/migrate/20220929204746_add_content_view_environment_content_facet.rb:11:in `up'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:870:in `public_send'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:870:in `exec_migration'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/usr/share/ruby/benchmark.rb:293:in `measure'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:850:in `block in migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:849:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1037:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/transactions.rb:209:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1380:in `ddl_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1302:in `each'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1251:in `block in migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1251:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1086:in `up'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/migration.rb:1061:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/databases.rake:90:in `each'
/usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/bin/rake:23:in `load'
/usr/bin/rake:23:in `<main>'
Tasks: TOP => db:migrate