Loading...

Type: Bug
Resolution: Done-Errata
Priority: Normal
Fix Version/s: 6.16.5.4
Affects Version/s: None
Component/s: Installation, Upgrades
Labels:

Story Points:
3
Target Version:

6.16.5.4
Blocked:
False
Fixed in Build:
foreman-installer-3.12.0.6
AssignedTeam:
sat-se

Release Note Type:
None
Release Note Text:
None
Release Note Status:
None

PX Impact Score:
PX Review Complete:
SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Test Coverage:

To Do

Market:

Description of problem:

Satellite 6.16 upgrade failed with following error:

2025-04-02 15:39:05 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2025-04-02 15:39:05 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2025-04-02 15:39:05 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2025-04-02 15:39:05 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

    /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in <top (required)>'     Tasks: TOP => db:abort_if_pending_migrations => db:load_config => environment     (See full trace by running task with --trace)   /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns     rake aborted!     Errno::EBUSY: Device or resource busy - getaddrinfo     /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:183:in getaddrinfo

Traceback shows that Foreman is unable to resolve the Redis server due to device busy error:

2025-04-16 14:51:45 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:migrate'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Errno::EBUSY: Device or resource busy - getaddrinfo
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:183:in `getaddrinfo'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:183:in `connect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:305:in `connect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:385:in `establish_connection'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:117:in `block in connect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:344:in `with_reconnect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:116:in `connect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:417:in `ensure_connected'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:269:in `block in process'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:356:in `logging'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:268:in `process'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:162:in `call'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:564:in `block in del'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `block in synchronize'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `synchronize'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:563:in `del'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:418:in `block (2 levels) in delete_entry'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:25:in `with'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:418:in `block in delete_entry'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:417:in `delete_entry'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/strategy/local_cache.rb:170:in `delete_entry'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:487:in `block in delete'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:726:in `block in instrument'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications.rb:205:in `instrument'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:726:in `instrument'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:486:in `delete'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/app/services/topbar_sweeper.rb:18:in `block in expire_cache_all_users'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `each'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `expire_cache_all_users'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/config/initializers/foreman.rb:31:in `block in <top (required)>'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:427:in `instance_exec'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:427:in `block in make_lambda'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:603:in `catch'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:603:in `block in default_terminator'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:199:in `block in halting'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:512:in `block in invoke_before'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:512:in `each'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:512:in `invoke_before'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:105:in `run_callbacks'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/reloader.rb:88:in `prepare!'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:32:in `instance_exec'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:32:in `run

The root cause is Redis started after the db migration rake run. The "expire_cache_all_users" method which is triggered by rake fails to connect to the Redis server. It retries the connection for each user and eventually fails with the "Device or resoucrece busy" error. It appears to be a bug that the connection failure doesn't cleanup the FD.

2025-04-16 14:51:25 [DEBUG ] [configure] Execforeman-rake-db:migrate: Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations' 2025-04-16 14:51:45 [DEBUG ] [configure] Execforeman-rake-db:migrate: Executing '/usr/sbin/foreman-rake db:migrate'
2025-04-16 14:51:45 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:migrate'
...
...
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl is-active -- redis'
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- redis'
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl show --property=NeedDaemonReload -- redis'
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl unmask -- redis' 28125:2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl start -- redis'
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- redis'

How reproducible:

Easy

Is this issue a regression from an earlier version:

Possibly. Since Foreman switched its caching method to Redis which I believe is Satellite 6.16.

Steps to Reproduce:

1. Prepare a Satellite 6.15 server.

2. Create more than 520 users.

3. Perform upgrade to Satellite 6.16.

Actual behavior:
Upgrade fails with the error above

Expected behavior:
Satellite should be upgraded successfully

Business Impact / Additional info:

clones

SAT-36447 Fail to upgrade to Satellite 6.16 when Satellite has more than 520 users

Closed

depends on

SAT-35336 Fixes #38517 - Ensure Redis is running before db:migrate by ekohl · Pull Request #1226 · theforeman/puppet-foreman · GitHub

Closed

links to

KCS

RHSA-2025:155064 Satellite 6.16.5.4 Async Update

1.

[QE] Fail to upgrade to Satellite 6.16 when Satellite has more than 520 users

New

Unassigned

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Sub-Tasks

Activity

People

Dates