Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-38406 Fail to upgrade to Satellite 6.16 when Satellite has more than 520 users
  3. SAT-38407

[QE] Fail to upgrade to Satellite 6.16 when Satellite has more than 520 users

XMLWordPrintable

    • Icon: Sub-task Sub-task
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • 6.16.z
    • None
    • Installation, Upgrades
    • False
    • Hide

      None

      Show
      None
    • False
    • 0
    • sat-rocket

      Description of problem:

      Satellite 6.16 upgrade failed with following error:

      2025-04-02 15:39:05 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2025-04-02 15:39:05 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2025-04-02 15:39:05 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2025-04-02 15:39:05 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

    /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in <top (required)>'     Tasks: TOP => db:abort_if_pending_migrations => db:load_config => environment     (See full trace by running task with --trace)   /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns     rake aborted!     Errno::EBUSY: Device or resource busy - getaddrinfo     /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:183:in getaddrinfo

       

      Traceback shows that Foreman is unable to resolve the Redis server due to device busy error:

      2025-04-16 14:51:45 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:migrate'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Errno::EBUSY: Device or resource busy - getaddrinfo
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:183:in `getaddrinfo'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:183:in `connect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:305:in `connect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:385:in `establish_connection'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:117:in `block in connect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:344:in `with_reconnect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:116:in `connect'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:417:in `ensure_connected'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:269:in `block in process'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:356:in `logging'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:268:in `process'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:162:in `call'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:564:in `block in del'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `block in synchronize'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `synchronize'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:563:in `del'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:418:in `block (2 levels) in delete_entry'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:25:in `with'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:418:in `block in delete_entry'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:417:in `delete_entry'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/strategy/local_cache.rb:170:in `delete_entry'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:487:in `block in delete'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:726:in `block in instrument'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications.rb:205:in `instrument'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:726:in `instrument'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:486:in `delete'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/app/services/topbar_sweeper.rb:18:in `block in expire_cache_all_users'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `each'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `expire_cache_all_users'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/config/initializers/foreman.rb:31:in `block in <top (required)>'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:427:in `instance_exec'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:427:in `block in make_lambda'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:603:in `catch'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:603:in `block in default_terminator'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:199:in `block in halting'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:512:in `block in invoke_before'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:512:in `each'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:512:in `invoke_before'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:105:in `run_callbacks'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/reloader.rb:88:in `prepare!'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:32:in `instance_exec'
2025-04-16 14:52:06 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:32:in `run

       

      The root cause is Redis started after the db migration rake run. The "expire_cache_all_users" method which is triggered by rake fails to connect to the Redis server. It retries the connection for each user and eventually fails with the "Device or resoucrece busy" error. It appears to be a bug that the connection failure doesn't cleanup the FD.

      2025-04-16 14:51:25 [DEBUG ] [configure] Execforeman-rake-db:migrate: Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations' 2025-04-16 14:51:45 [DEBUG ] [configure] Execforeman-rake-db:migrate: Executing '/usr/sbin/foreman-rake db:migrate'
2025-04-16 14:51:45 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:migrate'
...
...
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl is-active -- redis'
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- redis'
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl show --property=NeedDaemonReload -- redis'
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl unmask -- redis' 28125:2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl start -- redis'
2025-04-16 14:52:34 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- redis'

       

      How reproducible:

      Easy

       

      Is this issue a regression from an earlier version:

      Possibly. Since Foreman switched its caching method to Redis which I believe is Satellite 6.16.

       

      Steps to Reproduce:

      1. Prepare a Satellite 6.15 server.

      2. Create more than 520 users.

      3. Perform upgrade to Satellite 6.16.

      Actual behavior:
      Upgrade fails with the error above

      Expected behavior:
      Satellite should be upgraded successfully

       

      Business Impact / Additional info:

       

      QE Tracker for https://issues.redhat.com/browse/SAT-36447

              Unassigned Unassigned
              satellite-jira-automation@redhat.com Satellite Jira-Automation
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: