Description of problem:
When the redis cache option is enabled the notification drawer fails to open sporadically

Version-Release number of selected component (if applicable):

• Satellite 6.14
• foreman-redis-3.7.0.9-1.el8sat.noarch

How reproducible:

Steps to Reproduce:
1. Configure the Satellite to use redis cache

1. satellite-installer --foreman-rails-cache-store type:redis
   2. Try to open the notification drawer

Actual results:

• The notification drawer fails to open with an error message in /var/log/foreman/production.log

2023-12-11T14:59:40 [I|app|d9924adb] Started GET "/notification_recipients" for 192.168.1.30 at 2023-12-11 14:59:40 +0100
2023-12-11T14:59:40 [I|app|d9924adb] Processing by NotificationRecipientsController#index as JSON
2023-12-11T14:59:40 [W|app|d9924adb] Action failed
2023-12-11T14:59:40 [I|app|d9924adb] Backtrace for 'Action failed' error (TypeError): incompatible marshal file format (can't be read)
d9924adb | format version 4.8 required; 123.34 given
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/core_ext/marshal.rb:8:in `load'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/core_ext/marshal.rb:8:in `load'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:598:in `deserialize_entry'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:459:in `deserialize_entry'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:349:in `block in read_entry'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:347:in `read_entry'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:136:in `block in read_entry'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:78:in `block in fetch_entry'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:78:in `fetch'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:78:in `fetch_entry'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:134:in `read_entry'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:367:in `block in read'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `block in instrument'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:205:in `instrument'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `instrument'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:366:in `read'
d9924adb | /usr/share/foreman/app/services/ui_notifications/cache_handler.rb:10:in `payload'
d9924adb | /usr/share/foreman/app/controllers/notification_recipients_controller.rb:7:in `index'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:228:in `process_action'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
d9924adb | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
d9924adb | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
d9924adb | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
d9924adb | /usr/share/gems/gems/audited-5.3.2/lib/audited/sweeper.rb:16:in `around'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
d9924adb | /usr/share/gems/gems/audited-5.3.2/lib/audited/sweeper.rb:16:in `around'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:137:in `run_callbacks'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `block in instrument'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `instrument'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
d9924adb | /usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:165:in `process'
d9924adb | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:39:in `process'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:190:in `dispatch'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:254:in `dispatch'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:33:in `serve'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:50:in `block in serve'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `each'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `serve'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:842:in `call'
d9924adb | /usr/share/gems/gems/katello-4.9.0.16/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
d9924adb | /usr/share/gems/gems/katello-4.9.0.16/lib/katello/middleware/event_daemon.rb:10:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
d9924adb | /usr/share/gems/gems/apipie-rails-1.1.0/lib/apipie/static_dispatcher.rb:68:in `call'
d9924adb | /usr/share/gems/gems/apipie-rails-1.1.0/lib/apipie/extractor/recorder.rb:139:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
d9924adb | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
d9924adb | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
d9924adb | /usr/share/gems/gems/apipie-rails-1.1.0/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/tempfile_reaper.rb:15:in `call'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/etag.rb:27:in `call'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/conditional_get.rb:27:in `call'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/head.rb:12:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
d9924adb | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/session/abstract/id.rb:266:in `context'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/session/abstract/id.rb:260:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/cookies.rb:697:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
d9924adb | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:37:in `call_app'
d9924adb | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:28:in `call'
d9924adb | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
d9924adb | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/request_id.rb:26:in `call'
d9924adb | /usr/share/gems/gems/katello-4.9.0.16/lib/katello/prevent_json_parsing.rb:12:in `call'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/method_override.rb:24:in `call'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/runtime.rb:22:in `call'
d9924adb | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/sendfile.rb:110:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/ssl.rb:77:in `call'
d9924adb | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
d9924adb | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
d9924adb | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/engine.rb:539:in `call'
d9924adb | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `public_send'
d9924adb | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `method_missing'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/urlmap.rb:74:in `block in call'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/urlmap.rb:58:in `each'
d9924adb | /usr/share/gems/gems/rack-2.2.7/lib/rack/urlmap.rb:58:in `call'
d9924adb | /usr/share/gems/gems/puma-6.2.2/lib/puma/configuration.rb:270:in `call'
d9924adb | /usr/share/gems/gems/puma-6.2.2/lib/puma/request.rb:98:in `block in handle_request'
d9924adb | /usr/share/gems/gems/puma-6.2.2/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
d9924adb | /usr/share/gems/gems/puma-6.2.2/lib/puma/request.rb:97:in `handle_request'
d9924adb | /usr/share/gems/gems/puma-6.2.2/lib/puma/server.rb:431:in `process_client'
d9924adb | /usr/share/gems/gems/puma-6.2.2/lib/puma/server.rb:233:in `block in run'
d9924adb | /usr/share/gems/gems/puma-6.2.2/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
d9924adb | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-12-11T14:59:40 [I|app|d9924adb] Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (Duration: 5.8ms | Allocations: 6602)
2023-12-11T14:59:40 [I|app|d9924adb] Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 11.6ms | Allocations: 13073)
2023-12-11T14:59:40 [I|app|d9924adb] Completed 500 Internal Server Error in 56ms (Views: 12.4ms | ActiveRecord: 1.6ms | Allocations: 16113)

Expected results:

• The notification drawer should open

Additional info:

• This is not reproducible in Satellite 6.15 Snap 2.1
• As redis is considered a Tech Preview in Satellite 6.14 this bugzilla is just to document this known issue