Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-42829

Recurring cookie store unavailable: :hash, error: key not found: :hash in Satellite under bulk RHSM load

XMLWordPrintable

    • False
    • Important
    • Proton Refinement Backlog
    • sat-proton
    • None
    • None
    • None
    • None

      Encountering recurring 500 Internal Server Error responses in Red Hat Satellite with the error cookie store unavailable: :hash, error: key not found: :hash recorded in production.log. The failures occur during RHSM API calls such as

      2026-02-18T12:44:13 [I|app|631372e1] Started GET "/rhsm/consumers/b414f942-6f8b-4a37-844f-8b9d4c6f5859/certificates/serials" for 10.197.176.44 at 2026-02-18 12:44:13 -0500 2026-02-18T12:44:13 [W|app|631372e1] cookie store unavailable: :hash, error: key not found: :hash 2026-02-18T12:44:13 [I|app|631372e1] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON 2026-02-18T12:44:13 [I|app|631372e1] Parameters: {"id"=>"b414f942-6f8b-4a37-844f-8b9d4c6f5859"} 2026-02-18T12:44:13 [E|kat|631372e1] ArgumentError: cookie store unavailable: :hash, error: key not found: :hash 2026-02-18T12:44:13 [I|app|631372e1] Completed 500 Internal Server Error in 33ms (Views: 0.3ms | ActiveRecord: 16.1ms | Allocations: 10640)

       

      The Satellite is sized and tuned for an extra-large profile (128 GB RAM, 32 vCPUs). The environment includes 20 Capsules and 7 load balancers. Approximately 3,861 hosts are registered directly to Satellite, with the remaining hosts registered through Capsules. The observed RHSM call volume per hour is enormous and comparable to other larger Satellite environments that do not experience this issue.

      The error appears when Satellite processes a bulk volume of RHSM requests. Restarting Satellite services is the solution for this issue.

      Observed RHSM Call Volume:

       

      9810 subscription-manager/1.28.44-1.el8_10
      4778 (cmd=yum)
      2418 subscription-manager/1.29.47.1-1.el9_7
      2098 (cmd=rhsmcertd-worker)
      463 subscription-manager/1.28.42-1.el8
      248 (cmd=yum-cron)
      234 (cmd=rhsmd)
      146 (cmd=subscription-manager)
      64 subscription-manager/1.29.45.1-1.el9_6
      43 subscription-manager/1.29.40.1-1.el9_4

      In this case, the problem surfaced roughly one year after the upgrade, making the [KCS|http://access.redhat.com/solutions/7117482] recommendation to revert a snapshot and retry the upgrade not applicable.

      The customer is requesting two specific outcomes:

      i. A clear root cause analysis explaining why the cookie store becomes unavailable in an environment without virt-who and what condition is triggering it.

      ii. Guidance on a permanent resolution—whether through a code fix, configuration change, or additional tuning—to prevent recurrence instead of relying on service restarts.

              Unassigned Unassigned
              rhn-support-sahsharm Sahil Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: