Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-21305

"Actions::Katello::Applicability::Hosts::BulkGenerate" tasks are processed in the default queue instead of hosts_queue causing congestion.

XMLWordPrintable

    • 0
    • False
    • Important
    • Sprint 125, Sprint 127, Sprint 128, Sprint 129, Sprint 130, Sprint 131, Sprint 132
    • sat-proton
    • None
    • None
    • None
    • None

      Description of problem:
      "Actions::Katello::Applicability::Hosts::BulkGenerate" tasks can cause congestion in the Dynflow "default" queue, especially with this performance bug 2203077. The tasks are supposed to process by the "hosts_queue" worker but they go to the default queue instead.

      This is causing slowness to other tasks which need to be processed by the default queue worker. Content view publish task is hanging waiting the task status to be polled from Pulp.

      How reproducible:
      Easy

      Steps to Reproduce:
      1. Go to Web UI -> Hosts -> Content Hosts -> click in to the host -> Content -> Errata -> click "recalculate"
      2. Then on the Satellite command line, run the following command

      watch -n 0.1 systemctl status dynflow-sidekiq@worker-1.service

      Actual results:
      You should notice that 1 thread is busy like below:

      1. systemctl status dynflow-sidekiq@worker-1.service
        ● dynflow-sidekiq@worker-1.service - Foreman jobs daemon - worker-1 on sidekiq
        Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
        Active: active (running) since Wed 2023-11-15 00:40:26 UTC; 3h 14min ago
        Docs: https://theforeman.org
        Main PID: xxxxx (sidekiq)
        Status: "Everything ready for world: 57b50c94-dbfe-4271-8033-76bc74138212"
        Tasks: 15 (limit: 126231)
        Memory: 1.5G
        CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@worker-1.service
        └─1104255 sidekiq 6.3.1 [1 of 5 busy] <==================

      Expected results:
      Expect to see 1 thread to busy in "worker-hosts-queue-1" worker.

      Additional info:
      Setting['host_tasks_workers_pool_size'] is used during the initialization but Katello settings are still not loaded at that time. Therefore, the value is always 0 and the condition will never met.

      1. In lib/katello/engine.rb
        ~~~
        initializer "katello.register_actions", :before => :finisher_hook do |_app|
        ForemanTasks.dynflow.require!
        if (Setting.table_exists? rescue(false)) && Setting['host_tasks_workers_pool_size'].to_i > 0 <===============================
        ForemanTasks.dynflow.config.queues.add(HOST_TASKS_QUEUE, :pool_size => Setting['host_tasks_workers_pool_size'])
        end
        ~~~

      We can also see the below warning in the production.log
      ~~~
      2023-11-14T07:30:27 [W|app|] Setting host_tasks_workers_pool_size has no definition, please define it before using
      ~~~

      1. In "/usr/share/foreman/app/registries/foreman/setting_manager.rb" we can see the plugin settings will only be fully loaded in "config.to_prepare" state.
        ~~~
        Rails.application.config.to_prepare do
        Foreman::SettingManager.validations.setup!
        end
        ~~~

              rhn-support-hyu Hao Chang Yu
              jira-bugzilla-migration RH Bugzilla Integration
              Vijaykumar Sawant Vijaykumar Sawant
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 day, 4 hours
                  1d 4h