Description of problem:
When investigating some performance/scalability/stability issues, we would like to have information about dynflow/sidekiq workers utilisation, to e.g. understand delays in polling external tasks, handing new tasks requests etc. That information would help us to properly scale up/down number of workers, for example.

As those data shall be ideally collected by sos report automatically, we need either a command for it (faster than foreman-rake) or a periodic logging of the load to a log file (or potentially another solution, if there is some).

From a discussion with Adam Růžička, potential implementations:

1) of command (via rake console, so not welcomed to be in sos):
cat <<SCRIPT | foreman-rake console
redis_url = 'redis://localhost:6379/0'
Sidekiq.redis =

{ url: redis_url }

stats = ::Sidekiq::Stats.new
puts "Queue items"
stats.queues.each do |queue, count|
puts "#

{queue}: #{count}"
end
puts
puts "Queue latencies"
stats.queues.each_key do |queue|
puts "#{queue}

: #

{Sidekiq::Queue.new(queue).latency}

"
end
SCRIPT

2) periodic logging of data like above, similarly like is done in /var/log/foreman-proxy/proxy.log every 30 minutes:

2023-02-13T03:27:47 [I] Execution plan cleaner removing 0 execution plans.

(very very ideally to have a (hard coded but editable-upon-request) option defining the frequency here)

Version-Release number of selected component (if applicable):
any

How reproducible:
100%

Steps to Reproduce:
n.a.

Actual results:
no easy way to get the load data

Expected results:
some automated way to collect the data

Additional info: