IMPORTANT: Please see THREESCALE-2468 and do not duplicate the work done there. See Dev Notes.

The goal is to use yabeda-sidekiq to have better prometheus metrics for our sidekiq jobs.
Basically we want more metrics segmented per jobs and queues.

Yabeda sidekiq describes the metrics as:

• Total number of executed jobs: sidekiq_jobs_executed_total - (segmented by queue and class name)
• Number of jobs have been finished successfully: sidekiq_jobs_success_total (segmented by queue and class name)
• Number of jobs have been failed: sidekiq_jobs_failed_total (segmented by queue and class name)
• Time of job run: sidekiq_job_runtime (seconds per job execution, segmented by queue and class name)
• Time of the queue latency sidekiq_jobs_latency (the difference in seconds since the oldest job in the queue was enqueued)
• Number of jobs in queues: sidekiq_jobs_waiting_count (segmented by queue)
• Number of scheduled jobs: sidekiq_jobs_scheduled_count
• Number of jobs in retry set: sidekiq_jobs_retry_count
• Number of jobs in dead set (“morgue”): sidekiq_jobs_dead_count
• Active workers count: sidekiq_active_processes
• Active processes count: sidekiq_active_workers_count

Where are the metrics exposed:

Each sidekiq process exposes a webserver listening on port 9394 by default.
If more processes are spawn, the port number will be incremented by the index option of the sidekiq process. See this commit

Dev notes

Sidekiq generates metrics with both old and new targets, together
But the new targets(sidekiq_jobs_***) are only available for sidekiq 6 and above
So, sidekiq 5.x generates targets with sidekiq_job, where sidekiq 6.x generates sidekiq_job and sidekiq_jobs

3 metrics were implemented on https://issues.redhat.com/browse/THREESCALE-2468 for sidekiq 5

• sidekiq_job_runtime_seconds_bucket
• sidekiq_job_runtime_seconds_count
• sidekiq_job_runtime_seconds_sum

This ticket is about updating the yabeda-sidekiq gem and implement the other described metrics