Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-2468

Better prometheus metrics for Sidekiq

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Major Major
    • None
    • None
    • System

      The goal is to use yabeda-sidekiq to have better prometheus metrics for our sidekiq jobs.
      Basically we want more metrics segmented per jobs and queues.

      Yabeda sidekiq describes the metrics as:

      • Total number of executed jobs: sidekiq_jobs_executed_total - (segmented by queue and class name)
      • Number of jobs have been finished successfully: sidekiq_jobs_success_total (segmented by queue and class name)
      • Number of jobs have been failed: sidekiq_jobs_failed_total (segmented by queue and class name)
      • Time of job run: sidekiq_job_runtime (seconds per job execution, segmented by queue and class name)
      • Time of the queue latency sidekiq_jobs_latency (the difference in seconds since the oldest job in the queue was enqueued)
      • Number of jobs in queues: sidekiq_jobs_waiting_count (segmented by queue)
      • Number of scheduled jobs: sidekiq_jobs_scheduled_count
      • Number of jobs in retry set: sidekiq_jobs_retry_count
      • Number of jobs in dead set (“morgue”): sidekiq_jobs_dead_count
      • Active workers count: sidekiq_active_processes
      • Active processes count: sidekiq_active_workers_count

      Where are the metrics exposed:

      Each sidekiq process exposes a webserver listening on port 9394 by default.
      If more processes are spawned, the port number will be incremented by the index option of the sidekiq process. See this commit

      Dev notes

      Sidekiq generates metrics with both old and new targets, together
      But the new targets(sidekiq_jobs_***) are only available for sidekiq 6 and above
      So, sidekiq 5.x generates targets with sidekiq_job, where sidekiq 6.x generates sidekiq_job and sidekiq_jobs

      Only 3 metrics are present since the latest release of the gem was done on October 25, 2018

      • sidekiq_job_runtime_seconds_bucket
      • sidekiq_job_runtime_seconds_count
      • sidekiq_job_runtime_seconds_sum

      We decide to only implement those 3 metrics. 
      THREESCALE-5786 is about implementing the others 

              Unassigned Unassigned
              hramihaj Hery Ramihajamalala (Inactive)
              Martin Kudlej Martin Kudlej
              Hery Ramihajamalala Hery Ramihajamalala (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: