Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-5786

Better prometheus metrics for Sidekiq version 2

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Major
    • None
    • None
    • System
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started

    Description

      IMPORTANT: Please see THREESCALE-2468 and do not duplicate the work done there. See Dev Notes.

      The goal is to use yabeda-sidekiq to have better prometheus metrics for our sidekiq jobs.
      Basically we want more metrics segmented per jobs and queues.

      Yabeda sidekiq describes the metrics as:

      • Total number of executed jobs: sidekiq_jobs_executed_total - (segmented by queue and class name)
      • Number of jobs have been finished successfully: sidekiq_jobs_success_total (segmented by queue and class name)
      • Number of jobs have been failed: sidekiq_jobs_failed_total (segmented by queue and class name)
      • Time of job run: sidekiq_job_runtime (seconds per job execution, segmented by queue and class name)
      • Time of the queue latency sidekiq_jobs_latency (the difference in seconds since the oldest job in the queue was enqueued)
      • Number of jobs in queues: sidekiq_jobs_waiting_count (segmented by queue)
      • Number of scheduled jobs: sidekiq_jobs_scheduled_count
      • Number of jobs in retry set: sidekiq_jobs_retry_count
      • Number of jobs in dead set (“morgue”): sidekiq_jobs_dead_count
      • Active workers count: sidekiq_active_processes
      • Active processes count: sidekiq_active_workers_count

      Where are the metrics exposed:

      Each sidekiq process exposes a webserver listening on port 9394 by default.
      If more processes are spawn, the port number will be incremented by the index option of the sidekiq process. See this commit

      Dev notes

      Sidekiq generates metrics with both old and new targets, together
      But the new targets(sidekiq_jobs_***) are only available for sidekiq 6 and above
      So, sidekiq 5.x generates targets with sidekiq_job, where sidekiq 6.x generates sidekiq_job and sidekiq_jobs

      3 metrics were implemented on https://issues.redhat.com/browse/THREESCALE-2468 for sidekiq 5

      • sidekiq_job_runtime_seconds_bucket
      • sidekiq_job_runtime_seconds_count
      • sidekiq_job_runtime_seconds_sum

      This ticket is about updating the yabeda-sidekiq gem and implement the other described metrics

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              hramihaj Hery Ramihajamalala (Inactive)
              Martin Kudlej Martin Kudlej
              Srijita Mukherjee Srijita Mukherjee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: