Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-10452

Prometheus metrics for System (for Rails) show incorrect numbers

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 2.15.0 GA
    • SaaS, 2.14.0 GA, 2.14.0-mas (0.11.8)
    • System
    • False
    • Hide

      None

      Show
      None
    • False
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started

      There is a bug in the setup of the Prometheus metrics for System, that affects specifically the Rails-related ones (not Sidekiq), and exposed at /yabeda-metrics.
      The problem is that each Unicorn worker collects and reports its own metrics, so what's aggregated in Prometheus is not correct (as counter metrics jump back and forth, the resulting numbers are wrong).

      We're using yabeda-prometheus-mmap based on prometheus-client-mmap which supports multi-process environments, but for that we need to set prometheus_multiproc_dir environment variable to ensure all the processes (e.g. all unicorn workers) report to aggregated metrics counters.

              Unassigned Unassigned
              rhn-support-dmayorov Daria Mayorova
              Matej Dujava Matej Dujava
              Daria Mayorova Daria Mayorova
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: