Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-6446

3scale monitoring grafana/prometheus issues

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • 2.10 CR1
    • System
    • False
    • False
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • +
    • Undefined

    Description

      System dashboard:

      1)

      stats requests per second + heatmap

      Sending request to the backend listener internal api endpoint “internal/services/stats” does not produce a response in the grafana stats graph. Instead, the requests seems to be counted as services requests.

      curl -H "Authorization: Basic_ {Base64Encoded Username and Password} _" -k -v  -X GET "https://backend-3scale.thr-met.apps.example-cluster.com/internal/services/4/stats"

      2)

      usage limits requests per second + heatmap

      Sending request to the backend listener internal api endpoint “internal/services/

      {service_id}/plans/{plan_id}/usagelimits” does not produce a response in the grafana Usage Limits graph. Instead, the requests seems to be counted as services requests.
      curl -H "Authorization: Basic {Base64Encoded Username and Password}" -k -v  -X GET "https://backend-3scale.thr-met.apps.example-cluster.com/internal/services/4/plans/36/usagelimits"


      3)
      utilization requests per second + heatmap

      Sending request to the backend listener internal api endpoint “internal/services/{service_id}

      /applications/

      {app_id}

      /utilization” does not produce a response in the grafana Utilization graph. It is counted as application requests.

      curl -H "Authorization: Basic {Base64Encoded Username and Password}" -k -v  -X GET "https://backend-3scale.thr-met.apps.example-cluster.redhat.com/internal/services/4/applications/12/utilization"

      System dashboard

      4)

      system 5xx request per sec

      rails_requests_total

      {status=~'5[0-9]*'}

      Bug
      Tried to upload logo into developer portal without running minio - causes 500 error in UI, also seen in system-logs
      Prometheus didn’t register any traffic
      **

      5)

      notification requests per second

      rails_request_duration_seconds_bucket

      {controller='notifications'}

      Created and deleted couple of services
      it seems that there is wrong pod selected in query - it might be zync instead of system-app
      **

      6)

      tenants requests per second

      rails_request_duration_seconds_bucket

      {controller='tenants'}

      it seems that there is wrong pod selected in query - it might be zync instead of system-app
      **

      7)

      queue sizes

      Sidekiq_queue_latency
      Created and deleted couple of services
      Saw traffic ‘Jobs processed by queue’ and ‘Jobs pushed by queue’ but Prometheus didn’t register any traffic in queue sizes hence i assumed that it’s not working properly
      **

      8)

      queue latencies

      sidekiq_jobs_waiting_count
      Created and deleted couple of services
      Saw traffic ‘Jobs processed by queue’ and ‘Jobs pushed by queue’ but Prometheus didn’t register any traffic in queue latencies hence i assumed that it’s not working properly

      Attachments

        Issue Links

          is related to

          Bug - A problem that impairs or prevents the functions of the product. THREESCALE-6403 System Grafana Dashboards heatmap negative values

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          relates to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. THREESCALE-5722 Make monitoring resources available through the 3scale Operator (Phase 2)

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Feature Request - Feature requests from customers and/or users THREESCALE-4743 System grafana dashboard(s) and Prometheus rule definition (system)

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Feature Request - Feature requests from customers and/or users THREESCALE-4744 Zync components grafana dashboards and prometheus rules design (zync)

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. THREESCALE-6508 3scale monitoring grafana/prometheus issue - swapped descriptions

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              Unassigned Unassigned
              ficap_devel Filip Čáp
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: