Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-6446

3scale monitoring grafana/prometheus issues

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Done
    • 2.10 CR1
    • None
    • System

    Description

      System dashboard:

      1)

      stats requests per second + heatmap

      Sending request to the backend listener internal api endpoint “internal/services/stats” does not produce a response in the grafana stats graph. Instead, the requests seems to be counted as services requests.

      curl -H "Authorization: Basic_ {Base64Encoded Username and Password} _" -k -v  -X GET "https://backend-3scale.thr-met.apps.example-cluster.com/internal/services/4/stats"
      

      2)

      usage limits requests per second + heatmap

      Sending request to the backend listener internal api endpoint “internal/services/

      {service_id}/plans/{plan_id}/usagelimits” does not produce a response in the grafana Usage Limits graph. Instead, the requests seems to be counted as services requests.
      curl -H "Authorization: Basic {Base64Encoded Username and Password}" -k -v  -X GET "https://backend-3scale.thr-met.apps.example-cluster.com/internal/services/4/plans/36/usagelimits"
      


      3)
      utilization requests per second + heatmap

      Sending request to the backend listener internal api endpoint “internal/services/{service_id}

      /applications/

      {app_id}

      /utilization” does not produce a response in the grafana Utilization graph. It is counted as application requests.

      curl -H "Authorization: Basic {Base64Encoded Username and Password}" -k -v  -X GET "https://backend-3scale.thr-met.apps.example-cluster.redhat.com/internal/services/4/applications/12/utilization"
      

      System dashboard

      4)

      system 5xx request per sec

      rails_requests_total

      {status=~'5[0-9]*'}

      Bug
      Tried to upload logo into developer portal without running minio - causes 500 error in UI, also seen in system-logs
      Prometheus didn’t register any traffic
      **

      5)

      notification requests per second

      rails_request_duration_seconds_bucket

      {controller='notifications'}

      Created and deleted couple of services
      it seems that there is wrong pod selected in query - it might be zync instead of system-app
      **

      6)

      tenants requests per second

      rails_request_duration_seconds_bucket

      {controller='tenants'}

      it seems that there is wrong pod selected in query - it might be zync instead of system-app
      **

      7)

      queue sizes

      Sidekiq_queue_latency
      Created and deleted couple of services
      Saw traffic ‘Jobs processed by queue’ and ‘Jobs pushed by queue’ but Prometheus didn’t register any traffic in queue sizes hence i assumed that it’s not working properly
      **

      8)

      queue latencies

      sidekiq_jobs_waiting_count
      Created and deleted couple of services
      Saw traffic ‘Jobs processed by queue’ and ‘Jobs pushed by queue’ but Prometheus didn’t register any traffic in queue latencies hence i assumed that it’s not working properly

      Attachments

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              Unassigned Unassigned
              ficap_devel Filip Čáp
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: