Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-6446

3scale monitoring grafana/prometheus issues

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 2.10 CR1
    • System
    • False
    • False
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • +
    • Undefined

      System dashboard:

      1)

      stats requests per second + heatmap

      Sending request to the backend listener internal api endpoint “internal/services/stats” does not produce a response in the grafana stats graph. Instead, the requests seems to be counted as services requests.

      curl -H "Authorization: Basic_ {Base64Encoded Username and Password} _" -k -v  -X GET "https://backend-3scale.thr-met.apps.example-cluster.com/internal/services/4/stats"
      

      2)

      usage limits requests per second + heatmap

      Sending request to the backend listener internal api endpoint “internal/services/

      {service_id}/plans/{plan_id}/usagelimits” does not produce a response in the grafana Usage Limits graph. Instead, the requests seems to be counted as services requests.
      curl -H "Authorization: Basic {Base64Encoded Username and Password}" -k -v  -X GET "https://backend-3scale.thr-met.apps.example-cluster.com/internal/services/4/plans/36/usagelimits"
      


      3)
      utilization requests per second + heatmap

      Sending request to the backend listener internal api endpoint “internal/services/{service_id}

      /applications/

      {app_id}

      /utilization” does not produce a response in the grafana Utilization graph. It is counted as application requests.

      curl -H "Authorization: Basic {Base64Encoded Username and Password}" -k -v  -X GET "https://backend-3scale.thr-met.apps.example-cluster.redhat.com/internal/services/4/applications/12/utilization"
      

      System dashboard

      4)

      system 5xx request per sec

      rails_requests_total

      {status=~'5[0-9]*'}

      Bug
      Tried to upload logo into developer portal without running minio - causes 500 error in UI, also seen in system-logs
      Prometheus didn’t register any traffic
      **

      5)

      notification requests per second

      rails_request_duration_seconds_bucket

      {controller='notifications'}

      Created and deleted couple of services
      it seems that there is wrong pod selected in query - it might be zync instead of system-app
      **

      6)

      tenants requests per second

      rails_request_duration_seconds_bucket

      {controller='tenants'}

      it seems that there is wrong pod selected in query - it might be zync instead of system-app
      **

      7)

      queue sizes

      Sidekiq_queue_latency
      Created and deleted couple of services
      Saw traffic ‘Jobs processed by queue’ and ‘Jobs pushed by queue’ but Prometheus didn’t register any traffic in queue sizes hence i assumed that it’s not working properly
      **

      8)

      queue latencies

      sidekiq_jobs_waiting_count
      Created and deleted couple of services
      Saw traffic ‘Jobs processed by queue’ and ‘Jobs pushed by queue’ but Prometheus didn’t register any traffic in queue latencies hence i assumed that it’s not working properly

            Unassigned Unassigned
            ficap_devel Filip Čáp (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: