Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-29353

The resulting dataset is too large to graph

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.12.z
    • Observability UI
    • None
    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      When having for example 1300 pods in the same namespace, the dashboard     in OpenShift Console > Observe > Dashboards > Dashboard: Kubernetes / Compute Resources / Namespace (Pods) and Namespace: <namespace containing 1300 pods> fails to be displayed with the error "The resulting dataset is too large to graph
      
      The same result is obtained when trying to list the cpu for all the pods in the same namespace. 

      Version-Release number of selected component (if applicable):

      OCP 4.12

      How reproducible:

      Always

      Steps to Reproduce:

      1. Create a namespace containing 1300 pods:
      ```
      $ oc new-project metrics-test
      $ for number in `seq 1 1300`; do kubectl create deployment hello-node-$number --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname; sleep 1; done```     
        
      
      2. Verify that 1300 pods are running
      ```
      $ oc get pods -n metrics-test |grep -c Running
      1300
      ```
      
      Wait some minutes and go to the `OpenShift Console > Observe > Dashboards > Dashboard: Kubernetes / Compute Resources / Namespace (Pods) and Namespace: metrics-test`
          

      Actual results:

      Fails to display the metrics for the pods in the namespace with the error "The resulting dataset is too large to graph

       

      Expected results:

      The graph `OpenShift Console > Observe > Dashboards > Dashboard: Kubernetes / Compute Resources / Namespace (Pods) and Namespace: metrics-test displays the metrics for the pods. 

      Additional info:

      If the same query is run against the prometheus pods, the result is returned: 
      
      ```
      $ oc exec -c prometheus prometheus-k8s-0 -- curl -s --data-urlencode 'query=pod:container_cpu_usage:sum{pod=~"hello-node-.*",namespace="metrics-test"}'  --data-urlencode "&start=2024-02-09T00:00:00.781Z" --data-urlencode "end=2024-02-10T00:00:00.781Z"  'http://localhost:9090/api/v1/query?' 
      ```

      Similar bugs in the past for when returning so many metrics, for example for storage:

            gbernal@redhat.com Gabriel Bernal
            rhn-support-ocasalsa Oscar Casal Sanchez
            Junqi Zhao Junqi Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: