Hitting timeouts on a SDN to OVN migrated cluster when viewing Kubernetes/Compute Resources/Cluster Dashboard for custom range of multiple days

When viewing Kubernetes/Compute Resources/Cluster Dashboard for custom range of multiple days for a migrated cluster, continuously hitting timeouts and not able to see the data.   

4.16.0-0.nightly-2024-04-03-065948

Always

    1. Create a 4.14 Classic Rosa 24 node cluster or 120 node cluster 
    2. Load it with cluster-density-v2 
    3. Upgrade cluster to 4.15 nightly and then to 4.16 nightly
    4. Migrate cluster from SDN to OVN
    5. Few hours after migration is complete and cluster is stable, try to view the above dashboard and notice the errors.

Viewing dashboard for last 30 min works fine, but if you try to view for larger amount of data - 24 hours or multiple days or sometimes 12 hours, you will hit the errors.
    

An error occurred 
Call to /api/prometheus/api/v1/query_range?start=1712030400.001&end=1712289540&step=25914&query=sum%28container_memory_rss%7Bjob%3D%22kubelet%22%2C+metrics_path%3D%22%2Fmetrics%2Fcadvisor%22%2C+cluster%3D%22%22%2C+container%21%3D%22%22%7D%29+by+%28namespace%29&timeout=60s timed out after 60000ms.    

User should be able to view the dashboard without hitting timeouts.    

must-gather is collected