Based on the results of the spike as detailed in this document: https://docs.google.com/document/d/1EPEYd94NYS_LbFRFT4d1Mj9-ebibNYJYu_kHzwIwxaA/edit?usp=sharing we need to implement one of the suggested solutions for dashboards.

This story will focus on fixing timeout issues for label selectors and line charts. This will be done through breaking the long queries down into queries which are able to be responded to within the timeout period.

Suggested steps are:

• Perform query chunking on the panels based on time ranges. This is needed for label and line charts
  • Rather than wait for a query to timeout, we should preemptively split the query into manageable chunks (1 day)
• For the the line charts, rather than replacing all of the data for a single fetch, instead the data should replace its specific time period or all of the fetches should be aggregated before replacing all
  • For segment replacement, this should require retrieving all data for a panel, removing the new data’s time period, adding in the new data and then bounding the data for the full time range to prevent data sprawl if the webpage is left open for a long time.
  • Full replacement will work the same as the system currently works, but require waiting for the results of all the requests before combining them all and replacing the current data. This would overall be a simpler approach, but could leave potentially blank areas in a chart if one or more of the queries timeout where there was already data from a previous API call
• For the label selectors, the labels for each of the fetches should be combined before replacing the existing data.