A number of times in the last few weeks we have seen the cost model XL queue growing abnormally quickly. The seeming result is Ford tasks, these tasks are taking an extremely long time to run distribution calculations.

The goal here is to understand exactly what parts of the SQL are running slowly and to attempt to make changes to improve these running times.

Right now if Ford run in multiple workers these have a tendency to slow each other since they are inserting into the same tables. The current work around to prevent a single customer blocking all other customers tasks we add the to the rate limit flag. This means they are limited to how many workers can be saturated. Though this helps others it also means we have a growing backlog for a single customer.

Lets start by investigating how/what is causing this then open a new issue to plan/fix the problem.