Liquid statement looping through large number of services introduces unacceptable page load time

Please don't start working on this issue before THREESCALE-4053 and THREESCALE-401 are done.

Current behaviour

Looping through all service objects then all plan objects then any children objects of the plan causes a very long response time in the developer portal page load time for the user.

Expected behaviour

Looping through any number of objects results in an acceptable response time from the application and thus doesn't significantly impact page load time. The alternative would be to provide some way for the API provider to not have to loop through all objects and therefore reduce latency incurred by the liquid evaluation.

Some context

The customer wants to be able to render all application plans for all services on a single page but, due to the large number of services and plans within those services and metrics & limits within those plans, all those nested loops incur a huge amount of processing, hence the timeout experienced. The customer expects this to be possible in a reasonable amount of time from a UX point of view so the question is: "If we deem this to be unreasonable then we should decide what is supported and document that and provide examples?" OR "If we want to support this, is it a case of upgrading to Liquid 4, paginating the service collection and exposing that pagination via Liquid tags to the user?" Maybe once we are able to answer this it might be clearer how to define and move the issue forward.

Dev Notes
Upgrade liquid and implement pagination in the controller.
Customer is on SaaS. Pagination is the key. Liquid paginate. Pagination is only available in built-in pages.
The solution seems to be to upgrade Liquid.
Please read all comments before starting work on this issue.

Definition/Investigation

 We need to have some performance numbers:

• Reproduce the issue with the setup context
• Know whether it is SQL or rendering which takes time
• Breakdown the execution stack with the duration spent in each step:
  • flame graph or other tools permits us to find some bottleneck see for e.g. https://github.com/oozou/ruby-prof-flamegraph
  • ruby profiler https://github.com/ruby-prof/ruby-prof

From there we need to isolate the components that make the generation slow.

• Maybe we load too many objects, that are linked to the drop (plans, features, and nested drops of those objects)
• Maybe it is totally normal that hundreds of objects can't be rendered in time (less than the web server, unicorn, timeout)
• Maybe some SQL are too slow taking around 1 seconds to load an object, which could be multiplied thus ending with a timeout.

Maybe it is a real issue with Liquid, fixed in further versions

There are too many possibilities but those are the pointers we should start from.