Rate limit requests (Multitenant API)

Issues

Our Multitenant API is not rate limited. Though in SaaS customers are contracted to respect some limits, there is no clear definition of how many requests per day/hour/minute customers can make. Rate limiting will be done by ops team only at the network level so....should we also have rate limiting at the account level? That's what this issue is for.

This is troublesome because sometimes it overloads our web servers and database.
At worst leading to DoS and outages.

Another part is that some malicious scripts are run against functionalities of the UI (admin portal and developer portal) and we have no way to protect ourselves against that.

Definition

TODO: Needs definition by Engineering, Product and Operations

• Limits per day, per hour, per minute, per plan? on the API side
  For example, if the limit is X per day, then we could maybe extrapolate the following limits:

• X/24 per hour
• X/24/10 per minute (more bandwidth per minute)

If customer wants more bandwidth, it will need support calls to adjust per hour and per minute.

• where to implement the limits: in OCP? in the application? both? Think about on-premises
• how to communicate this to customer? and when they reach the limits?
• Detection of "Too many request" on the other endpoints of the UI (admin portal and developer portal) and implement account suspension based on some rules. The application would probably need access to the metrics. There is no "limit" on UI side but we should detect malicious behaviour