Issue:

• The customer is experiencing high latency in requests to 3scale products when the configuration is being loaded in their production and pre-production environments
• The customer utilizes APICAST_SERVICES_FILTER_BY_URL in their self-managed APIcast deployments
• Product count has been indicated to be greater than 200

Actions Taken

• Customer reported that after increasing the APIcast resources, request delay dropped from 30 seconds (which was causing timeouts) to approximately 10 seconds
• Confirmed that configurationLoadMode was set to boot for affected production APIcasts
• Requested customer create a product that does not utilize their custom policies for reproduction purposes
  • Customer created a new APIcast deployment and assigned a test product with no custom polices, which exhibited the same behavior
• Increased cacheConfigurationSeconds from 120 to 300.
  • The same latency spike was observed during config reload, and was observed at the new cacheConfigurationSeconds interval
• Configuration retrieval requests were observed  in system-app logs to be taking up to 2 seconds per page (2 pages total)
  • Customer horizontally scaled system-app, but no improvement was noted in config load times or delayed requests
• Customer performed the same request testing on the embedded APIcast, which exhibited the same behavior as the self-managed APIcasts

Impact: Increased latency is affecting the customer's production systems

Request:

• Requesting assistance understanding if any portion of the configuration reload process is expected to block requests
  • If this is expected, what changes can be made to reduce or eliminate the amount of request-blocking time
  • If this is not expected, requesting assistance determining why the requests are being delayed during configuration reload, and if there is a way to resolve this issue

Root cause analysis from Developer

Reloading a large configuration (by default, every five minutes) causes the worker to rebuild the whole policy chain for each service. During this process, incoming requests are stalled, with one customer reporting delays up to 12 seconds.