Because of the way that Openshift/Kubernetes create service link environment variables for all services in the same project, if you have too many services that start with the name "apicast-" it will eventually hit a limit in APIcast and throw an error.
Specifically APIcast is trying to parse all environment variables that start with APICAST_. Once you hit 100000 bytes worth it will throw the following error:
ERROR: /usr/local/share/lua/5.1/liquid.lua:2643: document fails length limit. length limit:100000 stack traceback: /usr/local/share/lua/5.1/liquid.lua:2643: in function 'check_length' /usr/local/share/lua/5.1/liquid.lua:2243: in function 'visit' /usr/local/share/lua/5.1/liquid.lua:1975: in function 'render' /opt/app-root/src/src/apicast/cli/command/start.lua:56: in function 'nginx_config' /opt/app-root/src/src/apicast/cli/command/start.lua:157: in function 'start' /opt/app-root/src/src/apicast/cli.lua:49: in function </opt/app-root/src/src/apicast/cli.lua:29> /tmp/o0F55m9Aat:60: in function 'file_gen' init_worker_by_lua:51: in function <init_worker_by_lua:49> [C]: in function 'xpcall' init_worker_by_lua:58: in function <init_worker_by_lua:56>
This is a common naming convention for customers that want to create several apicast deployments for isolation purposes.
At the very least I think we should see if we can set enableServiceLinks to false in the apicast-production and apicast-staging deployment configs because APIcast does not need these variables. This only works in OCP 4.x so this could be added to the operator but not the template based installations.