Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-6315

Too Many "apicast-" services in the same OCP project cause APIcast to error

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 2.4 GA, 2.5 GA, 2.6 GA, 2.7 GA, 2.8 GA, 2.9.1 GA
    • Gateway
    • False
    • False
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • +
    • Undefined
    • Workaround Exists
    • Hide

      There is a simple workaround in OCP 4.x. Edit the apicast deployment config and set enableServiceLinks to false on the pod spec.

      There only workaround in OCP 3.x is to delete extraneous services or name them differently.

      Show
      There is a simple workaround in OCP 4.x. Edit the apicast deployment config and set enableServiceLinks to false on the pod spec . There only workaround in OCP 3.x is to delete extraneous services or name them differently.
    • Hide
      1. Deploy AMP 2.9
      2. Create a bunch of services that start with "apicast-"
        e.g. `for i in $(seq 1 50); do oc create service clusterip "apicast-$i" --tcp=8080:8080; done`
      3. Redeploy apicast-staging
      4. Observe error in apicast-staging logs
      Show
      Deploy AMP 2.9 Create a bunch of services that start with "apicast-" e.g. `for i in $(seq 1 50); do oc create service clusterip "apicast-$i" --tcp=8080:8080; done` Redeploy apicast-staging Observe error in apicast-staging logs

      Because of the way that Openshift/Kubernetes create service link environment variables for all services in the same project, if you have too many services that start with the name "apicast-" it will eventually hit a limit in APIcast and throw an error.

      Specifically APIcast is trying to parse all environment variables that start with APICAST_. Once you hit 100000 bytes worth it will throw the following error:

      ERROR: /usr/local/share/lua/5.1/liquid.lua:2643: document fails length limit. length limit:100000
      stack traceback:
      	/usr/local/share/lua/5.1/liquid.lua:2643: in function 'check_length'
      	/usr/local/share/lua/5.1/liquid.lua:2243: in function 'visit'
      	/usr/local/share/lua/5.1/liquid.lua:1975: in function 'render'
      	/opt/app-root/src/src/apicast/cli/command/start.lua:56: in function 'nginx_config'
      	/opt/app-root/src/src/apicast/cli/command/start.lua:157: in function 'start'
      	/opt/app-root/src/src/apicast/cli.lua:49: in function </opt/app-root/src/src/apicast/cli.lua:29>
      	/tmp/o0F55m9Aat:60: in function 'file_gen'
      	init_worker_by_lua:51: in function <init_worker_by_lua:49>
      	[C]: in function 'xpcall'
      	init_worker_by_lua:58: in function <init_worker_by_lua:56> 

      This is a common naming convention for customers that want to create several apicast deployments for isolation purposes.

      At the very least I think we should see if we can set enableServiceLinks to false in the apicast-production and apicast-staging deployment configs because APIcast does not need these variables. This only works in OCP 4.x so this could be added to the operator but not the template based installations.

              Unassigned Unassigned
              rhn-support-spoole Shannon Poole
              David Rajnoha David Rajnoha (Inactive)
              Eloy Coto Eloy Coto
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: