Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-110

Pods are continuously crash looping

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • None
    • 2.0 ER4
    • Backend, Gateway, Infra, Memcached, System
    • None

      Issue Description:
      When running OpenShift in a resource constrained virtual machine, the 3scale pods end up in a crash loop until the containers fail to start. This mainly seems to impact the apicast-production and system-app deployments.

      Reason:
      The crash loop occurs due to the health checks in place for the apicast-production and system-app deployments. More specifically, the liveness and readiness probes [1] do not allocate sufficient time for the containers to fully start - and therefore mistakenly assume that they failed to start properly. This results in a redeployment of the pod, and a loop that eventually triggers a full 3scale deployment failure.

      Solution:
      Assuming that virtual machine resource constraints cannot be remedied, one can increase the value of the initialDelaySeconds for each deployment config that is affected to allow for additional time for the containers to start properly; the change would need need to be applied to both the readiness and liveness probes for the container.

      Additional details regarding health checks and probes can be found at the link below:
      [1] https://docs.openshift.com/enterprise/3.1/dev_guide/application_health.html

              Unassigned Unassigned
              vramosp Vanessa Ramos (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: