Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-16858

[OCP 4.1] Pod is not restarted when MP Health returns DOWN or UNDETERMINED

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • 7.2.3.GA, 7.3.0.Beta
    • 7.2.0.GA, 7.3.0.CD15
    • OpenShift
    • None

    Description

      Tested Scenario:
      Start deployment with DESIRED_STATE=DOWN (or UNDETERMINED) and register probes. Based on [1] readiness probes when returns "DOWN" in MP Health response restart should occure. This is happening on OCP 3.11 but is not happening on OCP 4.1.

      In EAP log I see

      [0m�[0m06:39:43,195 INFO [org.jboss.xpaas.microprofile.health.TestHealthCheck] (External Management Request Threads -- 2) Health Check called �[0m�[0m06:39:43,198 INFO [org.jboss.xpaas.microprofile.health.TestHealthCheck] (External Management Request Threads -- 1) Health Check called �[0m�[0m06:39:44,165 INFO [org.jboss.xpaas.microprofile.health.TestHealthCheck] (External Management Request Threads -- 1) Health Check called �[0m�[0m06:39:46,673 INFO [org.jboss.xpaas.microprofile.health.TestHealthCheck] (External Management Request Threads -- 1) Health Check called �[0m�[0m06:39:46,673 INFO [org.jboss.xpaas.microprofile.health.TestHealthCheck] (External Management Request Threads -- 2) Health Check called �[0m�[0m06:39:51,589 INFO [org.jboss.xpaas.microprofile.health.TestHealthCheck] (External Management Request Threads -- 1) Health Check called �[0m�[0m06:40:01,587 INFO [org.jboss.xpaas.microprofile.health.TestHealthCheck] (External Management Request Threads -- 1) Health Check called �[0m�[0m06:40:11,586 INFO [org.jboss.xpaas.microprofile.health.TestHealthCheck] (External Management Request Threads -- 1) Health Check called �[0m�[0m06:40:21,586 INFO [org.jboss.xpaas.microprofile.health.TestHealthCheck] (External Management Request Threads -- 1) Health Check called 
....

      In OCP web console I see Events 

      - Readiness probe failed: { "probe.eap.dmr.EapProbe": { "probe.eap.dmr.ServerStatusTest": "running", "probe.eap.dmr.DeploymentTest": { "ROOT.war": "OK" }, "probe.eap.dmr.BootErrorsTest": "No boot errors" }, "probe.eap.dmr.HealthCheckProbe": { "probe.eap.dmr.HealthCheckTest": "Status is DOWN" } } 

- Liveness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1

      Status of pod is Running but Readiness is ContainersNotReady

      When I try probes from terminal I see expected result

      sh-4.2$ /opt/eap/bin/livenessProbe.sh
{
    "probe.eap.dmr.EapProbe": {
        "probe.eap.dmr.ServerStatusTest": "running",
        "probe.eap.dmr.DeploymentTest": {
            "ROOT.war": "OK"
        },
        "probe.eap.dmr.BootErrorsTest": "No boot errors"
    },
    "probe.eap.dmr.HealthCheckProbe": {
        "probe.eap.dmr.HealthCheckTest": "Status is DOWN"
    }
}
sh-4.2$ /opt/eap/bin/readinessProbe.sh
{
    "probe.eap.dmr.EapProbe": {
        "probe.eap.dmr.ServerStatusTest": "running",
        "probe.eap.dmr.DeploymentTest": {
            "ROOT.war": "OK"
        },
        "probe.eap.dmr.BootErrorsTest": "No boot errors"
    },
    "probe.eap.dmr.HealthCheckProbe": {
        "probe.eap.dmr.HealthCheckTest": "Status is DOWN"
    }
}
sh-4.2$

      [1] https://issues.jboss.org/browse/CLOUD-2730?focusedCommentId=13616667&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13616667

      Attachments

        Issue Links

          is caused by

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) CLOUD-2742 [7.2.x-openjdk11] Allow kubernetes to control probe retries; avoid probes taking longer than kubernetes timeout settings

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • New
          relates to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-17464 Document required changes to liveness/readiness probes when migrating to OCP 4

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              kwills@redhat.com Ken Wills
              mchoma@redhat.com Martin Choma
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: