Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: None
Affects Version/s: 39.0.1.Final
Component/s: MP Health
Labels:
None

[QE] How to address?:
---
[QE] Why QE missed?:
---

When something goes wrong during startup of WildFly, these errors are recorded as boot-errors and reported through the health checks. However, these errors are only reported as part of /health/ready not as part of /health/live. The result is that a server that failed to start will forever report itself as live but not ready. The live check returning UP will cause the server to not be restarted. As the boot-errors cannot be fixed, the server will remain in this state until manually restarted. IMHO, boot-errors should always be considered a liveness failure and result in the server being restarted.

In my particular instance, something went wrong initializing the infinispan cluster and the server reports this under /health/live:

{
  "status": "UP",
  "checks": [
    {
      "name": "keyhub-backend",
      "status": "UP"
    },
    {
      "name": "deployment.keyhub-web-authenticator.war.keyhub-frontend",
      "status": "UP"
    },
    {
      "name": "keyhub-cluster",
      "status": "UP"
    }
  ]
}

And this under /health/ready:

{
  "status": "DOWN",
  "checks": [
    {
      "name": "server-state",
      "status": "UP",
      "data": {
        "value": "running"
      }
    },
    {
      "name": "deployments-status",
      "status": "DOWN",
      "data": {
        "keyhub-ear.ear": "FAILED",
        "keyhub-manual.war": "OK",
        "keyhub-version-ear.ear": "OK",
        "keyhub-connector-ear.ear": "OK",
        "keyhub-web-authenticator.war": "OK",
        "keyhub-web-console.war": "OK",
        "keyhub-web-root.war": "OK"
      }
    },
    {
      "name": "suspend-state",
      "status": "UP",
      "data": {
        "value": "RUNNING"
      }
    },
    {
      "name": "boot-errors",
      "status": "DOWN",
      "data": {
        "boot-errors": "[{\"failed-operation\" : {\"operation\" : \"add\", \"address\" : [{ \"subsystem\" : \"infinispan\" },{ \"cache-container\" : \"server\" }]}, \"failure-description\" : \"{\\\"WFLYCTL0080: Failed services\\\" => {\\\"org.wildfly.clustering.infinispan.cache-container.cluster\\\" => \\\"org.infinispan.manager.EmbeddedCacheManagerStartupException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: java.util.concurrent.CompletionException: org.infinispan.commons.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1\\n    Caused by: org.infinispan.commons.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1\\\"}}\", \"failed-services\" : {\"org.wildfly.clustering.infinispan.cache-container.cluster\" : \"org.infinispan.manager.EmbeddedCacheManagerStartupException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: java.util.concurrent.CompletionException: org.infinispan.commons.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1\\n    Caused by: org.infinispan.commons.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1\"}},{\"failed-operation\" : {\"operation\" : \"add\", \"address\" : [{ \"subsystem\" : \"infinispan\" },{ \"cache-container\" : \"ejb\" }]}, \"failure-description\" : \"{\\\"WFLYCTL0080: Failed services\\\" => {\\\"org.wildfly.clustering.infinispan.cache-container.sfsb\\\" => \\\"org.infinispan.manager.EmbeddedCacheManagerStartupException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: java.util.concurrent.CompletionException: org.infinispan.commons.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1\\n    Caused by: org.infinispan.commons.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1\\\"}}\", \"failed-services\" : {\"org.wildfly.clustering.infinispan.cache-container.sfsb\" : \"org.infinispan.manager.EmbeddedCacheManagerStartupException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000659: Component ComponentWrapper{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.CONFIG@tkh-server-eac3721d59], status=FAILED} failed to start\\n    Caused by: java.util.concurrent.CompletionException: org.infinispan.commons.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1\\n    Caused by: org.infinispan.commons.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1\"}}]"
      }
    },
    {
      "name": "keyhub-backend",
      "status": "UP"
    },
    {
      "name": "ready-deployment.keyhub-web-root.war",
      "status": "UP"
    },
    {
      "name": "ready-deployment.keyhub-web-console.war",
      "status": "UP"
    },
    {
      "name": "ready-deployment.keyhub-connector-ear.ear.keyhub-connector-engine.jar",
      "status": "UP"
    },
    {
      "name": "ready-deployment.keyhub-version-ear.ear.keyhub-versioning.jar",
      "status": "UP"
    },
    {
      "name": "deployment.keyhub-web-authenticator.war.keyhub-frontend",
      "status": "UP"
    },
    {
      "name": "keyhub-cluster",
      "status": "UP"
    },
    {
      "name": "keyhub-available",
      "status": "UP"
    }
  ]
}

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates