Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-17099

Clients fails on "java.net.SocketTimeoutException: Read timed out" after killing and restart EAP pod in cluster

    XMLWordPrintable

Details

    • Hide

      Follow those steps to run test against OCP 4 cluster - NOT 100% reproducer:

      git clone git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/openshift-eap-tests.git
      cd openshift-eap-tests
      
      # create test.properties - change properties based on last build in https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-create-openshift-4-cluster
      xtf.openshift.url=...
      xtf.openshift.token=...
      xtf.openshift.admin.token=...
      xtf.config.master.ssh_key_path=/home/hudson/.ssh/id_rsa_hudson # see internal comment for hudson private key, create file with its content and change path to this file
      xtf.config.master.jump.ssh_hostname=...
      xtf.openshift.namespace=user-mynamespace
      xtf.bm.namespace=xtf-builds-premens
      xtf.openshift.version=4.1.0-201904090034.git.0.1016bda.el7
      xtf.config.master.ssh_username=core
      xtf.config.master.jump.ssh_username=centos
      xtf.ocp4x=true
      
      
      Start test by:
      mvn clean test -P72,load -Dtest=HAServletCounterUpstreamDnsPingTest#dockerKillOneOfThree
      
      Show
      Follow those steps to run test against OCP 4 cluster - NOT 100% reproducer : git clone git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/openshift-eap-tests.git cd openshift-eap-tests # create test.properties - change properties based on last build in https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-create-openshift-4-cluster xtf.openshift.url=... xtf.openshift.token=... xtf.openshift.admin.token=... xtf.config.master.ssh_key_path=/home/hudson/.ssh/id_rsa_hudson # see internal comment for hudson private key, create file with its content and change path to this file xtf.config.master.jump.ssh_hostname=... xtf.openshift.namespace=user-mynamespace xtf.bm.namespace=xtf-builds-premens xtf.openshift.version=4.1.0-201904090034.git.0.1016bda.el7 xtf.config.master.ssh_username=core xtf.config.master.jump.ssh_username=centos xtf.ocp4x= true Start test by: mvn clean test -P72,load -Dtest=HAServletCounterUpstreamDnsPingTest#dockerKillOneOfThree

    Description

      Clients fails on "java.net.SocketTimeoutException: Read timed out" after killing and restart EAP pod in cluster. This issue was on OCP 4.1 but might not be directly related.

      Customer Impact: In case that any of pods crashes and Openshift starts new one, all HTTP clients (accross all pods in cluster) may start to fail on java.net.SocketTimeoutException: Read timed out.

      Test scenario:

      • Start 3 EAP pods and wait until all are ready with deployed servlet. (there is no wait for forming cluster)
      • Start 64 clients which are repeatedly (every 5 seconds) calling servlet and incrementing counter in its session, clients are load balanced to all pods.
        • Every-time client calls servlet, it gets counter increased by one.
      • Randomly kill one of the pods (there is used "runc" command to kill container in pod)

      Expected result:
      New pod with the same will be started. Clients will failover to remaining pods or reconnect to the same pod. Client session will not be reset.

      Actual result:
      Sometimes happens that all clients starts to get:

      java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out
              at com.redhat.xpaas.eap.ha.syntest.HttpSynTestUtils.lambda$httpRequestVerifier$5(HttpSynTestUtils.java:258)
              at com.redhat.xpaas.eap.ha.syntest.SynTestUtils$RepeatFunctionTestWorkerIterator.nextIteration(SynTestUtils.java:125)
              at com.redhat.xpaas.eap.ha.syntest.SynTestUtils$FunctionTestWorker.run(SynTestUtils.java:183)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.net.SocketTimeoutException: Read timed out
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
              at java.net.SocketInputStream.read(SocketInputStream.java:171)
              at java.net.SocketInputStream.read(SocketInputStream.java:141)
              at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
              at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
              at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
              at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
              at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
              at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
              at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
              at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
              at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
              at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
              at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
              at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
              at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
              at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
              at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
              at com.redhat.xpaas.eap.ha.syntest.HttpSynTestUtils.lambda$httpRequestVerifier$5(HttpSynTestUtils.java:202)
              ... 3 common frames omitted
      

      This exception is thrown by clients on not-killed EAP pods as well.

      Investigation:
      There lots of exceptions in logs of all EAP servers. Attaching logs.zip with test logs (com.redhat.xpaas.eap.ha.HAServletCounterUpstreamDnsPingTest.dockerKillOneOfThree) and logs from all pods. Only thing which seems to be suspicious is following:

      Name of the EAP pod in which container was killed is hsc-1-v54g7. In this case newly started pod has the same name. Looking at ths logs of another EAP pod it seems that restarted pod hsc-1-v54g7 joined the cluster before the "killed" one could be removed from topology:

      [0m[0m15:04:01,759 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN000094: Received new cluster view for channel ee: [hsc-1-hrscq|3] (4) [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7, hsc-1-v54g7]
      [0m[0m15:04:01,759 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN100000: Node hsc-1-v54g7 joined the cluster
      [0m[0m15:04:01,760 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN000094: Received new cluster view for channel ee: [hsc-1-hrscq|3] (4) [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7, hsc-1-v54g7]
      [0m[0m15:04:01,760 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN100000: Node hsc-1-v54g7 joined the cluster
      [0m[0m15:04:01,760 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN000094: Received new cluster view for channel ee: [hsc-1-hrscq|3] (4) [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7, hsc-1-v54g7]
      [0m[0m15:04:01,760 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN100000: Node hsc-1-v54g7 joined the cluster
      [0m[0m15:04:01,761 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN000094: Received new cluster view for channel ee: [hsc-1-hrscq|3] (4) [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7, hsc-1-v54g7]
      [0m[0m15:04:01,761 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN100000: Node hsc-1-v54g7 joined the cluster
      [0m[0m15:04:01,842 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN000094: Received new cluster view for channel ee: [hsc-1-hrscq|4] (3) [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7]
      [0m[0m15:04:01,843 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN100001: Node hsc-1-v54g7 left the cluster
      [0m[0m15:04:01,843 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN000094: Received new cluster view for channel ee: [hsc-1-hrscq|4] (3) [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7]
      [0m[0m15:04:01,843 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN100001: Node hsc-1-v54g7 left the cluster
      [0m[0m15:04:01,844 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN000094: Received new cluster view for channel ee: [hsc-1-hrscq|4] (3) [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7]
      [0m[0m15:04:01,844 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN100001: Node hsc-1-v54g7 left the cluster
      [0m[0m15:04:01,845 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN000094: Received new cluster view for channel ee: [hsc-1-hrscq|4] (3) [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7]
      [0m[0m15:04:01,845 INFO  [org.infinispan.CLUSTER] (thread-4,null,null) ISPN100001: Node hsc-1-v54g7 left the cluster
      [0m[0m15:04:03,118 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t2) [Context=client-mappings] ISPN100002: Starting rebalance with members [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7, hsc-1-v54g7], phase READ_OLD_WRITE_ALL, topology id 10
      [0m[0m15:04:03,118 INFO  [org.infinispan.CLUSTER] (remote-thread--p4-t7) [Context=default-server] ISPN100002: Starting rebalance with members [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7, hsc-1-v54g7], phase READ_OLD_WRITE_ALL, topology id 10
      [0m[0m15:04:03,129 INFO  [org.infinispan.CLUSTER] (remote-thread--p4-t7) [Context=ROOT.war] ISPN100002: Starting rebalance with members [hsc-1-hrscq, hsc-1-nk7rf, hsc-1-v54g7, hsc-1-v54g7], phase READ_OLD_WRITE_ALL, topology id 10
      

      After this point there hundreds of errors like:

      [0m[31m15:04:13,217 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p6-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAttributesKey(oQuT5r4q-32bSZ5JZsqcfcBGyz7DVNPYbouZZZcF), SessionCreationMetaDataKey(oQuT5r4q-32bSZ5JZsqcfcBGyz7DVNPYbouZZZcF), SessionAccessMetaDataKey(oQuT5r4q-32bSZ5JZsqcfcBGyz7DVNPYbouZZZcF)]: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 11
      	at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:336)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      [0m[31m15:04:13,219 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p6-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(mPKSVW-Aczm5P7HFWNQ2nLGPDmJQrcibTiRjZr1L), SessionAccessMetaDataKey(mPKSVW-Aczm5P7HFWNQ2nLGPDmJQrcibTiRjZr1L), SessionAttributesKey(mPKSVW-Aczm5P7HFWNQ2nLGPDmJQrcibTiRjZr1L)]: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 11
      	at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:336)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      [0m[31m15:04:13,220 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p6-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(qRhq7-nAKgD0v9TmS-YFzyy3q1TM9nD0vlitAsRC), SessionAttributesKey(qRhq7-nAKgD0v9TmS-YFzyy3q1TM9nD0vlitAsRC), SessionAccessMetaDataKey(qRhq7-nAKgD0v9TmS-YFzyy3q1TM9nD0vlitAsRC)]: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 11
      	at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:336)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      [0m[31m15:04:13,221 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p6-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(RPFBGcLLUJgeZKWuRjO2Pg30p9UWB9sdijRjjLms), SessionAttributesKey(RPFBGcLLUJgeZKWuRjO2Pg30p9UWB9sdijRjjLms), SessionAccessMetaDataKey(RPFBGcLLUJgeZKWuRjO2Pg30p9UWB9sdijRjjLms)]: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 11
      	at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:336)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      [0m[31m15:04:13,220 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-18) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 11
      	at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
      	at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
      	at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:127)
      	at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
      	at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
      	at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
      	at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
      	at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
      	at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
      	at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:87)
      	at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
      	at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
      	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:351)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
      	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
      	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
      	at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
      	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
      	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
      	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
      	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
      	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 11
      	at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:336)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	... 1 more
      	Suppressed: org.infinispan.util.logging.TraceException
      		at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
      		at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
      		at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
      		at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
      		at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:127)
      		at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
      		at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
      		at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
      		at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
      		at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
      		at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
      		at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:87)
      		at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
      		at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
      		at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:351)
      		at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
      		at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
      		at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
      		at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
      		at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
      		at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
      		at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
      		at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
      		at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
      		at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
      		at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
      		at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
      		at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
      		at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
      		at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
      		at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      		at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
      		at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
      		at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
      		... 1 more
      

      Configuration of EAP server:

      <?xml version='1.0' encoding='UTF-8'?>
      
      <server xmlns="urn:jboss:domain:10.0">
          <extensions>
              <extension module="org.jboss.as.clustering.infinispan"/>
              <extension module="org.jboss.as.clustering.jgroups"/>
              <extension module="org.jboss.as.connector"/>
              <extension module="org.jboss.as.deployment-scanner"/>
              <extension module="org.jboss.as.ee"/>
              <extension module="org.jboss.as.ejb3"/>
              <extension module="org.jboss.as.jaxrs"/>
              <extension module="org.jboss.as.jdr"/>
              <extension module="org.jboss.as.jmx"/>
              <extension module="org.jboss.as.jpa"/>
              <extension module="org.jboss.as.jsf"/>
              <extension module="org.jboss.as.logging"/>
              <extension module="org.jboss.as.mail"/>
              <extension module="org.jboss.as.modcluster"/>
              <extension module="org.jboss.as.naming"/>
              <extension module="org.jboss.as.pojo"/>
              <extension module="org.jboss.as.remoting"/>
              <extension module="org.jboss.as.sar"/>
              <extension module="org.jboss.as.security"/>
              <extension module="org.jboss.as.transactions"/>
              <extension module="org.jboss.as.webservices"/>
              <extension module="org.jboss.as.weld"/>
              <extension module="org.wildfly.extension.batch.jberet"/>
              <extension module="org.wildfly.extension.bean-validation"/>
              <extension module="org.wildfly.extension.clustering.singleton"/>
              <extension module="org.wildfly.extension.discovery"/>
              <extension module="org.wildfly.extension.ee-security"/>
              <extension module="org.wildfly.extension.elytron"/>
              <extension module="org.wildfly.extension.io"/>
              <extension module="org.wildfly.extension.messaging-activemq"/>
              <extension module="org.wildfly.extension.microprofile.config-smallrye"/>
              <extension module="org.wildfly.extension.microprofile.health-smallrye"/>
              <extension module="org.wildfly.extension.microprofile.metrics-smallrye"/>
              <!-- ##TRACING_EXTENSION## -->
              <extension module="org.wildfly.extension.request-controller"/>
              <extension module="org.wildfly.extension.security.manager"/>
              <extension module="org.wildfly.extension.undertow"/>
              <!-- ##KEYCLOAK_EXTENSION## -->
          </extensions>
          <management>
              <security-realms>
                  <security-realm name="ManagementRealm">
                      <authentication>
                          <local default-user="$local" skip-group-loading="true"/>
                          <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                      </authentication>
                      <authorization map-groups-to-roles="false">
                          <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                      </authorization>
                  </security-realm>
                  <security-realm name="ApplicationRealm">
                      <!-- No SSL configuration discovered -->
                      <authentication>
                          <local default-user="$local" allowed-users="*" skip-group-loading="true"/>
                          <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                      </authentication>
                      <authorization>
                          <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                      </authorization>
                  </security-realm>
              </security-realms>
              <audit-log>
                  <formatters>
                      <json-formatter name="json-formatter"/>
                  </formatters>
                  <handlers>
                      <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
                  </handlers>
                  <logger log-boot="true" log-read-only="false" enabled="false">
                      <handlers>
                          <handler name="file"/>
                      </handlers>
                  </logger>
              </audit-log>
              <management-interfaces>
                  <http-interface console-enabled="false"><!-- ##MGMT_IFACE_REALM## -->
                      <http-upgrade enabled="true"/>
                      <socket-binding http="management-http"/>
                  </http-interface>
              </management-interfaces>
              <access-control provider="simple">
                  <role-mapping>
                      <role name="SuperUser">
                          <include>
                              <user name="$local"/>
                          </include>
                      </role>
                  </role-mapping>
              </access-control>
          </management>
          <profile>
              <subsystem xmlns="urn:jboss:domain:logging:6.0">
                  <console-handler name="CONSOLE">
                      <formatter>
                          <named-formatter name="COLOR-PATTERN"/>
                      </formatter>
                  </console-handler>
                  <logger category="com.arjuna">
                      <level name="WARN"/>
                  </logger>
                  <logger category="io.jaegertracing.Configuration">
                      <level name="WARN"/>
                  </logger>
                  <logger category="org.jboss.as.config">
                      <level name="DEBUG"/>
                  </logger>
                  <logger category="sun.rmi">
                      <level name="WARN"/>
                  </logger>
                  <!-- ##LOGGER-CATEGORY## -->
                  <root-logger>
                      <level name="INFO"/>
                      <handlers>
                          <handler name="CONSOLE"/>
                      </handlers>
                  </root-logger>
                  <formatter name="OPENSHIFT">
                      <json-formatter>
                          <exception-output-type value="formatted"/>
                          <key-overrides timestamp="@timestamp"/>
                          <meta-data>
                              <property name="@version" value="1"/>
                          </meta-data>
                      </json-formatter>
                  </formatter>
                  <formatter name="COLOR-PATTERN">
                      <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
                  </formatter>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:batch-jberet:2.0">
                       <default-job-repository name="in-memory"/>
                  <default-thread-pool name="batch"/>
                  <job-repository name="in-memory">
                      <in-memory/>
                  </job-repository>
                  <!-- ##JOB_REPOSITORY## -->
                  <thread-pool name="batch">
                      <max-threads count="10"/>
                      <keepalive-time time="30" unit="seconds"/>
                  </thread-pool>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>
              <subsystem xmlns="urn:jboss:domain:datasources:5.0">
                  <datasources>
                      <datasource jta="true" jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}"> <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url> <driver>h2</driver> <security> <user-name>sa</user-name> <password>sa</password> </security> </datasource><!-- ##DATASOURCES## -->
                      <drivers>
                          <driver name="h2" module="com.h2database.h2">
                              <xa-datasource-class&gt;org.h2.jdbcx.JdbcDataSource</xa-datasource-class&gt;
                          </driver>
                          <driver name="mysql" module="com.mysql">
                              <xa-datasource-class&gt;com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class&gt;
                          </driver>
                          <driver name="postgresql" module="org.postgresql">
                              <xa-datasource-class&gt;org.postgresql.xa.PGXADataSource</xa-datasource-class&gt;
                          </driver>
                          <!-- ##DRIVERS## -->
                      </drivers>
                  </datasources>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">
                  <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}" auto-deploy-exploded="false"/>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:discovery:1.0"/>
              <subsystem xmlns="urn:jboss:domain:ee:4.0">
                  <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
                  <concurrent>
                      <context-services>
                          <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>
                      </context-services>
                      <managed-thread-factories>
                          <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>
                      </managed-thread-factories>
                      <managed-executor-services>
                          <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" keepalive-time="5000"/>
                      </managed-executor-services>
                      <managed-scheduled-executor-services>
                          <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" keepalive-time="3000"/>
                      </managed-scheduled-executor-services>
                  </concurrent>
                  <default-bindings context-service="java:jboss/ee/concurrency/context/default"
                                    datasource="java:jboss/datasources/ExampleDS"
                                    jms-connection-factory="java:jboss/DefaultJMSConnectionFactory"
                                    managed-executor-service="java:jboss/ee/concurrency/executor/default"
                                    managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default"
                                    managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:ee-security:1.0"/>
              <subsystem xmlns="urn:jboss:domain:ejb3:5.0">
                  <session-bean>
                      <stateless>
                          <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
                      </stateless>
                      <stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>
                      <singleton default-access-timeout="5000"/>
                  </session-bean>
                  <mdb>
                      <resource-adapter-ref resource-adapter-name="${ejb.resource-adapter-name:activemq-ra.rar}"/>
                      <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
                  </mdb>
                  <pools>
                      <bean-instance-pools>
                          <strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                          <strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                      </bean-instance-pools>
                  </pools>
                  <caches>
                      <cache name="simple"/>
                      <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>
                  </caches>
                  <passivation-stores>
                      <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>
                  </passivation-stores>
                  <async thread-pool-name="default"/>
                              <timer-service thread-pool-name="default" default-data-store="default-file-store">                <data-stores>                    <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>                    <!-- ##DATASTORES## -->                </data-stores>            </timer-service>
                  <remote connector-ref="http-remoting-connector" thread-pool-name="default">
                      <channel-creation-options>
                          <option name="READ_TIMEOUT" value="${prop.remoting-connector.read.timeout:20}" type="xnio"/>
                          <option name="MAX_OUTBOUND_MESSAGES" value="1234" type="remoting"/>
                      </channel-creation-options>
                  </remote>
                  <thread-pools>
                      <thread-pool name="default">
                          <max-threads count="10"/>
                          <keepalive-time time="100" unit="milliseconds"/>
                      </thread-pool>
                  </thread-pools>
                  <default-security-domain value="other"/>
                  <default-missing-method-permissions-deny-access value="true"/>
                  <statistics enabled="${wildfly.ejb3.statistics-enabled:${wildfly.statistics-enabled:false}}"/>
                  <log-system-exceptions value="true"/>
                  
              </subsystem>
              <subsystem xmlns="urn:wildfly:elytron:6.0" final-providers="combined-providers" disallowed-providers="OracleUcrypto">
                  <providers>
                      <aggregate-providers name="combined-providers">
                          <providers name="elytron"/>
                          <providers name="openssl"/>
                      </aggregate-providers>
                      <provider-loader name="elytron" module="org.wildfly.security.elytron"/>
                      <provider-loader name="openssl" module="org.wildfly.openssl"/>
                  </providers>
                  <audit-logging>
                      <file-audit-log name="local-audit" path="audit.log" relative-to="jboss.server.log.dir" format="JSON"/>
                  </audit-logging>
                  <security-domains>
                      
                      <security-domain name="ApplicationDomain" default-realm="ApplicationRealm" permission-mapper="default-permission-mapper">
                          <realm name="ApplicationRealm" role-decoder="groups-to-roles"/>
                          <realm name="local"/>
                      </security-domain>
                      <security-domain name="ManagementDomain" default-realm="ManagementRealm" permission-mapper="default-permission-mapper">
                          <realm name="ManagementRealm" role-decoder="groups-to-roles"/>
                          <realm name="local" role-mapper="super-user-mapper"/>
                      </security-domain>
                  </security-domains>
                  <security-realms>
                      <identity-realm name="local" identity="$local"/>
                      <properties-realm name="ApplicationRealm">
                          <users-properties path="application-users.properties" relative-to="jboss.server.config.dir" digest-realm-name="ApplicationRealm"/>
                          <groups-properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                      </properties-realm>
                      <properties-realm name="ManagementRealm">
                          <users-properties path="mgmt-users.properties" relative-to="jboss.server.config.dir" digest-realm-name="ManagementRealm"/>
                          <groups-properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                      </properties-realm>
                  </security-realms>
                  <mappers>
                      <simple-permission-mapper name="default-permission-mapper" mapping-mode="first">
                          <permission-mapping>
                              <principal name="anonymous"/>
                              <permission-set name="default-permissions"/>
                          </permission-mapping>
                          <permission-mapping match-all="true">
                              <permission-set name="login-permission"/>
                              <permission-set name="default-permissions"/>
                          </permission-mapping>
                      </simple-permission-mapper>
                      <constant-realm-mapper name="local" realm-name="local"/>
                      <simple-role-decoder name="groups-to-roles" attribute="groups"/>
                      <constant-role-mapper name="super-user-mapper">
                          <role name="SuperUser"/>
                      </constant-role-mapper>
                  </mappers>
                  <permission-sets>
                      <permission-set name="login-permission">
                          <permission class-name="org.wildfly.security.auth.permission.LoginPermission"/>
                      </permission-set>
                      <permission-set name="default-permissions">
                          <permission class-name="org.wildfly.extension.batch.jberet.deployment.BatchPermission" module="org.wildfly.extension.batch.jberet" target-name="*"/>
                          <permission class-name="org.wildfly.transaction.client.RemoteTransactionPermission" module="org.wildfly.transaction.client"/>
                          <permission class-name="org.jboss.ejb.client.RemoteEJBPermission" module="org.jboss.ejb-client"/>
                      </permission-set>
                  </permission-sets>
                  <http>
                      
                      <http-authentication-factory name="application-http-authentication" http-server-mechanism-factory="global" security-domain="ApplicationDomain">
                          <mechanism-configuration>
                              <mechanism mechanism-name="BASIC">
                                  <mechanism-realm realm-name="ApplicationRealm"/>
                              </mechanism>
                              <mechanism mechanism-name="FORM"/>
                          </mechanism-configuration>
                      </http-authentication-factory>
                      <http-authentication-factory name="management-http-authentication" http-server-mechanism-factory="global" security-domain="ManagementDomain">
                          <mechanism-configuration>
                              <mechanism mechanism-name="DIGEST">
                                  <mechanism-realm realm-name="ManagementRealm"/>
                              </mechanism>
                          </mechanism-configuration>
                      </http-authentication-factory>
                      <provider-http-server-mechanism-factory name="global"/>
                  </http>
                  <sasl>
                      <sasl-authentication-factory name="application-sasl-authentication" sasl-server-factory="configured" security-domain="ApplicationDomain">
                          <mechanism-configuration>
                              <mechanism mechanism-name="JBOSS-LOCAL-USER" realm-mapper="local"/>
                              <mechanism mechanism-name="DIGEST-MD5">
                                  <mechanism-realm realm-name="ApplicationRealm"/>
                              </mechanism>
                          </mechanism-configuration>
                      </sasl-authentication-factory>
                      <sasl-authentication-factory name="management-sasl-authentication" sasl-server-factory="configured" security-domain="ManagementDomain">
                          <mechanism-configuration>
                              <mechanism mechanism-name="JBOSS-LOCAL-USER" realm-mapper="local"/>
                              <mechanism mechanism-name="DIGEST-MD5">
                                  <mechanism-realm realm-name="ManagementRealm"/>
                              </mechanism>
                          </mechanism-configuration>
                      </sasl-authentication-factory>
                      <configurable-sasl-server-factory name="configured" sasl-server-factory="elytron">
                          <properties>
                              <property name="wildfly.sasl.local-user.default-user" value="$local"/>
                          </properties>
                      </configurable-sasl-server-factory>
                      <mechanism-provider-filtering-sasl-server-factory name="elytron" sasl-server-factory="global">
                          <filters>
                              <filter provider-name="WildFlyElytron"/>
                          </filters>
                      </mechanism-provider-filtering-sasl-server-factory>
                      <provider-sasl-server-factory name="global"/>
                  </sasl>
                  
                           <tls>
                  <key-stores>
                      <key-store name="jgroups.jceks">
       <credential-reference clear-text="xpaasQEjgroups"/>
       <implementation type="JCEKS"/>
       <file path="/etc/jgroups-encrypt-secret-volume/jgroups.jceks" />
       </key-store><!-- ##ELYTRON_KEY_STORE## -->
                  </key-stores>
                  <key-managers>
                      <!-- ##ELYTRON_KEY_MANAGER## -->
                  </key-managers>
                  <server-ssl-contexts>
                      <!-- ##ELYTRON_SERVER_SSL_CONTEXT## -->
                  </server-ssl-contexts>
               </tls>
      
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:infinispan:8.0">
                  <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
                      <transport lock-timeout="60000"/>
                      <replicated-cache name="default">
                          <transaction mode="BATCH"/>
                      </replicated-cache>
                  </cache-container>
                  <cache-container name="web" default-cache="repl" module="org.wildfly.clustering.web.infinispan">
                      <transport lock-timeout="60000"/>
                      <replicated-cache name="repl">
                          <locking isolation="REPEATABLE_READ"/>
                          <transaction mode="BATCH"/>
                          <file-store/>
                      </replicated-cache>
                      <distributed-cache name="dist">
                          <locking isolation="REPEATABLE_READ"/>
                          <transaction mode="BATCH"/>
                          <file-store/>
                      </distributed-cache>
                  </cache-container>
                  <cache-container name="ejb" aliases="sfsb" default-cache="repl" module="org.wildfly.clustering.ejb.infinispan">
                      <transport lock-timeout="60000"/>
                      <replicated-cache name="repl">
                          <locking isolation="REPEATABLE_READ"/>
                          <object-memory size="10000"/>
                          <transaction mode="BATCH"/>
                          <file-store/>
                      </replicated-cache>
                      <distributed-cache name="dist">
                          <locking isolation="REPEATABLE_READ"/>
                          <transaction mode="BATCH"/>
                          <file-store/>
                      </distributed-cache>
                  </cache-container>
                  <cache-container name="hibernate" default-cache="local-query" module="org.infinispan.hibernate-cache">
                      <transport lock-timeout="60000"/>
                      <local-cache name="local-query">
                          <object-memory size="10000"/>
                          <expiration max-idle="100000"/>
                      </local-cache>
                      <invalidation-cache name="entity">
                          <object-memory size="10000"/>
                          <expiration max-idle="100000"/>
                      </invalidation-cache>
                      <replicated-cache name="timestamps"/>
                  </cache-container>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:io:3.0">
                  <worker name="default"/>
                  <buffer-pool name="default"/>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
              <subsystem xmlns="urn:jboss:domain:jca:5.0">
                  <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
                  <bean-validation enabled="true"/>
                  <default-workmanager>
                      <short-running-threads>
                          <core-threads count="50"/>
                          <queue-length count="50"/>
                          <max-threads count="50"/>
                          <keepalive-time time="10" unit="seconds"/>
                      </short-running-threads>
                      <long-running-threads>
                          <core-threads count="50"/>
                          <queue-length count="50"/>
                          <max-threads count="50"/>
                          <keepalive-time time="10" unit="seconds"/>
                      </long-running-threads>
                  </default-workmanager>
                  <cached-connection-manager/>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
              <subsystem xmlns="urn:jboss:domain:jgroups:6.0">
                  <channels default="ee">
                      <channel name="ee" stack="tcp"/>
                  </channels>
                  <stacks>
                      <stack name="udp">
                          <transport type="UDP" socket-binding="jgroups-udp"/>
                          <protocol type="dns.DNS_PING" ><property name="dns_query">ping-service</property><property name="async_discovery_use_separate_thread_per_request">true</property></protocol>
                          <protocol type="MERGE3"/>
                          <protocol type="FD_SOCK"/>
                          <protocol type="FD_ALL"/>
                          <protocol type="VERIFY_SUSPECT"/>
                          <encrypt-protocol type="SYM_ENCRYPT" key-store="jgroups.jceks" key-alias="xpaasqe"> <key-credential-reference clear-text="xpaasQEjgroups"/> </encrypt-protocol>
                          <protocol type="pbcast.NAKACK2"/>
                          <protocol type="UNICAST3"/>
                          <protocol type="pbcast.STABLE"/>
                          <!--WARNING: No password defined for JGroups cluster. AUTH protocol has been disabled. Please define JGROUPS_CLUSTER_PASSWORD. -->
                          <protocol type="pbcast.GMS"/>
                          <protocol type="UFC"/>
                          <protocol type="MFC"/>
                          <protocol type="FRAG3"/>
                      </stack>
                      <stack name="tcp">
                          <transport type="TCP" socket-binding="jgroups-tcp"/>
                          <protocol type="dns.DNS_PING" ><property name="dns_query">ping-service</property><property name="async_discovery_use_separate_thread_per_request">true</property></protocol>
                          <protocol type="MERGE3"/>
                          <protocol type="FD_SOCK"/>
                          <protocol type="FD_ALL"/>
                          <protocol type="VERIFY_SUSPECT"/>
                          <encrypt-protocol type="SYM_ENCRYPT" key-store="jgroups.jceks" key-alias="xpaasqe"> <key-credential-reference clear-text="xpaasQEjgroups"/> </encrypt-protocol>
                          <protocol type="pbcast.NAKACK2"/>
                          <protocol type="UNICAST3"/>
                          <protocol type="pbcast.STABLE"/>
                          <!--WARNING: No password defined for JGroups cluster. AUTH protocol has been disabled. Please define JGROUPS_CLUSTER_PASSWORD. -->
                          <protocol type="pbcast.GMS"/>
                          <protocol type="MFC"/>
                          <protocol type="FRAG3"/>
                      </stack>
                  </stacks>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:jmx:1.3">
                  <expose-resolved-model/>
                  <expose-expression-model/>
                  <remoting-connector/>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:jpa:1.1">
                  <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:jsf:1.1"/>
              <!-- ##KEYCLOAK_SUBSYSTEM## -->
              <!-- ##KEYCLOAK_SAML_SUBSYSTEM## -->
              <subsystem xmlns="urn:jboss:domain:mail:3.0">
                  <mail-session name="default" jndi-name="java:jboss/mail/Default">
                      <smtp-server outbound-socket-binding-ref="mail-smtp"/>
                  </mail-session>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:messaging-activemq:6.0">
                      <server name="default">
              <statistics enabled="${wildfly.messaging-activemq.statistics-enabled:${wildfly.statistics-enabled:false}}" />
              <journal pool-files="10"/>
              <security-setting name="#">
                  <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
              </security-setting>
              <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
              <http-connector name="http-connector" socket-binding="http-messaging" endpoint="http-acceptor"/>
              <http-connector name="http-connector-throughput" socket-binding="http-messaging" endpoint="http-acceptor-throughput">
                  <param name="batch-delay" value="50"/>
              </http-connector>
              <!-- ##AMQ_REMOTE_CONNECTOR## -->
              <in-vm-connector name="in-vm" server-id="0">
                  <param name="buffer-pooling" value="false"/>
              </in-vm-connector>
              <http-acceptor name="http-acceptor" http-listener="default"/>
              <http-acceptor name="http-acceptor-throughput" http-listener="default">
                  <param name="batch-delay" value="50"/>
                  <param name="direct-deliver" value="false"/>
              </http-acceptor>
              <in-vm-acceptor name="in-vm" server-id="0">
                  <param name="buffer-pooling" value="false"/>
              </in-vm-acceptor>
              <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
              <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
              
              <connection-factory name="InVmConnectionFactory" connectors="in-vm" entries="java:/ConnectionFactory"/>
              <connection-factory name="RemoteConnectionFactory" connectors="http-connector" entries="java:jboss/exported/jms/RemoteConnectionFactory" reconnect-attempts="-1" />
              <pooled-connection-factory name="activemq-ra" transaction="xa" connectors="in-vm" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory"/>
              <!-- ##AMQ_POOLED_CONNECTION_FACTORY## -->
          </server>
              </subsystem>
              <subsystem xmlns="urn:wildfly:microprofile-config-smallrye:1.0">
                  <!-- ##MICROPROFILE_CONFIG_SOURCE## -->
              </subsystem>
              <subsystem xmlns="urn:wildfly:microprofile-health-smallrye:1.0"/>
              <subsystem xmlns="urn:wildfly:microprofile-metrics-smallrye:2.0" security-enabled="false" exposed-subsystems="*" prefix="${wildfly.metrics.prefix:jboss}"/>
              <!-- ##TRACING_SUBSYSTEM## -->
              <subsystem xmlns="urn:jboss:domain:modcluster:5.0">
                  <proxy name="default" advertise-socket="modcluster" listener="ajp">
                      <dynamic-load-provider>
                          <load-metric type="cpu"/>
                      </dynamic-load-provider>
                  </proxy>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:naming:2.0">
                  <!-- ##MESSAGING_REMOTE_BINDINGS## -->
                  <!-- ##AMQ_REMOTE_CONTEXT## -->
                  <remote-naming/>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
              <subsystem xmlns="urn:jboss:domain:remoting:4.0">
                  <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>
              <subsystem xmlns="urn:jboss:domain:resource-adapters:5.0">
                  <resource-adapters>
                      <!-- ##RESOURCE_ADAPTERS## -->
                  </resource-adapters>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:sar:1.0"/>
              <subsystem xmlns="urn:jboss:domain:security:2.0">
                  
                  <security-domains>
                      <security-domain name="other" cache-type="default">
                          <authentication>
                              <login-module code="Remoting" flag="optional">
                                  <module-option name="password-stacking" value="useFirstPass"/>
                              </login-module>
                              <login-module code="RealmDirect" flag="required">
                                  <module-option name="password-stacking" value="useFirstPass"/>
                              </login-module>
                              <!-- ##OTHER_LOGIN_MODULES## -->
                          </authentication>
                      </security-domain>
                      <security-domain name="jboss-web-policy" cache-type="default">
                          <authorization>
                              <policy-module code="Delegating" flag="required"/>
                          </authorization>
                      </security-domain>
                      <security-domain name="jboss-ejb-policy" cache-type="default">
                          <authorization>
                              <policy-module code="Delegating" flag="required"/>
                          </authorization>
                      </security-domain>
                      <!-- ##KEYCLOAK_SECURITY_DOMAIN## -->
                      <!-- no additional security domains configured --><!-- ##ADDITIONAL_SECURITY_DOMAINS## -->
                  </security-domains>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:security-manager:1.0">
                  <deployment-permissions>
                      <maximum-set>
                          <permission class="java.security.AllPermission"/>
                      </maximum-set>
                  </deployment-permissions>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:singleton:1.0">
                  <singleton-policies default="default">
                      <singleton-policy name="default" cache-container="server">
                          <simple-election-policy/>
                      </singleton-policy>
                  </singleton-policies>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:transactions:5.0">
                  <core-environment node-identifier="${jboss.node.name}">
                      <process-id>
                          <uuid/>
                      </process-id>
                  </core-environment>
                  <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager" recovery-listener="true"/>
                  <coordinator-environment statistics-enabled="${wildfly.transactions.statistics-enabled:${wildfly.statistics-enabled:false}}"/>
                  <!-- ##JDBC_STORE## -->
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:undertow:8.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other" statistics-enabled="${wildfly.undertow.statistics-enabled:${wildfly.statistics-enabled:false}}">
                  <buffer-cache name="default"/>
                  <server name="default-server">
                      <ajp-listener name="ajp" socket-binding="ajp"/>
                      <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true" proxy-address-forwarding="true"/>
                      <!-- No HTTPS configuration discovered -->
                      <host name="default-host" alias="localhost">
                          <location name="/" handler="welcome-content"/>
                          <!-- ##ACCESS_LOG_VALVE## -->
                          <!-- ##FILTER_REFS## -->
                          <http-invoker security-realm="ApplicationRealm"/>
                      </host>
                  </server>
                  <servlet-container name="default">
                      <jsp-config/>
                      <websockets/>
                  </servlet-container>
                  <handlers>
                      <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
                  </handlers>
                  
                  <!-- ##HTTP_FILTERS_MARKER## -->
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:webservices:2.0" statistics-enabled="${wildfly.webservices.statistics-enabled:${wildfly.statistics-enabled:false}}">
                  <modify-wsdl-address>true</modify-wsdl-address>
                  <wsdl-host>jbossws.undefined.host</wsdl-host>
                  <endpoint-config name="Standard-Endpoint-Config"/>
                  <endpoint-config name="Recording-Endpoint-Config">
                      <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
                          <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
                      </pre-handler-chain>
                  </endpoint-config>
                  <client-config name="Standard-Client-Config"/>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:weld:4.0"/>
          </profile>
          <interfaces>
              <interface name="management">
                  <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
              </interface>
              <interface name="public">
                  <inet-address value="${jboss.bind.address:127.0.0.1}"/>
              </interface>
              <interface name="private">
                  <inet-address value="${jboss.bind.address.private:127.0.0.1}"/>
              </interface>
              <interface name="bindall">
                  <inet-address value="0.0.0.0"/>
              </interface>
          </interfaces>
          <socket-binding-group name="standard-sockets" default-interface="public" port-offset="0">
              <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
              <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
              <socket-binding name="ajp" interface="bindall" port="${jboss.ajp.port:8009}"/>
              <socket-binding name="http" interface="bindall" port="${jboss.http.port:8080}"/>
              <socket-binding name="https" interface="bindall" port="${jboss.https.port:8443}"/>
              <socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
              <socket-binding name="jgroups-tcp" interface="private" port="7600"/>
              <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
              <socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>
              <outbound-socket-binding name="http-messaging">
                  <remote-destination host="${jboss.messaging.host:localhost}" port="${jboss.http.port:8080}"/>
              </outbound-socket-binding>
              <socket-binding name="messaging" port="5445"/><socket-binding name="messaging-throughput" port="5455"/>
              <!-- ##AMQ_MESSAGING_SOCKET_BINDING## -->
              <socket-binding name="txn-recovery-environment" port="4712"/>
              <socket-binding name="txn-status-manager" port="4713"/>
              <outbound-socket-binding name="mail-smtp">
                  <remote-destination host="localhost" port="25"/>
              </outbound-socket-binding>
          </socket-binding-group>
      </server>
      

      Attachments

        1. logs.zip
          3.69 MB
        2. logs2.zip
          50 kB

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              mnovak1@redhat.com Miroslav Novak
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: