Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
7.2.0.GA.CR1
-
None
-
As by explanation in comment: Lock timeouts of this nature are expected in jvm kill scenarios.
Description
This issue produces a few errors (10 errors on 333712 samples) on the client and that is why its marked as Major.
It was observed in JVM kill failover scenario.
perflab_eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync
In a scenario perflab_eap-7x-failover-ejb-ejbservlet-jvmkill-dist-sync where SFSB are invoked though servlets and failover is introduced by killing the server JVM, we observed this error;
Node perf20 is killed at 08:54:37 and restarted at 08:55:42.
Meanwhile on node perf21 we observe the following:
[JBossINF] �[0m�[0m08:54:43,029 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-137,ejb,perf21) ISPN000094: Received new cluster view for channel ejb: [perf21|8] (3) [perf21, perf18, perf19] [JBossINF] �[0m�[0m08:54:43,031 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-137,ejb,perf21) ISPN100001: Node perf20 left the cluster [JBossINF] �[0m�[0m08:54:43,031 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-137,ejb,perf21) ISPN000094: Received new cluster view for channel ejb: [perf21|8] (3) [perf21, perf18, perf19] [JBossINF] �[0m�[0m08:54:43,031 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-137,ejb,perf21) ISPN100001: Node perf20 left the cluster [JBossINF] �[0m�[0m08:54:43,032 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-137,ejb,perf21) ISPN000094: Received new cluster view for channel ejb: [perf21|8] (3) [perf21, perf18, perf19] [JBossINF] �[0m�[0m08:54:43,032 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-137,ejb,perf21) ISPN100001: Node perf20 left the cluster [JBossINF] �[0m�[0m08:54:43,033 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-137,ejb,perf21) ISPN000094: Received new cluster view for channel ejb: [perf21|8] (3) [perf21, perf18, perf19] [JBossINF] �[0m�[0m08:54:43,033 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-137,ejb,perf21) ISPN100001: Node perf20 left the cluster [JBossINF] �[0m�[0m08:54:43,165 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t94) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100007: After merge (or coordinator change), recovered members [perf20, perf21, perf18, perf19] with topology id 33 [JBossINF] �[0m�[0m08:54:43,168 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t94) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100008: Updating cache members list [perf21, perf18, perf19], topology id 34 [JBossINF] �[0m�[0m08:54:43,169 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t54) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [perf20, perf21, perf18, perf19] with topology id 33 [JBossINF] �[0m�[0m08:54:43,172 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t96) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [perf20, perf21, perf18, perf19] with topology id 33 [JBossINF] �[0m�[0m08:54:43,197 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t55) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [perf20, perf21, perf18, perf19] with topology id 33 [JBossINF] �[0m�[0m08:54:43,210 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t92) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100007: After merge (or coordinator change), recovered members [perf20, perf21, perf18, perf19] with topology id 33 [JBossINF] �[0m�[0m08:54:43,211 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t95) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100007: After merge (or coordinator change), recovered members [perf20, perf21, perf18, perf19] with topology id 33 [JBossINF] �[0m�[0m08:54:43,215 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t92) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100008: Updating cache members list [perf21, perf18, perf19], topology id 34 [JBossINF] �[0m�[0m08:54:43,216 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t95) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100008: Updating cache members list [perf21, perf18, perf19], topology id 34 [JBossINF] �[0m�[0m08:54:43,219 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t55) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [perf21, perf18, perf19], topology id 34 [JBossINF] �[0m�[0m08:54:43,281 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t54) [Context=client-mappings] ISPN100008: Updating cache members list [perf21, perf18, perf19], topology id 34 [JBossINF] �[0m�[0m08:54:43,291 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t96) [Context=default-server] ISPN100008: Updating cache members list [perf21, perf18, perf19], topology id 34 [JBossINF] �[0m�[0m08:54:43,327 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t95) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100002: Starting rebalance with members [perf21, perf18, perf19], phase READ_OLD_WRITE_ALL, topology id 35 [JBossINF] �[0m�[0m08:54:43,331 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t94) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100002: Starting rebalance with members [perf21, perf18, perf19], phase READ_OLD_WRITE_ALL, topology id 35 [JBossINF] �[0m�[0m08:54:43,338 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t55) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [perf21, perf18, perf19], phase READ_OLD_WRITE_ALL, topology id 35 [JBossINF] �[0m�[0m08:54:43,338 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p20-t92) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100002: Starting rebalance with members [perf21, perf18, perf19], phase READ_OLD_WRITE_ALL, topology id 35 [JBossINF] �[0m�[0m08:54:43,610 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t58) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 36 [JBossINF] �[0m�[0m08:54:43,644 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 36 [JBossINF] �[0m�[0m08:54:43,658 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t64) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 37 [JBossINF] �[0m�[0m08:54:43,727 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t63) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 37 [JBossINF] �[0m�[0m08:54:43,732 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t32) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 36 [JBossINF] �[0m�[0m08:54:43,740 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t66) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-granular.war] ISPN100010: Finished rebalance with members [perf21, perf18, perf19], topology id 38 [JBossINF] �[0m�[0m08:54:43,781 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t61) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war] ISPN100010: Finished rebalance with members [perf21, perf18, perf19], topology id 38 [JBossINF] �[0m�[0m08:54:43,781 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t32) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 37 [JBossINF] �[0m�[0m08:54:43,822 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t66) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 36 [JBossINF] �[0m�[0m08:54:43,838 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t52) [Context=clusterbench-ee7.ear/clusterbench-ee7-ejb.jar] ISPN100010: Finished rebalance with members [perf21, perf18, perf19], topology id 38 [JBossINF] �[0m�[0m08:54:43,883 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t64) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 37 [JBossINF] �[0m�[0m08:54:44,057 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t57) [Context=clusterbench-ee7.ear.clusterbench-ee7-web-default.war] ISPN100010: Finished rebalance with members [perf21, perf18, perf19], topology id 38 [JBossINF] �[0m�[31m08:55:02,351 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (default task-311) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(2PiYDJcGvSqUsLTSGx6-O5ALG9N9PmTMlv0oipV9) in behalf of transaction GlobalTx:perf21:310749. Current owner GlobalTx:perf18:107113. [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:252) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:220) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:119) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:146) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:73) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLock(PessimisticLockingInterceptor.java:92) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLockAndInvokeNext(PessimisticLockingInterceptor.java:283) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:71) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:168) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:209) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:411) [JBossINF] at org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [JBossINF] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:241) [JBossINF] at org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:291) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] �[0m�[31m08:55:02,361 ERROR [io.undertow.request] (default task-311) UT005023: Exception handling request to /clusterbench/ejbservlet: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(2PiYDJcGvSqUsLTSGx6-O5ALG9N9PmTMlv0oipV9) in behalf of transaction GlobalTx:perf21:310749. Current owner GlobalTx:perf18:107113. [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:209) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:411) [JBossINF] at org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [JBossINF] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:241) [JBossINF] at org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:291) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(2PiYDJcGvSqUsLTSGx6-O5ALG9N9PmTMlv0oipV9) in behalf of transaction GlobalTx:perf21:310749. Current owner GlobalTx:perf18:107113. [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:252) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:220) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:119) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:146) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:73) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLock(PessimisticLockingInterceptor.java:92) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLockAndInvokeNext(PessimisticLockingInterceptor.java:283) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:71) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:168) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [JBossINF] ... 39 more [JBossINF] Suppressed: org.infinispan.util.logging.TraceException [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) [JBossINF] ... 39 more [JBossINF] [JBossINF] �[0m�[31m08:55:17,365 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (default task-311) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(2PiYDJcGvSqUsLTSGx6-O5ALG9N9PmTMlv0oipV9) in behalf of transaction GlobalTx:perf21:322998. Current owner GlobalTx:perf18:107113. [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:252) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:220) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:119) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:146) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:73) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLock(PessimisticLockingInterceptor.java:92) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLockAndInvokeNext(PessimisticLockingInterceptor.java:283) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:71) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:168) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:209) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579) [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] �[0m�[31m08:55:21,368 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (default task-290) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(2PiYDJcGvSqUsLTSGx6-O5ALG9N9PmTMlv0oipV9) in behalf of transaction GlobalTx:perf21:326105. Current owner GlobalTx:perf18:107113. [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:252) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:220) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:119) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:146) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:73) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLock(PessimisticLockingInterceptor.java:92) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLockAndInvokeNext(PessimisticLockingInterceptor.java:283) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:71) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:168) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:209) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:411) [JBossINF] at org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [JBossINF] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:241) [JBossINF] at org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:291) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] �[0m�[31m08:55:21,372 ERROR [io.undertow.request] (default task-290) UT005023: Exception handling request to /clusterbench/ejbservlet: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(2PiYDJcGvSqUsLTSGx6-O5ALG9N9PmTMlv0oipV9) in behalf of transaction GlobalTx:perf21:326105. Current owner GlobalTx:perf18:107113. [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:209) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:411) [JBossINF] at org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [JBossINF] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:241) [JBossINF] at org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:291) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(2PiYDJcGvSqUsLTSGx6-O5ALG9N9PmTMlv0oipV9) in behalf of transaction GlobalTx:perf21:326105. Current owner GlobalTx:perf18:107113. [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:252) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:220) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:119) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:146) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:73) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLock(PessimisticLockingInterceptor.java:92) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLockAndInvokeNext(PessimisticLockingInterceptor.java:283) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:71) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:168) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [JBossINF] ... 39 more [JBossINF] Suppressed: org.infinispan.util.logging.TraceException [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250) [JBossINF] ... 39 more [JBossINF] [JBossINF] �[0m�[31m08:55:32,368 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (default task-311) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(2PiYDJcGvSqUsLTSGx6-O5ALG9N9PmTMlv0oipV9) in behalf of transaction GlobalTx:perf21:335413. Current owner GlobalTx:perf18:107113. [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:252) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:220) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:119) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:146) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:73) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLock(PessimisticLockingInterceptor.java:92) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLockAndInvokeNext(PessimisticLockingInterceptor.java:283) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:71) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:107) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:313) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:185) [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:168) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106) [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:209) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858) [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:411) [JBossINF] at com.sun.faces.application.WebappLifecycleListener.requestDestroyed(WebappLifecycleListener.java:166) [JBossINF] at com.sun.faces.config.ConfigureListener.requestDestroyed(ConfigureListener.java:408) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestDestroyed(ApplicationListeners.java:260) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:342) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] [JBossINF] �[0m�[31m08:55:32,381 ERROR [io.undertow.servlet.request] (default task-311) UT015005: Error invoking method requestDestroyed on listener class com.sun.faces.config.ConfigureListener: java.lang.UnsupportedOperationException [JBossINF] at javax.faces.context.FacesContext.getExceptionHandler(FacesContext.java:304) [JBossINF] at javax.faces.event.ExceptionQueuedEventContext.getListenersForEventClass(ExceptionQueuedEventContext.java:272) [JBossINF] at com.sun.faces.application.ApplicationImpl.invokeComponentListenersFor(ApplicationImpl.java:2230) [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:337) [JBossINF] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:295) [JBossINF] at javax.faces.application.ApplicationWrapper.publishEvent(ApplicationWrapper.java:748) [JBossINF] at com.sun.faces.application.WebappLifecycleListener.requestDestroyed(WebappLifecycleListener.java:175) [JBossINF] at com.sun.faces.config.ConfigureListener.requestDestroyed(ConfigureListener.java:408) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestDestroyed(ApplicationListeners.java:260) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:342) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748)
that produces an error on the client side:
2018/11/13 08:55:02:364 EST [WARN ][Runner - 1265] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
2018/11/13 08:55:02:365 EST [WARN ][Runner - 1265] SFCORE_LOG - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:164)
at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
at java.lang.Thread.run(Thread.java:748)
perflab_eap-7x-failover-http-session-jvmkill-repl-sync-shared-sessions
In scenario perflab_eap-7x-failover-http-session-jvmkill-repl-sync-shared-sessions we can observe the error but without impacting the client.
This is a scenario where two WAR files in the same deployment share http sessions.
All the nodes are jvm killed and restarted; the error occurs after all nodes are killed and re-stated; we can observe this in the following log snippet:
[JBossINF] �[0m�[0m04:35:00,691 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: JBoss EAP 7.2.0.GA (WildFly Core 6.0.11.Final-redhat-00001) started in 13402ms - Started 869 of 1075 services (464 services are lazy, passive or on-demand) [JBossINF] �[0m�[0m04:36:05,875 INFO [org.infinispan.CLUSTER] (thread-25,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] �[0m�[0m04:36:05,882 INFO [org.infinispan.CLUSTER] (thread-25,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] �[0m�[0m04:36:05,883 INFO [org.infinispan.CLUSTER] (thread-25,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] �[0m�[0m04:36:05,884 INFO [org.infinispan.CLUSTER] (thread-25,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] �[0m�[0m04:36:05,886 INFO [org.infinispan.CLUSTER] (thread-25,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] �[0m�[0m04:36:05,886 INFO [org.infinispan.CLUSTER] (thread-25,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] �[0m�[0m04:36:05,887 INFO [org.infinispan.CLUSTER] (thread-25,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18] [JBossINF] �[0m�[0m04:36:05,894 INFO [org.infinispan.CLUSTER] (thread-25,ejb,perf18) ISPN100001: Node perf19 left the cluster [JBossINF] �[0m�[0m04:37:08,462 INFO [org.infinispan.CLUSTER] (thread-8,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] �[0m�[0m04:37:08,464 INFO [org.infinispan.CLUSTER] (thread-8,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] �[0m�[0m04:37:08,464 INFO [org.infinispan.CLUSTER] (thread-8,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] �[0m�[0m04:37:08,465 INFO [org.infinispan.CLUSTER] (thread-8,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] �[0m�[0m04:37:08,466 INFO [org.infinispan.CLUSTER] (thread-8,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] �[0m�[0m04:37:08,466 INFO [org.infinispan.CLUSTER] (thread-8,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] �[0m�[0m04:37:08,467 INFO [org.infinispan.CLUSTER] (thread-8,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19] [JBossINF] �[0m�[0m04:37:08,468 INFO [org.infinispan.CLUSTER] (thread-8,ejb,perf18) ISPN100000: Node perf19 joined the cluster [JBossINF] �[0m�[0m04:38:19,789 INFO [org.infinispan.CLUSTER] (thread-105,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|8] (3) [perf21, perf18, perf19] [JBossINF] �[0m�[0m04:38:19,790 INFO [org.infinispan.CLUSTER] (thread-105,ejb,perf18) ISPN100001: Node perf20 left the cluster [JBossINF] �[0m�[0m04:38:19,791 INFO [org.infinispan.CLUSTER] (thread-105,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|8] (3) [perf21, perf18, perf19] [JBossINF] �[0m�[0m04:38:19,791 INFO [org.infinispan.CLUSTER] (thread-105,ejb,perf18) ISPN100001: Node perf20 left the cluster [JBossINF] �[0m�[0m04:38:19,792 INFO [org.infinispan.CLUSTER] (thread-105,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|8] (3) [perf21, perf18, perf19] [JBossINF] �[0m�[0m04:38:19,793 INFO [org.infinispan.CLUSTER] (thread-105,ejb,perf18) ISPN100001: Node perf20 left the cluster [JBossINF] �[0m�[0m04:38:19,793 INFO [org.infinispan.CLUSTER] (thread-105,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|8] (3) [perf21, perf18, perf19] [JBossINF] �[0m�[0m04:38:19,794 INFO [org.infinispan.CLUSTER] (thread-105,ejb,perf18) ISPN100001: Node perf20 left the cluster [JBossINF] �[0m�[0m04:39:22,472 INFO [org.infinispan.CLUSTER] (thread-107,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|9] (4) [perf21, perf18, perf19, perf20] [JBossINF] �[0m�[0m04:39:22,474 INFO [org.infinispan.CLUSTER] (thread-107,ejb,perf18) ISPN100000: Node perf20 joined the cluster [JBossINF] �[0m�[0m04:39:22,474 INFO [org.infinispan.CLUSTER] (thread-107,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|9] (4) [perf21, perf18, perf19, perf20] [JBossINF] �[0m�[0m04:39:22,475 INFO [org.infinispan.CLUSTER] (thread-107,ejb,perf18) ISPN100000: Node perf20 joined the cluster [JBossINF] �[0m�[0m04:39:22,476 INFO [org.infinispan.CLUSTER] (thread-107,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|9] (4) [perf21, perf18, perf19, perf20] [JBossINF] �[0m�[0m04:39:22,477 INFO [org.infinispan.CLUSTER] (thread-107,ejb,perf18) ISPN100000: Node perf20 joined the cluster [JBossINF] �[0m�[0m04:39:22,479 INFO [org.infinispan.CLUSTER] (thread-107,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|9] (4) [perf21, perf18, perf19, perf20] [JBossINF] �[0m�[0m04:39:22,481 INFO [org.infinispan.CLUSTER] (thread-107,ejb,perf18) ISPN100000: Node perf20 joined the cluster [JBossINF] �[0m�[0m04:40:33,459 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-125,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf18|10] (3) [perf18, perf19, perf20] [JBossINF] �[0m�[0m04:40:33,461 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-125,ejb,perf18) ISPN100001: Node perf21 left the cluster [JBossINF] �[0m�[0m04:40:33,461 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-125,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf18|10] (3) [perf18, perf19, perf20] [JBossINF] �[0m�[0m04:40:33,462 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-125,ejb,perf18) ISPN100001: Node perf21 left the cluster [JBossINF] �[0m�[0m04:40:33,463 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-125,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf18|10] (3) [perf18, perf19, perf20] [JBossINF] �[0m�[0m04:40:33,463 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-125,ejb,perf18) ISPN100001: Node perf21 left the cluster [JBossINF] �[0m�[0m04:40:33,464 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-125,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf18|10] (3) [perf18, perf19, perf20] [JBossINF] �[0m�[0m04:40:33,464 INFO [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-125,ejb,perf18) ISPN100001: Node perf21 left the cluster [JBossINF] �[0m�[0m04:40:33,523 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t44) [Context=clusterbench-ee7-shared-sessions.ear] ISPN100007: After merge (or coordinator change), recovered members [perf21, perf18, perf19, perf20] with topology id 42 [JBossINF] �[0m�[0m04:40:33,523 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t37) [Context=default-server] ISPN100007: After merge (or coordinator change), recovered members [perf21, perf18, perf19, perf20] with topology id 42 [JBossINF] �[0m�[0m04:40:33,524 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t52) [Context=client-mappings] ISPN100007: After merge (or coordinator change), recovered members [perf21, perf18, perf19, perf20] with topology id 42 [JBossINF] �[0m�[0m04:40:33,526 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t50) [Context=clusterbench-ee7-shared-sessions.ear/clusterbench-ee7-ejb.jar] ISPN100007: After merge (or coordinator change), recovered members [perf21, perf18, perf19, perf20] with topology id 42 [JBossINF] �[0m�[0m04:40:33,528 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t50) [Context=clusterbench-ee7-shared-sessions.ear/clusterbench-ee7-ejb.jar] ISPN100008: Updating cache members list [perf18, perf19, perf20], topology id 43 [JBossINF] �[0m�[0m04:40:33,539 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t52) [Context=client-mappings] ISPN100008: Updating cache members list [perf18, perf19, perf20], topology id 43 [JBossINF] �[0m�[0m04:40:33,542 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t44) [Context=clusterbench-ee7-shared-sessions.ear] ISPN100008: Updating cache members list [perf18, perf19, perf20], topology id 43 [JBossINF] �[0m�[0m04:40:33,546 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t52) [Context=client-mappings] ISPN100002: Starting rebalance with members [perf18, perf19, perf20], phase READ_OLD_WRITE_ALL, topology id 44 [JBossINF] �[0m�[0m04:40:33,546 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p17-t50) [Context=clusterbench-ee7-shared-sessions.ear/clusterbench-ee7-ejb.jar] ISPN100002: Starting rebalance with members [perf18, perf19, perf20], phase READ_OLD_WRITE_ALL, topology id 44 [JBossINF] �[0m�[0m04:40:33,546 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t37) [Context=default-server] ISPN100008: Updating cache members list [perf18, perf19, perf20], topology id 43 [JBossINF] �[0m�[0m04:40:33,547 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t44) [Context=clusterbench-ee7-shared-sessions.ear] ISPN100002: Starting rebalance with members [perf18, perf19, perf20], phase READ_OLD_WRITE_ALL, topology id 44 [JBossINF] �[0m�[0m04:40:33,549 INFO [org.infinispan.CLUSTER] (stateTransferExecutor-thread--p19-t37) [Context=default-server] ISPN100002: Starting rebalance with members [perf18, perf19, perf20], phase READ_OLD_WRITE_ALL, topology id 44 [JBossINF] �[0m�[0m04:40:33,680 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t178) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 45 [JBossINF] �[0m�[0m04:40:33,680 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t189) [Context=clusterbench-ee7-shared-sessions.ear] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 45 [JBossINF] �[0m�[0m04:40:33,682 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t7) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 45 [JBossINF] �[0m�[0m04:40:33,731 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t8) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 46 [JBossINF] �[0m�[31m04:40:33,729 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p5-t191) ISPN000136: Error executing command LockControlCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(nJ0-BdoILtH2ru7yFTp8jfBaWzf-sXHJOGtOW-97) in behalf of transaction GlobalTx:perf19:49051. Current owner GlobalTx:perf18:104143. [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:252) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitOn(DefaultPendingLockManager.java:337) [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForAllKeys(DefaultPendingLockManager.java:137) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:153) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:122) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.localLockCommandWork(PessimisticLockingInterceptor.java:209) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.lambda$new$0(PessimisticLockingInterceptor.java:46) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:81) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:192) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:183) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitLockControlCommand(TxInterceptor.java:222) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:22) [JBossINF] at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25) [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70) [JBossINF] at org.infinispan.interceptors.InvocationStage.thenApply(InvocationStage.java:45) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:224) [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:46) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:90) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:160) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:117) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234) [JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:126) [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94) [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99) [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71) [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47) [JBossINF] at java.lang.Thread.run(Thread.java:748)