Details
-
Bug
-
Resolution: Done
-
Major
-
None
-
8.0.0.Beta-CR1
-
None
Description
We have 4 nodes cluster and we have deployed a clustered application using EJB + Servlet; the application is un-deployed and then re-deployed from each node of the cluster in sequence;
After the application is un-deployed from Node-1 we observe the following errors which are basically endless and go on an on until the test ends and the nodes are finally shut down (complete logs in wildfly.zip
):
Node-1:
2023-03-23 10:46:19,040 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 89) WFLYUT0022: Unregistered web context: '/clusterbench' from server 'default-server'
2023-03-23 10:46:19,041 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 90) WFLYUT0022: Unregistered web context: '/clusterbench-passivating' from server 'default-server'
2023-03-23 10:46:19,044 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 94) WFLYUT0022: Unregistered web context: '/clusterbench-granular' from server 'default-server'
2023-03-23 10:46:19,071 ERROR [io.undertow.request] (default task-2) UT005023: Exception handling request to /clusterbench/ejbservlet: org.jboss.as.ejb3.component.EJBComponentUnavailableException: WFLYEJB0421: Invocation cannot proceed as component is shutting down
at org.jboss.as.ejb3@8.0.0.GA-redhat-20230321//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:59)
at org.jboss.invocation@2.0.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3@8.0.0.GA-redhat-20230321//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation@2.0.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee@8.0.0.GA-redhat-20230321//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation@2.0.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
...
Node-2:
2023-03-23 10:47:54,473 INFO [org.infinispan.LIFECYCLE] (thread-42,ejb,wildfly2) [Context=clusterbench-ee10.ear/clusterbench-ee10-ejb.jar/default] ISPN100010: Finished rebalance with members [wildfly2, wildfly3, wildfly4, wildfly1], topology id 19
2023-03-23 10:49:31,111 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 23) WFLYUT0022: Unregistered web context: '/web-logger' from server 'default-server'
2023-03-23 10:49:31,192 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment web-logger.war (runtime-name: web-logger.war) in 85ms
2023-03-23 10:49:31,297 INFO [org.jboss.as.repository] (management-handler-thread - 1) WFLYDR0002: Content removed from location /tmp/tests-clustering/jboss-eap-2/standalone/data/content/4a/d32c946992dc12f8358cd103e4f04e5fc09cc6/content
2023-03-23 10:49:31,298 INFO [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0009: Undeployed "web-logger.war" (runtime-name: "web-logger.war")
2023-03-23 10:49:31,448 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 89) WFLYUT0022: Unregistered web context: '/clusterbench-passivating' from server 'default-server'
2023-03-23 10:49:31,449 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 90) WFLYUT0022: Unregistered web context: '/clusterbench' from server 'default-server'
2023-03-23 10:49:31,475 ERROR [io.undertow.request] (default task-12) UT005023: Exception handling request to /clusterbench/ejbservlet: java.lang.NullPointerException
at org.jboss.as.weld.ejb@8.0.0.GA-redhat-20230321//org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.isRemoved(StatefulSessionObjectReferenceImpl.java:133)
at org.jboss.weld.core@5.1.0.Final-redhat-00001//org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:126)
at org.jboss.weld.core@5.1.0.Final-redhat-00001//org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
Node-3:
2023-03-23 10:49:41,715 INFO [org.infinispan.LIFECYCLE] (thread-45,ejb,wildfly3) [Context=default-server] ISPN100002: Starting rebalance with members [wildfly3, wildfly4, wildfly1], phase READ_OLD_WRITE_ALL, topology id 24
2023-03-23 10:49:41,716 INFO [org.infinispan.LIFECYCLE] (non-blocking-thread--p8-t3) [Context=default-server] ISPN100010: Finished rebalance with members [wildfly3, wildfly4, wildfly1], topology id 24
2023-03-23 10:49:45,808 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p15-t1) ISPN000136: Error executing command GetKeyValueCommand on Cache 'clusterbench-ee10.ear.clusterbench-ee10-web.war', writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 127453 from wildfly4 after 10 seconds
at org.infinispan@14.0.7.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:171)
at org.infinispan@14.0.7.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:88)
at org.infinispan@14.0.7.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Node-4:
topology id 24
2023-03-23 10:49:41,714 INFO [org.infinispan.LIFECYCLE] (thread-59,ejb,wildfly4) [Context=default-server] ISPN100002: Starting rebalance with members [wildfly3, wildfly4, wildfly1], phase READ_OLD_WRITE_ALL, topology id 24
2023-03-23 10:49:41,715 INFO [org.infinispan.LIFECYCLE] (non-blocking-thread--p4-t2) [Context=default-server] ISPN100010: Finished rebalance with members [wildfly3, wildfly4, wildfly1], topology id 24
2023-03-23 10:49:46,790 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p16-t1) ISPN000136: Error executing command GetKeyValueCommand on Cache 'clusterbench-ee10.ear.clusterbench-ee10-web.war', writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey(wHvSa_Pbl8i4e33vq9JkOd6qJfj-5fDgOTJYD5cY) and requestor GlobalTransaction{id=158455, addr=wildfly4, remote=false, xid=null, internalId=-1}. Lock is held by GlobalTransaction{id=158298, addr=wildfly2, remote=true, xid=null, internalId=-1} (pending)
at org.infinispan@14.0.7.Final//org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:148)
at org.infinispan@14.0.7.Final//org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager$PendingLockPromiseImpl.onRelease(DefaultPendingLockManager.java:328)
at org.infinispan@14.0.7.Final//org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager$PendingLockPromiseImpl.call(DefaultPendingLockManager.java:293)
at org.infinispan@14.0.7.Final//org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager$PendingLockPromiseImpl.call(DefaultPendingLockManager.java:246)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
The client is affected by this error;