Uploaded image for project: 'JBoss Enterprise Application Platform 6'
  1. JBoss Enterprise Application Platform 6
  2. JBPAPP6-1274

IllegalStateException: Cache is in 'TERMINATED'/'STOPPING' state.

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Migrated to another ITS
    • EAP 6.0.1 ER 2, EAP 6.0.1 ER 3, EAP 6.0.1
    • TBD EAP 6
    • Clustering
    • None
    • Hide
      1. node1 is running with clusterbench (https://github.com/rhusar/clusterbench) deployed
      2. node2 deploys clusterbench
      3. undeploy clusterbench from node1 soon after it is deployed on node2
      4. some of subsequent client requests will fail with 500 and this exception (node1)

      I was able to reproduce this issue manually, roughly 1 in 7 tries will succeed at reproducing this.

      Show
      node1 is running with clusterbench ( https://github.com/rhusar/clusterbench ) deployed node2 deploys clusterbench undeploy clusterbench from node1 soon after it is deployed on node2 some of subsequent client requests will fail with 500 and this exception (node1) I was able to reproduce this issue manually, roughly 1 in 7 tries will succeed at reproducing this.

    Description

      There was an exception during manual ipv6 clustering failover testing, scenario ejbservlet-repl-async-undeploy. Full stack trace is below:

      06:11:11,653 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host]] (ajp-/2620:52:0:105f:0:0:ffff:4%2:8009-4) Exception sending request initialized lifecycle event to listener instance of class org.jboss.weld.servlet.WeldListener: java.lang.IllegalStateException: Cache 'org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91)
      	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
      	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
      	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
      	at org.infinispan.interceptors.IsMarshallableInterceptor.visitGetKeyValueCommand(IsMarshallableInterceptor.java:94)
      	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:80)
      	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
      	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
      	at org.infinispan.CacheImpl.get(CacheImpl.java:278)
      	at org.infinispan.CacheImpl.get(CacheImpl.java:270)
      	at org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:303)
      	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore$2.invoke(InfinispanBackingCacheEntryStore.java:199)
      	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore$2.invoke(InfinispanBackingCacheEntryStore.java:196)
      	at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:19)
      	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:60)
      	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.invoke(InfinispanBackingCacheEntryStore.java:243)
      	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.get(InfinispanBackingCacheEntryStore.java:202)
      	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.get(SerializationGroupMemberContainer.java:290) [jboss-as-ejb3-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
      	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.get(SerializationGroupMemberContainer.java:51) [jboss-as-ejb3-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
      	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.peek(PassivatingBackingCacheImpl.java:205) [jboss-as-ejb3-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
      	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.contains(AbstractCache.java:78) [jboss-as-ejb3-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
      	at org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.isRemoved(StatefulSessionObjectReferenceImpl.java:164) [jboss-as-weld-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
      	at org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.getBusinessObject(StatefulSessionObjectReferenceImpl.java:123) [jboss-as-weld-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
      	at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:108) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
      	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
      	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
      	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.toString(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java) [clusterbench-ee6-ejb.jar:]
      	at java.lang.String.valueOf(String.java:2826) [rt.jar:1.6.0_30]
      	at java.lang.StringBuilder.append(StringBuilder.java:115) [rt.jar:1.6.0_30]
      	at org.jboss.weld.context.SerializableContextualInstanceImpl.toString(SerializableContextualInstanceImpl.java:60) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
      	at java.lang.String.valueOf(String.java:2826) [rt.jar:1.6.0_30]
      	at java.lang.StringBuilder.append(StringBuilder.java:115) [rt.jar:1.6.0_30]
      	at org.jboss.weld.context.beanstore.AttributeBeanStore.attach(AttributeBeanStore.java:109) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
      	at org.jboss.weld.context.AbstractBoundContext.activate(AbstractBoundContext.java:66) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
      	at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:141) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [jbossweb-7.0.17.Final-redhat-1.jar:]
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.17.Final-redhat-1.jar:]
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.17.Final-redhat-1.jar:]
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372) [jbossweb-7.0.17.Final-redhat-1.jar:]
      	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.17.Final-redhat-1.jar:]
      	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453) [jbossweb-7.0.17.Final-redhat-1.jar:]
      	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) [jbossweb-7.0.17.Final-redhat-1.jar:]
      	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      

      This occurred for cache in TERMINATED as well as STOPPING state and triggered code 500 for the client. Although the issue occurred on RHEL6-x86_64 with ipv6 it may not be limited to this environment.

      Attachments

        Activity

          People

            pferraro@redhat.com Paul Ferraro
            rjanik@redhat.com Richard Janik
            Archiver:
            samahaja@redhat.com Sagar Mahajan

            Dates

              Created:
              Updated:
              Resolved:
              Archived:

              PagerDuty