Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-6043

TransactionTable should ignore view changes during shutdown

XMLWordPrintable

      During shutdown, TransactionTable unregisters itself as a view change listener, but it can still receive view change notifications after it stopped the executor service. When that happens, it causes a RejectedExecutionException that is eventually logged by JGroups:

      pbcast.GMS  - JGRP000027: failed passing message up
      java.lang.RuntimeException: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.util.concurrent.RejectedExecutionException] while invoking method [public void org.infinispan.transaction.TransactionTable.onViewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.transaction.TransactionTable@3d5ab0ba
      	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:682)
      	at org.jgroups.JChannel.up(JChannel.java:733)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
      	at org.jgroups.protocols.RSVP.up(RSVP.java:201)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
      	at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
      	at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:146)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:922)
      	at org.jgroups.stack.Protocol.up(Protocol.java:412)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
      	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)
      	at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982)
      	at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912)
      	at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
      	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
      	at org.jgroups.protocols.FD.up(FD.java:255)
      	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
      	at org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:291)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1572)
      	at org.jgroups.protocols.TP$MyHandler.run(TP.java:1791)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      Caused by: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.util.concurrent.RejectedExecutionException] while invoking method [public void org.infinispan.transaction.TransactionTable.onViewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.transaction.TransactionTable@3d5ab0ba
      	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:287)
      	at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)
      	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:305)
      	at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.notifyViewChange(CacheManagerNotifierImpl.java:88)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$NotifyViewChange.emitNotification(JGroupsTransport.java:638)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.viewAccepted(JGroupsTransport.java:708)
      	at org.jgroups.blocks.MessageDispatcher.handleUpEvent(MessageDispatcher.java:602)
      	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:679)
      	... 25 more
      Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1f5986a3 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@2e964769[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1696]
      	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
      	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
      	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
      	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:546)
      	at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:646)
      	at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641)
      	at org.infinispan.transaction.TransactionTable.onViewChange(TransactionTable.java:491)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:601)
      	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:282)
      	... 32 more
      

      The exception is harmless for the stopping cache, the problem is that the following view change listeners are also skipped. We should fix both TransactionTable to avoid throwing the exception, and CacheManagerNotifier to ignore any exceptions during view changes.

              dberinde@redhat.com Dan Berindei (Inactive)
              dberinde@redhat.com Dan Berindei (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: