Loading...

Details

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: 8.2.0.Beta1, 8.2.0.Final, 8.1.1.Final
Affects Version/s: 8.1.0.Final
Component/s: Core
Labels:
None

Git Pull Request:
https://github.com/infinispan/infinispan/pull/3932
Bugzilla References:
https://bugzilla.redhat.com/show_bug.cgi?id=1310583, https://bugzilla.redhat.com/show_bug.cgi?id=1310652

Description

During shutdown, TransactionTable unregisters itself as a view change listener, but it can still receive view change notifications after it stopped the executor service. When that happens, it causes a RejectedExecutionException that is eventually logged by JGroups:

pbcast.GMS  - JGRP000027: failed passing message up
java.lang.RuntimeException: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.util.concurrent.RejectedExecutionException] while invoking method [public void org.infinispan.transaction.TransactionTable.onViewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.transaction.TransactionTable@3d5ab0ba
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:682)
	at org.jgroups.JChannel.up(JChannel.java:733)
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
	at org.jgroups.protocols.RSVP.up(RSVP.java:201)
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
	at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
	at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:146)
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:922)
	at org.jgroups.stack.Protocol.up(Protocol.java:412)
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)
	at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982)
	at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912)
	at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846)
	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
	at org.jgroups.protocols.FD.up(FD.java:255)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
	at org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
	at org.jgroups.protocols.Discovery.up(Discovery.java:291)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1572)
	at org.jgroups.protocols.TP$MyHandler.run(TP.java:1791)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
Caused by: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.util.concurrent.RejectedExecutionException] while invoking method [public void org.infinispan.transaction.TransactionTable.onViewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.transaction.TransactionTable@3d5ab0ba
	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:287)
	at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)
	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:305)
	at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.notifyViewChange(CacheManagerNotifierImpl.java:88)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$NotifyViewChange.emitNotification(JGroupsTransport.java:638)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.viewAccepted(JGroupsTransport.java:708)
	at org.jgroups.blocks.MessageDispatcher.handleUpEvent(MessageDispatcher.java:602)
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:679)
	... 25 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1f5986a3 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@2e964769[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1696]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:546)
	at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:646)
	at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641)
	at org.infinispan.transaction.TransactionTable.onViewChange(TransactionTable.java:491)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:282)
	... 32 more

The exception is harmless for the stopping cache, the problem is that the following view change listeners are also skipped. We should fix both TransactionTable to avoid throwing the exception, and CacheManagerNotifier to ignore any exceptions during view changes.

Attachments

Issue Links

blocks

JDG-934 TransactionTable should ignore view changes during shutdown

Closed

TransactionTable should ignore view changes during shutdown

Details

Description

Attachments

Issue Links

Activity

People

Dates