Details
-
Bug
-
Resolution: Done
-
Major
-
12.1.0.Final
-
None
Description
When a proto schema X.proto is removed, ProtobufMetadataManagerInterceptorTest.testStateTransfer adds Y.proto.error keys for all the other schemas that were referencing X.proto and are now unresolved. It ALSO removes Y.proto.error keys (if present) for all the schemas that are still resolved, even though removing X.proto doesn't couldn't have changed their status from unresolved to resolved.
Since ISPN-11832, the removals of the Y.proto.error keys are done in parallel, but they are still using the same InvocationContext, and InvocationContext is not thread-safe.
The concurrent modification of InvocationContext leads to some entries being lost and to random failures in ProtobufMetadataManagerInterceptorTest, e.g.
10:23:41,603 TRACE (testng-Test:[]) [InvocationContextInterceptor] Invoked with command RemoveCommand{key=state.proto, value=null, metadata=null, internalMetadata=null, flags=[], commandInvocationId=, valueMatcher=MATCH_ALWAYS, topologyId=-1} and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext@2ee09de7] 10:23:41,609 TRACE (jgroups-7,Test-NodeA:[]) [InvocationContextInterceptor] Invoked with command RemoveCommand{key=any.proto.errors, value=null, metadata=null, internalMetadata=null, flags=[], commandInvocationId=, valueMatcher=MATCH_ALWAYS, topologyId=-1} and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext@2ee09de7] 10:23:41,610 TRACE (jgroups-7,Test-NodeA:[]) [InvocationContextInterceptor] Invoked with command RemoveCommand{key=org/infinispan/query/remote/client/query.proto.errors, value=null, metadata=null, internalMetadata=null, flags=[], commandInvocationId=, valueMatcher=MATCH_ALWAYS, topologyId=-1} and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext@2ee09de7] 10:23:41,611 TRACE (jgroups-5,Test-NodeA:[]) [QueueAsyncInvocationStage] Resuming invocation of command LockControlCommand{cache=defaultcache, keys=[org/infinispan/query/remote/client/query.proto.errors], flags=[], unlock=false, gtx=GlobalTransaction{id=220, addr=Test-NodeA, remote=false, xid=null, internalId=-1}} with 1 handlers 10:23:41,611 TRACE (jgroups-6,Test-NodeA:[]) [QueueAsyncInvocationStage] Resuming invocation of command LockControlCommand{cache=defaultcache, keys=[any.proto.errors], flags=[], unlock=false, gtx=GlobalTransaction{id=220, addr=Test-NodeA, remote=false, xid=null, internalId=-1}} with 1 handlers 10:23:41,611 TRACE (jgroups-6,Test-NodeA:[]) [EntryFactoryImpl] Added context entry ReadCommittedEntry(3c1324c3){key=any.proto.errors, value=null, oldValue=null, isCreated=true, isChanged=false, isRemoved=false, isExpired=false, isCommited=false, skipLookup=false, metadata=null, oldMetadata=null, internalMetadata=null} 10:23:41,611 TRACE (jgroups-5,Test-NodeA:[]) [EntryFactoryImpl] Added context entry ReadCommittedEntry(1ba42578){key=org/infinispan/query/remote/client/query.proto.errors, value=null, oldValue=null, isCreated=true, isChanged=false, isRemoved=false, isExpired=false, isCommited=false, skipLookup=false, metadata=null, oldMetadata=null, internalMetadata=null} # Key any.proto.errors was added to invocation context but if not found 10:23:41,612 TRACE (jgroups-6,Test-NodeA:[]) [BaseDistributionInterceptor] Key any.proto.errors became local after wrapping, retrying command. Command topology is 9, current topology is 9 10:23:41,612 ERROR (jgroups-6,Test-NodeA:[]) [InvocationContextInterceptor] ISPN000136: Error executing command RemoveCommand on Cache 'defaultcache', writing keys [any.proto.errors] java.lang.IllegalStateException: null at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.remoteGetSingleKey(BaseDistributionInterceptor.java:169) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.remoteGetSingleKey(TxDistributionInterceptor.java:560) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.handleTxWriteCommand(TxDistributionInterceptor.java:411) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRemoveCommand(TxDistributionInterceptor.java:141) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:53) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:232) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:758) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:414) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:53) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:55) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.DDAsyncInterceptor.visitRemoveCommand(DDAsyncInterceptor.java:66) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:53) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.query.remote.impl.ProtobufMetadataManagerInterceptor.visitRemoveCommand(ProtobufMetadataManagerInterceptor.java:387) ~[classes/:?] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:53) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:27) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.SyncInvocationStage.thenApply(SyncInvocationStage.java:44) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:246) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLockAndInvokeNext(PessimisticLockingInterceptor.java:285) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.lambda$visitDataWriteCommand$3(PessimisticLockingInterceptor.java:157) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25) ~[infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:124) [infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:87) [infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:33) [infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) [?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) [?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) [?:?] at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:67) [infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:105) [infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51) [infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1415) [infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1317) [infinispan-core-12.1.1-SNAPSHOT.jar:12.1.1-SNAPSHOT]