Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-6723

Remote rolled back because originator is no longer in the cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Minor
    • 7.1.0.DR13
    • 7.1.0.DR7, 7.1.0.DR9, 7.2.0.GA.CR1
    • Clustering
    • None

    Description

      Affects scenario eap-7x-failover-ejb-ejbservlet-undeploy-repl-async-haproxy.

      We saw this error on server directly after new cluster view was received:

      [JBossINF] [0m[31m16:47:23,410 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p6-t3) ISPN000136: Error executing command PrepareCommand, writing keys []: org.infinispan.commons.CacheException: ISPN000332: Remote transaction GlobalTransaction:<perf18>:90907:remote rolled back because originator is no longer in the cluster
[JBossINF] 	at org.infinispan.interceptors.TxInterceptor.verifyRemoteTransaction(TxInterceptor.java:518)
[JBossINF] 	at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:161)
[JBossINF] 	at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
[JBossINF] 	at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
[JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
[JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:100)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)

      This error was logged on perf19, perf20 and perf21 around 200-times during one second at 16:47:23.
      From 16:47:22:557 until 16:47:24:213 WARN messages were logged on client:

      2016/10/27 16:47:23:000 EDT [WARN ][Runner - 980] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 404 Content: <html><head><title>Error</title></head><body>404 - Not Found</body></html>>
        org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 404 Content: <html><head><title>Error</title></head><body>404 - Not Found</body></html>
	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
	at java.lang.Thread.run(Thread.java:745)

2016/10/27 16:47:23:000 EDT [WARN ][Runner - 980] SFCORE_LOG - Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 404 Content: <html><head><title>Error</title></head><body>404 - Not Found</body></html>>
        org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 404 Content: <html><head><title>Error</title></head><body>404 - Not Found</body></html>
	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
	at java.lang.Thread.run(Thread.java:745)

      New cluster view INFO messages were logged directly afterwards:

      16:48:24,272 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ee,perf21) ISPN000094: Received new cluster view for channel server: [perf21|5] (4) [perf21, perf19, perf20, perf18]
[JBossINF] [0m[0m16:48:24,273 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ee,perf21) ISPN000094: Received new cluster view for channel web: [perf21|5] (4) [perf21, perf19, perf20, perf18]
[JBossINF] [0m[0m16:48:24,273 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ee,perf21) ISPN000094: Received new cluster view for channel ejb: [perf21|5] (4) [perf21, perf19, perf20, perf18]
[JBossINF] 16:48:24,273 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ee,perf21) ISPN000094: Received new cluster view for channel hibernate: [perf21|5] (4) [perf21, perf19, perf20, perf18]

      Link to server log:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-ejb-ejbservlet-undeploy-repl-async-haproxy/1/console-perf20/

      Attachments

        Issue Links

          is incorporated by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-7048 Upgrade Infinispan to 8.2.5.Final

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              dcihak@redhat.com Daniel Cihak
              Daniel Cihak Daniel Cihak
              Daniel Cihak Daniel Cihak
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: