Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-9095

NPE during server shutdown when using scattered cache

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 9.4.0.Beta1, 9.3.3.Final
    • 9.2.1.Final
    • Core
    • None

    Description

      We hit NPE when running tests for RFE EAP7-867.

      EAP distribution was built from https://github.com/pferraro/wildfly/tree/scattered .
      Test description: Positive stress test (no failover), 4-node EAP cluster, clients: starting with 400 clients in the beginning, raising the number of clients to 6000 in the end of the test.

      During clean server shutdown in the end of the test, server logged NPE and got stuck:

      [JBossINF] [0m[31m07:55:57,643 ERROR [org.infinispan.scattered.impl.ScatteredStateConsumerImpl] (thread-200,ejb,dev214) ISPN000471: Failed processing values received from remote node during rebalance.: java.lang.NullPointerException
[JBossINF] 	at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.applyValues(ScatteredStateConsumerImpl.java:505)
[JBossINF] 	at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.lambda$getValuesAndApply$8(ScatteredStateConsumerImpl.java:475)
[JBossINF] 	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
[JBossINF] 	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
[JBossINF] 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
[JBossINF] 	at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:66)
[JBossINF] 	at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:56)
[JBossINF] 	at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:35)
[JBossINF] 	at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1304)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1207)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1342)
[JBossINF] 	at org.jgroups.JChannel.up(JChannel.java:819)
[JBossINF] 	at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
[JBossINF] 	at org.jgroups.stack.Protocol.up(Protocol.java:340)
[JBossINF] 	at org.jgroups.protocols.FORK.up(FORK.java:134)
[JBossINF] 	at org.jgroups.protocols.FRAG3.up(FRAG3.java:166)
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
[JBossINF] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
[JBossINF] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
[JBossINF] 	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
[JBossINF] 	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
[JBossINF] 	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
[JBossINF] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
[JBossINF] 	at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
[JBossINF] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
[JBossINF] 	at org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
[JBossINF] 	at org.jgroups.protocols.Discovery.up(Discovery.java:267)
[JBossINF] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
[JBossINF] 	at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] 	at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
[JBossINF] 	at java.lang.Thread.run(Thread.java:748)
[JBossINF]

      Scattered cache was configured with bias-lifespan="0".

      Server configuration:
      http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-scattered-sync/14/artifact/report/config/jboss-dev214/standalone-ha.xml

      Server link:
      http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-scattered-sync/14/console-dev214/

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. WFLY-10389 NPE during server shutdown when using scattered cache

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is incorporated by

          Bug - A problem that impairs or prevents the functions of the product. ISPN-9128 RehashWithSharedStoreTest.testRehashes random failures

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              rvansa1@redhat.com Radim Vansa (Inactive)
              pferraro@redhat.com Paul Ferraro
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: