Details
-
Bug
-
Resolution: Obsolete
-
Minor
-
None
-
7.4.0.CD19, 7.4.0.Beta
-
None
Description
We got java.lang.NullPointerException in Clustering tests where a scattered cache is used;
The client fail rate is very low (0.002%);
This is the server configuration:
embed-server --server-config=standalone-ha.xml /subsystem=modcluster/mod-cluster-config=configuration:write-attribute(name=advertise,value=false) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=mod_cluster:add(host=10.0.144.220,port=7000) /subsystem=modcluster/mod-cluster-config=configuration:write-attribute(name=proxies,value=[mod_cluster]) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) /subsystem=infinispan/cache-container=web/scattered-cache=testScattered:add() /subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testScattered)
The error happens on node3 right after the first node in cluster (node1) is failed (see eap-7.x-clustering-http-session-shutdown-scattered#134):
2020-04-02 19:16:45,600 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-4) New session created: wD2cb7cc0ic74mZR1jbxqSfsWrXL923hY5Ywfc-L with 4kB cargo 2020-04-02 19:17:19,198 INFO [org.infinispan.CLUSTER] (thread-22,ejb,wildfly3) ISPN000094: Received new cluster view for channel ejb: [wildfly2|4] (3) [wildfly2, wildfly3, wildfly4] 2020-04-02 19:17:19,199 INFO [org.infinispan.CLUSTER] (thread-22,ejb,wildfly3) ISPN100001: Node wildfly1 left the cluster 2020-04-02 19:17:19,206 INFO [org.infinispan.CLUSTER] (thread-22,ejb,wildfly3) ISPN000094: Received new cluster view for channel ejb: [wildfly2|4] (3) [wildfly2, wildfly3, wildfly4] 2020-04-02 19:17:19,207 INFO [org.infinispan.CLUSTER] (thread-22,ejb,wildfly3) ISPN100001: Node wildfly1 left the cluster 2020-04-02 19:17:19,207 INFO [org.infinispan.CLUSTER] (thread-22,ejb,wildfly3) ISPN000094: Received new cluster view for channel ejb: [wildfly2|4] (3) [wildfly2, wildfly3, wildfly4] 2020-04-02 19:17:19,207 INFO [org.infinispan.CLUSTER] (thread-22,ejb,wildfly3) ISPN100001: Node wildfly1 left the cluster 2020-04-02 19:17:19,217 INFO [org.infinispan.CLUSTER] (thread-22,ejb,wildfly3) ISPN000094: Received new cluster view for channel ejb: [wildfly2|4] (3) [wildfly2, wildfly3, wildfly4] 2020-04-02 19:17:19,218 INFO [org.infinispan.CLUSTER] (thread-22,ejb,wildfly3) ISPN100001: Node wildfly1 left the cluster 2020-04-02 19:17:19,384 ERROR [stderr] (KeyAffinityService - 2) Exception in thread "KeyAffinityService - 2" java.lang.NullPointerException 2020-04-02 19:17:19,384 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.util.concurrent.ConcurrentHashMap.containsKey(ConcurrentHashMap.java:964) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan.commons@9.4.18.Final-redhat-00001//org.infinispan.commons.util.concurrent.ConcurrentHashSet.contains(ConcurrentHashSet.java:65) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl.interestedInAddress(KeyAffinityServiceImpl.java:336) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl.access$800(KeyAffinityServiceImpl.java:42) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl$KeyGeneratorWorker.generateKeys(KeyAffinityServiceImpl.java:268) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl$KeyGeneratorWorker.run(KeyAffinityServiceImpl.java:238) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at org.jboss.as.clustering.common@7.4.0.CD19-redhat-00001//org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49) 2020-04-02 19:17:19,385 ERROR [stderr] (KeyAffinityService - 2) at org.jboss.as.clustering.common@7.4.0.CD19-redhat-00001//org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70) 2020-04-02 19:17:19,386 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.lang.Thread.run(Thread.java:834) 2020-04-02 19:17:19,386 ERROR [stderr] (KeyAffinityService - 2) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Complete logs attached;
Note the error is present also in stress tests where there is no fail-over: for example in run eap-7.x-clustering-stress-session-scattered#92 we have four nodes (node1, node2, node3, node4) and the error happens on nodes 2,3,4 at the end of the test right when node1 receives the stop command:
node1:
2020-04-02 17:51:36,745 INFO [org.jboss.as.server] (management-handler-thread - 2) WFLYSRV0211: Suspending server with 15000 ms timeout.
node2,3,4:
2020-04-02 17:51:36,821 ERROR [stderr] (KeyAffinityService - 1) Exception in thread "KeyAffinityService - 1" java.lang.NullPointerException 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at java.base/java.util.concurrent.ConcurrentHashMap.containsKey(ConcurrentHashMap.java:964) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at org.infinispan.commons@9.4.18.Final-redhat-00001//org.infinispan.commons.util.concurrent.ConcurrentHashSet.contains(ConcurrentHashSet.java:65) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl.interestedInAddress(KeyAffinityServiceImpl.java:336) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl.access$800(KeyAffinityServiceImpl.java:42) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl$KeyGeneratorWorker.generateKeys(KeyAffinityServiceImpl.java:268) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl$KeyGeneratorWorker.run(KeyAffinityServiceImpl.java:238) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at org.jboss.as.clustering.common@7.4.0.CD19-redhat-00001//org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49) 2020-04-02 17:51:36,822 ERROR [stderr] (KeyAffinityService - 1) at org.jboss.as.clustering.common@7.4.0.CD19-redhat-00001//org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70) 2020-04-02 17:51:36,823 ERROR [stderr] (KeyAffinityService - 1) at java.base/java.lang.Thread.run(Thread.java:834) 2020-04-02 17:51:36,823 ERROR [stderr] (KeyAffinityService - 1) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:485) 2020-04-02 17:51:36,841 ERROR [stderr] (KeyAffinityService - 3) Exception in thread "KeyAffinityService - 3" java.lang.NullPointerException 2020-04-02 17:51:36,842 ERROR [stderr] (KeyAffinityService - 3) at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) 2020-04-02 17:51:36,842 ERROR [stderr] (KeyAffinityService - 3) at java.base/java.util.concurrent.ConcurrentHashMap.containsKey(ConcurrentHashMap.java:964) 2020-04-02 17:51:36,842 ERROR [stderr] (KeyAffinityService - 3) at org.infinispan.commons@9.4.18.Final-redhat-00001//org.infinispan.commons.util.concurrent.ConcurrentHashSet.contains(ConcurrentHashSet.java:65) 2020-04-02 17:51:36,842 ERROR [stderr] (KeyAffinityService - 3) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl.interestedInAddress(KeyAffinityServiceImpl.java:336) 2020-04-02 17:51:36,842 ERROR [stderr] (KeyAffinityService - 3) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl.access$800(KeyAffinityServiceImpl.java:42) 2020-04-02 17:51:36,842 ERROR [stderr] (KeyAffinityService - 3) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl$KeyGeneratorWorker.generateKeys(KeyAffinityServiceImpl.java:268) 2020-04-02 17:51:36,843 ERROR [stderr] (KeyAffinityService - 3) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl$KeyGeneratorWorker.run(KeyAffinityServiceImpl.java:238) 2020-04-02 17:51:36,843 ERROR [stderr] (KeyAffinityService - 3) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2020-04-02 17:51:36,843 ERROR [stderr] (KeyAffinityService - 3) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2020-04-02 17:51:36,843 ERROR [stderr] (KeyAffinityService - 3) at org.jboss.as.clustering.common@7.4.0.CD19-redhat-00001//org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49) 2020-04-02 17:51:36,843 ERROR [stderr] (KeyAffinityService - 3) at org.jboss.as.clustering.common@7.4.0.CD19-redhat-00001//org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70) 2020-04-02 17:51:36,843 ERROR [stderr] (KeyAffinityService - 3) at java.base/java.lang.Thread.run(Thread.java:834) 2020-04-02 17:51:36,843 ERROR [stderr] (KeyAffinityService - 3) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:485) 2020-04-02 17:51:36,853 ERROR [stderr] (KeyAffinityService - 2) Exception in thread "KeyAffinityService - 2" java.lang.NullPointerException 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.util.concurrent.ConcurrentHashMap.containsKey(ConcurrentHashMap.java:964) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan.commons@9.4.18.Final-redhat-00001//org.infinispan.commons.util.concurrent.ConcurrentHashSet.contains(ConcurrentHashSet.java:65) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl.interestedInAddress(KeyAffinityServiceImpl.java:336) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl.access$800(KeyAffinityServiceImpl.java:42) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl$KeyGeneratorWorker.generateKeys(KeyAffinityServiceImpl.java:268) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at org.infinispan@9.4.18.Final-redhat-00001//org.infinispan.affinity.impl.KeyAffinityServiceImpl$KeyGeneratorWorker.run(KeyAffinityServiceImpl.java:238) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at org.jboss.as.clustering.common@7.4.0.CD19-redhat-00001//org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49) 2020-04-02 17:51:36,854 ERROR [stderr] (KeyAffinityService - 2) at org.jboss.as.clustering.common@7.4.0.CD19-redhat-00001//org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70) 2020-04-02 17:51:36,855 ERROR [stderr] (KeyAffinityService - 2) at java.base/java.lang.Thread.run(Thread.java:834) 2020-04-02 17:51:36,855 ERROR [stderr] (KeyAffinityService - 2) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:485)