-
Bug
-
Resolution: Done
-
Critical
-
8.0.0.GA-CR1
-
False
-
None
-
False
-
Regression
-
-
-
-
-
-
-
-
Scenario: we have a 4 nodes cluster where each node is configured as follows:
embed-server --server-config=standalone-ha.xml if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) else /subsystem=jgroups:write-attribute(name=default-stack,value=tcp) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) end-if /subsystem=infinispan/cache-container=web/scattered-cache=testScattered:add() /subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testScattered)
We use MOD_CLUSTER as load balance;
Fail over is triggered by shutting down each node in a sequence;
When the the first node is shut down and then re-started, we observe the following error during re-start (complete logs in wildfly-http-session-shutdown-scattered#200.zip
):
2023-09-27 16:22:50,508 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 81) WFLYCLINF0002: Started clusterbench-ee10.ear.clusterbench-ee10-web.war cache from web container
2023-09-27 16:22:50,695 WARN [org.infinispan.CLUSTER] (thread-23,ejb,wildfly1) ISPN000071: Caught exception when handling command InvalidateVersionsCommand{topologyId=-1, removed=false: SessionAccessMetaDataKey(5gN8SEXMZo4eMr0D8PWY0xCJqjNde-jwJ3ZijCfF) -> 470, SessionAccessMetaDataKey(py3K7yTaJeOwmSQzMvBfmWBh2baD44usZLH0sWRL) -> 448, SessionAccessMetaDataKey(anlwS91d09ZZ08jTHik7wXzUXDVCziUEDuQvDBVP) -> 475, SessionAccessMetaDataKey(paWPnr0ylHljFR_AB2FYd8sANCDwjBTAhj9pYNfh) -> 291, SessionAccessMetaDataKey(yoYNUa36YxKSXKSWt2ruayrtaNr3ZWpRhkJFfNVX) -> 595, SessionAccessMetaDataKey(rRdoeVEZINGkaZnMOt3PF2hnTcpJ4XQ3hrYA7ew-) -> 769, SessionAccessMetaDataKey(_jReiLmA7v4WPXo9JGZmew5JqjShSkECMtqer3_3) -> 953, SessionAttributesKey(wbzAwzGcVWGmOC-UPxOLWxOHTIiomN0Wr2zL3F1O) -> 1063, SessionAccessMetaDataKey(rub3djV0D3sKf4XjulNCgaTVgN_Yx1BWqqkcIR1b) -> 589, SessionAttributesKey(rr1-XLKUhCblLIY-Yg6Vx-Zpx87WwUV-UlzaM4cY) -> 1067, SessionAttributesKey(DO8vtvR1hih3HausToK-WxN8U-9NFSvsnhzUV0KE) -> 600, SessionAccessMetaDataKey(iMq--Igc0To2K0FeN-dogYsnIHFKO0iulQZI9yHZ) -> 553, SessionAttributesKey(QLpFEzg9e2oPeDRqwwGkpe5DxXZI8e1-CINjloMJ) -> 768, SessionAccessMetaDataKey(rx5FABeeYbAdCvcwF7xdj3pj10YUe9JCBDMYQG_v) -> 278, SessionAccessMetaDataKey(3OgZdvXCGc_wqd1peBhPRlRP_haY8jVv-rD9XXVx) -> 773, SessionAttributesKey(sCADXNBWwdhsHoinyR_m3dN408XiaiqciCKVHAT2) -> 298, SessionAccessMetaDataKey(CA5tODEyCBUlnHgvU4cWihcRBJ_YnBnfV4Ut4coG) -> 593, SessionAttributesKey(CYx3y2b52Y3-gn9XlddOqDlMA4Wjd54_vXYTmgEZ) -> 865, SessionAttributesKey(tUz4m5zGAQuCw7muiPeWe0P8UNDNS1T_7WCGbFye) -> 923, SessionAttributesKey(vI93BCX5juIxaH_Wg2-2Rr-PlVeIiG3tCgyNfS5x) -> 760, SessionAccessMetaDataKey(LKAxz-k1UOwVrR3Kzse5wsJnlt7I6U5tOZv3RQnr) -> 955, SessionAccessMetaDataKey(3IS24GzTFwq1mi8DMRm2Dudv74rtjMb7r70OEmJn) -> 532, SessionAccessMetaDataKey(s4aJHMBU9gY48AeKJ0YmBcbemewUbsjWCdDapybj) -> 870, SessionAccessMetaDataKey(xDS8kA01UfKuII63qlvhBR50pBvX4H6PXXrnbLIj) -> 409, SessionAttributesKey(09WPQFsfrsyYyx40JZxTj010ZBAGl-K1MUpBCuMY) -> 874, SessionAccessMetaDataKey(JU9fP3s5fnzq843-PNktBOX4HaDnVnvwofHUoO8j) -> 361, SessionAttributesKey(X-O9DdPVYJ7hf47WsFyeOoeTBoo3uNSdvrQ6B7QT) -> 792, SessionAttributesKey(tBFVHCv90QrYAOgyjoP9RgdkSZ7XS6LtyZa4Yk37) -> 643, SessionAttributesKey(NVoBD3FmG67MKjj1P0C5oT6CTVrOQZhjn78SGM23) -> 556, SessionAttributesKey(OdAEA-G3hefqAhK7OJ5kZ4L3c8CV5wKzMPVULRzK) -> 1108, SessionAccessMetaDataKey(nh1S6AgCvnEhhlJLRSaiSjGoFUmmjea8rYzfAJA2) -> 597, SessionAccessMetaDataKey(dKkMaRxlz-nqbeQnVtFQFu3qT9Mbab_82c1dO8Rt) -> 1151, SessionAttributesKey(UdrzOOddaleuzKjg5Yd3mEF8DcsjDrZYUq5Fu9Z_) -> 271, SessionAttributesKey(8gtaMkir3pgrBGYb2BK4KJpwDp54JjGXwDW_tqHZ) -> 839, SessionAccessMetaDataKey(lrE1U715i-RtWNB4FnzrsH4brtY6I5b8KJdepOGP) -> 640, SessionAccessMetaDataKey(9TEZzyobTqywRB3dbezyAahQAawSJ6ySs8qQODfL) -> 1044, SessionAccessMetaDataKey(W9Xq2puhVc3eQsfznWK-sDszLtqHPO3nbf8Q3-V-) -> 601, SessionAttributesKey(m-nDjdHZ4MdlWlx2-UUbmjPitr1RL0OP298Hf0HA) -> 708, SessionAttributesKey(S0EYCkaK4DQkaj3HKG2cPOCJc5s7h9UIhHyU-AKY) -> 699, SessionAttributesKey(qjk_c6fsxooGQ-TXV9oXQUTugYs4cWFvgkdHDmlr) -> 667, SessionAttributesKey(9TEZzyobTqywRB3dbezyAahQAawSJ6ySs8qQODfL) -> 1045, SessionAccessMetaDataKey(8gtaMkir3pgrBGYb2BK4KJpwDp54JjGXwDW_tqHZ) -> 838, SessionAccessMetaDataKey(SAoshjrjw7yefcHOb4GGNzkH8jIaKa5I22j5DpNc) -> 462, SessionAttributesKey(Ym3QTBJf4kFE6k-WpYlAbyNAY9E194prRo3cG7CG) -> 756, SessionAccessMetaDataKey(TdTKLPwOMQrjzwFHTJXiXSdnwORad5dUe0xlVf_6) -> 866, SessionAccessMetaDataKey(Mez0WZgjNekcx99DdMYhJOZ1RJxsoItH8OPCuQ9j) -> 919, SessionAccessMetaDataKey(qjk_c6fsxooGQ-TXV9oXQUTugYs4cWFvgkdHDmlr) -> 666, SessionAttributesKey(1ZJep2PuhhipUDo9xrfrTq0SP5ICWUiZBqUGz1m8) -> 412, SessionAccessMetaDataKey(UdrzOOddaleuzKjg5Yd3mEF8DcsjDrZYUq5Fu9Z_) -> 270, SessionAttributesKey(rx5FABeeYbAdCvcwF7xdj3pj10YUe9JCBDMYQG_v) -> 279, SessionAccessMetaDataKey(53Gop1u28K9YuX6xXL4ve-wXjKbSGPMeyzgiLl0r) -> 1042, SessionAccessMetaDataKey(tBFVHCv90QrYAOgyjoP9RgdkSZ7XS6LtyZa4Yk37) -> 642, SessionAccessMetaDataKey(wbzAwzGcVWGmOC-UPxOLWxOHTIiomN0Wr2zL3F1O) -> 1062, SessionAttributesKey(NmABev_p_Nl_OqPdTgvEeByHBQYfdwkNwnWspdbj) -> 962, SessionAccessMetaDataKey(9YDOvI8Eyh3ElGhZttWNLeEbQjj4J79bnJ6VQ55s) -> 565, SessionAccessMetaDataKey(-heicHqrj9X1E0kTeu7_pFTeKD4_Om4cZ9mtHvdL) -> 1068, SessionAccessMetaDataKey(fUcI5JWA4NsQvFAkHuW1EDMHxmm68jfwnXJE7k-2) -> 666, SessionAttributesKey(R9CjvbklzwsNCPvfhSfcTcvfBqdT_qljwnAWjFeC) -> 875, SessionAccessMetaDataKey(LtxPD8dolg2mw59Xqzcgnr7iasBGvLt7zkkd5vGz) -> 397, SessionAttributesKey(wavUkmhAykrERdJkkt4rGq6bT414xujYV9gGMdF6) -> 873, SessionAttributesKey(2Pd0aWJkPpAEAqtlA5rlGhHTmyLIBop6zBVsVGMY) -> 643, SessionAttributesKey(kO--n_SiZmZjjp5PSlsSk1P_LlMx8iHKmaJaW85N) -> 649, SessionAccessMetaDataKey(sCADXNBWwdhsHoinyR_m3dN408XiaiqciCKVHAT2) -> 297, SessionAccessMetaDataKey(yw-E90OApAdwffnaiw0RnNt4J_kAo8jTcUcqOmyk) -> 840, SessionAttributesKey(SqgGA6gHg7XXEoidKvbWVcpn8jGTkdhk613iyfGQ) -> 964, SessionAccessMetaDataKey(ul0qfb8FZFJSbq0F0wQIbk7nOCXCs98gdRJIzWzz) -> 765, SessionAccessMetaDataKey(BNXfDgJvremICRftc0ha9S4HpbrXMR9tNMOY1ZT9) -> 662, SessionAttributesKey(CA5tODEyCBUlnHgvU4cWihcRBJ_YnBnfV4Ut4coG) -> 594, SessionAttributesKey(-ikLVTM4xryzo38j7buKYDBBs-TPFwmcX-TWgGys) -> 236, SessionAttributesKey(8sD03wOy6yHLupiqcnR_opENSWTEKp94YrkBuHrj) -> 531, SessionAccessMetaDataKey(NE1neM0JNXJMYgaIjOm9VfKBTNqyJK5LokrzqSAs) -> 753, SessionAccessMetaDataKey(09WPQFsfrsyYyx40JZxTj010ZBAGl-K1MUpBCuMY) -> 873, SessionAccessMetaDataKey(tUz4m5zGAQuCw7muiPeWe0P8UNDNS1T_7WCGbFye) -> 922, SessionAccessMetaDataKey(S0EYCkaK4DQkaj3HKG2cPOCJc5s7h9UIhHyU-AKY) -> 698, SessionAccessMetaDataKey(NmABev_p_Nl_OqPdTgvEeByHBQYfdwkNwnWspdbj) -> 961, SessionAccessMetaDataKey(X-O9DdPVYJ7hf47WsFyeOoeTBoo3uNSdvrQ6B7QT) -> 791, SessionAttributesKey(xDS8kA01UfKuII63qlvhBR50pBvX4H6PXXrnbLIj) -> 410, SessionAttributesKey(yw-E90OApAdwffnaiw0RnNt4J_kAo8jTcUcqOmyk) -> 841, SessionAccessMetaDataKey(m-nDjdHZ4MdlWlx2-UUbmjPitr1RL0OP298Hf0HA) -> 707, SessionAttributesKey(BNXfDgJvremICRftc0ha9S4HpbrXMR9tNMOY1ZT9) -> 663, SessionAttributesKey(13a68xqeUrVklubkccSbXPxFsgKrst0dUp5APiGn) -> 767, SessionAccessMetaDataKey(QLpFEzg9e2oPeDRqwwGkpe5DxXZI8e1-CINjloMJ) -> 767, SessionAccessMetaDataKey(R9CjvbklzwsNCPvfhSfcTcvfBqdT_qljwnAWjFeC) -> 872, SessionAttributesKey(rub3djV0D3sKf4XjulNCgaTVgN_Yx1BWqqkcIR1b) -> 590, SessionAttributesKey(3OgZdvXCGc_wqd1peBhPRlRP_haY8jVv-rD9XXVx) -> 774, SessionAttributesKey(nh1S6AgCvnEhhlJLRSaiSjGoFUmmjea8rYzfAJA2) -> 598, SessionAttributesKey(lrE1U715i-RtWNB4FnzrsH4brtY6I5b8KJdepOGP) -> 641, SessionAttributesKey(rRdoeVEZINGkaZnMOt3PF2hnTcpJ4XQ3hrYA7ew-) -> 770, SessionAttributesKey(paWPnr0ylHljFR_AB2FYd8sANCDwjBTAhj9pYNfh) -> 292, SessionAccessMetaDataKey(lqer8YZOAvLFPzqbqdXS-FrnDa1-RTGneNa_Fi8H) -> 757, SessionAttributesKey(5gN8SEXMZo4eMr0D8PWY0xCJqjNde-jwJ3ZijCfF) -> 471, SessionAttributesKey(3IS24GzTFwq1mi8DMRm2Dudv74rtjMb7r70OEmJn) -> 533, SessionAttributesKey(yoYNUa36YxKSXKSWt2ruayrtaNr3ZWpRhkJFfNVX) -> 596, SessionAttributesKey(iMq--Igc0To2K0FeN-dogYsnIHFKO0iulQZI9yHZ) -> 554, SessionAccessMetaDataKey(CYx3y2b52Y3-gn9XlddOqDlMA4Wjd54_vXYTmgEZ) -> 864, SessionAttributesKey(dKkMaRxlz-nqbeQnVtFQFu3qT9Mbab_82c1dO8Rt) -> 1152, SessionAttributesKey(s4aJHMBU9gY48AeKJ0YmBcbemewUbsjWCdDapybj) -> 871, SessionAttributesKey(_jReiLmA7v4WPXo9JGZmew5JqjShSkECMtqer3_3) -> 954, SessionAccessMetaDataKey(oyP7_KcD92RRnOT9FEPzHGiwewLxw_FToigpTkRf) -> 771, SessionAttributesKey(LtxPD8dolg2mw59Xqzcgnr7iasBGvLt7zkkd5vGz) -> 398, SessionAccessMetaDataKey(wavUkmhAykrERdJkkt4rGq6bT414xujYV9gGMdF6) -> 872, SessionAttributesKey(oyP7_KcD92RRnOT9FEPzHGiwewLxw_FToigpTkRf) -> 772, SessionAttributesKey(9YDOvI8Eyh3ElGhZttWNLeEbQjj4J79bnJ6VQ55s) -> 566, SessionAccessMetaDataKey(vI93BCX5juIxaH_Wg2-2Rr-PlVeIiG3tCgyNfS5x) -> 759, SessionAccessMetaDataKey(NVoBD3FmG67MKjj1P0C5oT6CTVrOQZhjn78SGM23) -> 555, SessionAttributesKey(ul0qfb8FZFJSbq0F0wQIbk7nOCXCs98gdRJIzWzz) -> 766, SessionAccessMetaDataKey(1ZJep2PuhhipUDo9xrfrTq0SP5ICWUiZBqUGz1m8) -> 411, SessionAttributesKey(-heicHqrj9X1E0kTeu7_pFTeKD4_Om4cZ9mtHvdL) -> 1069, SessionAccessMetaDataKey(rr1-XLKUhCblLIY-Yg6Vx-Zpx87WwUV-UlzaM4cY) -> 1066, SessionAttributesKey(lqer8YZOAvLFPzqbqdXS-FrnDa1-RTGneNa_Fi8H) -> 758, SessionAttributesKey(W9Xq2puhVc3eQsfznWK-sDszLtqHPO3nbf8Q3-V-) -> 602, SessionAccessMetaDataKey(Ym3QTBJf4kFE6k-WpYlAbyNAY9E194prRo3cG7CG) -> 755, SessionAttributesKey(53Gop1u28K9YuX6xXL4ve-wXjKbSGPMeyzgiLl0r) -> 1043, SessionAttributesKey(TdTKLPwOMQrjzwFHTJXiXSdnwORad5dUe0xlVf_6) -> 867, SessionAttributesKey(py3K7yTaJeOwmSQzMvBfmWBh2baD44usZLH0sWRL) -> 449, SessionAttributesKey(gVhsrXenh1n_nbqEAPeIV2JEWhh6fyxIM0jmTnbG) -> 768, SessionAttributesKey(JU9fP3s5fnzq843-PNktBOX4HaDnVnvwofHUoO8j) -> 362, SessionAttributesKey(anlwS91d09ZZ08jTHik7wXzUXDVCziUEDuQvDBVP) -> 476, SessionAttributesKey(NE1neM0JNXJMYgaIjOm9VfKBTNqyJK5LokrzqSAs) -> 754, SessionAccessMetaDataKey(DO8vtvR1hih3HausToK-WxN8U-9NFSvsnhzUV0KE) -> 599, SessionAccessMetaDataKey(2Pd0aWJkPpAEAqtlA5rlGhHTmyLIBop6zBVsVGMY) -> 642, SessionAccessMetaDataKey(-ikLVTM4xryzo38j7buKYDBBs-TPFwmcX-TWgGys) -> 235, SessionAccessMetaDataKey(kO--n_SiZmZjjp5PSlsSk1P_LlMx8iHKmaJaW85N) -> 648, SessionAttributesKey(fUcI5JWA4NsQvFAkHuW1EDMHxmm68jfwnXJE7k-2) -> 667, SessionAttributesKey(LKAxz-k1UOwVrR3Kzse5wsJnlt7I6U5tOZv3RQnr) -> 956, SessionAttributesKey(SAoshjrjw7yefcHOb4GGNzkH8jIaKa5I22j5DpNc) -> 463, SessionAccessMetaDataKey(SqgGA6gHg7XXEoidKvbWVcpn8jGTkdhk613iyfGQ) -> 963, SessionAttributesKey(Mez0WZgjNekcx99DdMYhJOZ1RJxsoItH8OPCuQ9j) -> 920}: java.lang.NullPointerException
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.persistence.manager.OrderedUpdatesManagerImpl.checkLockAndRemove(OrderedUpdatesManagerImpl.java:145)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.persistence.manager.OrderedUpdatesManagerImpl.invalidate(OrderedUpdatesManagerImpl.java:103)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.persistence.manager.OrderedUpdatesManagerImpl.invalidate(OrderedUpdatesManagerImpl.java:22)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.commands.write.InvalidateVersionsCommand.invokeAsync(InvalidateVersionsCommand.java:109)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:108)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:111)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:79)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:48)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.inboundhandler.TrianglePerCacheInboundInvocationHandler.handleDefaultCommand(TrianglePerCacheInboundInvocationHandler.java:148)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.inboundhandler.TrianglePerCacheInboundInvocationHandler.handle(TrianglePerCacheInboundInvocationHandler.java:101)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:165)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:112)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1554)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1477)
at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1685)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.JChannel.up(JChannel.java:733)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:131)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.stack.Protocol.up(Protocol.java:314)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FORK.up(FORK.java:145)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FRAG2.up(FRAG2.java:139)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FlowControl.up(FlowControl.java:253)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FlowControl.up(FlowControl.java:261)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.up(GMS.java:845)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:226)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1083)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:822)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:804)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:453)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:679)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:105)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:180)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:188)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.MERGE3.up(MERGE3.java:274)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.Discovery.up(Discovery.java:294)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.stack.Protocol.up(Protocol.java:314)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.TP.passMessageUp(TP.java:1183)
at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:107)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.wildfly.clustering.context@8.0.0.GA-redhat-00009//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
at org.wildfly.clustering.context@8.0.0.GA-redhat-00009//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
at java.base/java.lang.Thread.run(Thread.java:834)
at the same time, we observe the following errors on two other nodes:
node3:
2023-09-27 16:22:50,749 INFO [org.infinispan.scattered.impl.ScatteredVersionManagerImpl] (thread-25,ejb,wildfly3) ISPN000137: Failed invalidating remote cache: java.util.concurrent.CompletionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from wildfly1, see cause for remote stack trace at org.infinispan.commons@14.0.11.Final-redhat-00001//org.infinispan.commons.util.concurrent.CompletableFutures.asCompletionException(CompletableFutures.java:160) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.impl.MapResponseCollector.finish(MapResponseCollector.java:86) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.impl.MapResponseCollector$IgnoreLeavers.finish(MapResponseCollector.java:103) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:100) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1578) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1480) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1685) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.JChannel.up(JChannel.java:733) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:131) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.stack.Protocol.up(Protocol.java:314) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FORK.up(FORK.java:145) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FRAG2.up(FRAG2.java:139) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FlowControl.up(FlowControl.java:245) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FlowControl.up(FlowControl.java:245) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.up(GMS.java:845) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:226) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1083) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:822) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:804) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:453) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:679) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:105) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:180) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:188) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.MERGE3.up(MERGE3.java:274) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.Discovery.up(Discovery.java:294) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.stack.Protocol.up(Protocol.java:314) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.TP.passMessageUp(TP.java:1183) at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:107) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.wildfly.clustering.context@8.0.0.GA-redhat-00009//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49) at org.wildfly.clustering.context@8.0.0.GA-redhat-00009//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from wildfly1, see cause for remote stack trace at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:25) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.impl.MapResponseCollector.addException(MapResponseCollector.java:65) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.impl.MapResponseCollector$IgnoreLeavers.addException(MapResponseCollector.java:103) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:93) ... 31 more Caused by: java.lang.NullPointerException at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.exts.ThrowableExternalizer.newThrowableInstance(ThrowableExternalizer.java:285) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.exts.ThrowableExternalizer.readGenericThrowable(ThrowableExternalizer.java:265) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:240) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:44) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220) at org.infinispan@14.0.11.Final-redhat-00001//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1570) ... 29 more
node4:
2023-09-27 16:20:52,848 ERROR [io.undertow.servlet.request] (default task-12) UT015005: Error invoking method requestInitialized on listener class org.jboss.weld.module.web.servlet.WeldInitialListener: java.lang.NullPointerException at org.wildfly.clustering.server.infinispan@8.0.0.GA-redhat-00009//org.wildfly.clustering.server.infinispan.group.CacheGroup.createNode(CacheGroup.java:130) at org.wildfly.clustering.server.infinispan@8.0.0.GA-redhat-00009//org.wildfly.clustering.server.infinispan.group.CacheGroup.createNode(CacheGroup.java:64) at org.wildfly.clustering.ee.infinispan@8.0.0.GA-redhat-00009//org.wildfly.clustering.ee.infinispan.PrimaryOwnerLocator.apply(PrimaryOwnerLocator.java:57) at org.wildfly.clustering.ee.infinispan@8.0.0.GA-redhat-00009//org.wildfly.clustering.ee.infinispan.PrimaryOwnerLocator.apply(PrimaryOwnerLocator.java:38) at org.wildfly.clustering.web.infinispan@8.0.0.GA-redhat-00009//org.wildfly.clustering.web.infinispan.routing.PrimaryOwnerRouteLocator.locate(PrimaryOwnerRouteLocator.java:55) at org.wildfly.clustering.web.undertow@8.0.0.GA-redhat-00009//org.wildfly.clustering.web.undertow.routing.DistributableSessionIdentifierCodec.encode(DistributableSessionIdentifierCodec.java:45) at org.wildfly.extension.undertow@8.0.0.GA-redhat-00009//org.wildfly.extension.undertow.session.CodecSessionConfig.setSessionId(CodecSessionConfig.java:46) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.spec.ServletContextImpl$ServletContextSessionConfig.setSessionId(ServletContextImpl.java:1257) at org.wildfly.clustering.web.undertow@8.0.0.GA-redhat-00009//org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:235) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:884) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:454) at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:260) at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:146) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:263) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow@8.0.0.GA-redhat-00009//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430) at org.wildfly.extension.undertow@8.0.0.GA-redhat-00009//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430) at org.wildfly.extension.undertow@8.0.0.GA-redhat-00009//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430) at org.wildfly.extension.undertow@8.0.0.GA-redhat-00009//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256) at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101) at io.undertow.core@2.3.7.SP1-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:393) at io.undertow.core@2.3.7.SP1-redhat-00001//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859) at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990) at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363) at org.jboss.xnio@3.8.9.Final-redhat-00001//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282) at java.base/java.lang.Thread.run(Thread.java:834)
Please note that overall fail rate is still very very low;
- depends on
-
JBEAP-25867 Drop support for scattered cache
-
- Closed
-
- relates to
-
-
- Open
-