/qa/tools/opt/amd64/jdk1.8.0_last/bin/java -Xms1g -Xmx1g -Dispn.home=/home/mcimbora/temp/server/infinispan-server-8.2.1-SNAPSHOT -Dspark.home=/home/hudson/users-tmp/rmacor/spark-1.5.1-bin-hadoop2.6 -Dspark.master.host=172.18.1.2 -Dispn.server.list=172.18.1.18:11222 -DhotrodServer=172.18.1.18 -Dserver.config=/home/hudson/hudson_workspace/workspace/mcimbora-spark-wf-perf/etc/configs/distro/ISPN8/server.xml -Dlog4j.configuration=file:/home/hudson/hudson_workspace/workspace/mcimbora-spark-wf-perf/etc/log4j/log4j-info.xml -Dram_percentage=3.6 -Dnum_executions=10 -DnumPartitions=32 -Dprint.results=false -Duse_intern=false -Dsegments.count=512 -Djboss.default.multicast.address=234.99.54.15 -Djgroups.udp.bind_addr=172.18.1.32 -Djava.net.preferIPv4Stack=true -Dlog4j.file.prefix=edg-perf16.mw.lab.eng.bos.redhat.com-13621 -classpath /qa/home/mcimbora/radargun/RadarGun/lib/HdrHistogram-2.1.0.jar:/qa/home/mcimbora/radargun/RadarGun/lib/hdrhistogram-2.2.0-SNAPSHOT.jar:/qa/home/mcimbora/radargun/RadarGun/lib/jboss-transaction-api-1.0.1.GA.jar:/qa/home/mcimbora/radargun/RadarGun/lib/log4j-1.2.16.jar:/qa/home/mcimbora/radargun/RadarGun/lib/radargun-core-2.2.0-SNAPSHOT.jar:/qa/home/mcimbora/radargun/RadarGun/lib/radargun-core-2.2.0-SNAPSHOT-tests.jar:/qa/home/mcimbora/radargun/RadarGun/conf:/qa/tools/opt/amd64/jdk1.8.0_last/lib/tools.jar org.radargun.Slave --master 172.18.1.15 --slaveIndex 24 -------------------------------------------------------------------------------- 12:13:07,487 INFO [org.radargun.RemoteMasterConnection] (main) Attempting to connect to master 172.18.1.15:2103 12:13:09,497 INFO [org.radargun.RemoteMasterConnection] (main) Successfully established connection with master at: 172.18.1.15:2103 12:13:14,207 INFO [org.radargun.Slave] (main) Received slave index 24 12:13:14,207 INFO [org.radargun.Slave] (main) Received slave count 26 12:13:15,427 INFO [org.radargun.Slave] (sc-main) Service is Infinispan60ServerService {args=-Dsun.nio.ch.bugLevel= -Djboss.bind.address=172.18.1.32 -Djboss.bind.address.management=172.18.1.32 -Djboss.default.multicast.address=234.99.54.15 -Djava.net.preferIPv4Stack=true -Dsegments.count=512 -Dispn.log.level=INFO , cacheManagerName=clustered, command=null, env=JAVA_OPTS=-server -Xms24g -Xmx24g -XX:+UseLargePages -XX:MaxPermSize=512m -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps , executorPoolSize=2, file=/home/hudson/hudson_workspace/workspace/mcimbora-spark-wf-perf/etc/configs/distro/ISPN8/server.xml, home=/home/mcimbora/temp/server/infinispan-server-8.2.1-SNAPSHOT, java=null, javaOpts=null, jgroupsDumpPeriod=10.000 secs, jgroupsDumperEnabled=false, jmxConnectionEnabled=true, jmxDomain=jboss.datagrid-infinispan, logDir=null, os=unix, startTimeout=1 mins 0 secs, stderrToStdout=false, stopTimeout=1 mins 0 secs, viewCheckPeriod=10.000 secs, windows=false } 12:13:16,303 INFO [org.radargun.Slave] (sc-main) Starting stage ScenarioInit 12:13:16,380 INFO [org.radargun.Slave] (sc-main) Finished stage ScenarioInit 12:13:16,393 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 12:13:17,255 INFO [org.radargun.Slave] (sc-main) Starting stage Define 12:13:17,255 INFO [org.radargun.Slave] (sc-main) Finished stage Define 12:13:17,256 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 12:13:17,294 INFO [org.radargun.Slave] (sc-main) Starting stage RepeatBegin 12:13:17,295 INFO [org.radargun.Slave] (sc-main) Finished stage RepeatBegin 12:13:17,296 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 12:13:17,339 INFO [org.radargun.Slave] (sc-main) Stage should not be executed 12:13:17,340 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 12:13:19,604 INFO [org.radargun.Slave] (sc-main) Stage should not be executed 12:13:19,605 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 12:13:36,462 INFO [org.radargun.Slave] (sc-main) Stage should not be executed 12:13:36,464 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 12:13:44,001 INFO [org.radargun.Slave] (sc-main) Starting stage ServiceStart 12:13:45,022 INFO [org.radargun.stages.lifecycle.ServiceStartStage] (sc-main) Startup staggering, this is the slave with index 7. Sleeping for 8500 millis. 12:13:53,523 INFO [org.radargun.stages.lifecycle.ServiceStartStage] (sc-main) Ack master's StartCluster stage. Local address is: /172.18.1.31. This slave's index is: 24 12:13:54,549 INFO [org.radargun.service.InfinispanServerLifecycle] (sc-main) Environment: {JAVA_OPTS=-server -Xms24g -Xmx24g -XX:+UseLargePages -XX:MaxPermSize=512m -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps, JBOSS_HOME=/home/mcimbora/temp/server/infinispan-server-8.2.1-SNAPSHOT} 12:13:54,550 INFO [org.radargun.service.InfinispanServerLifecycle] (sc-main) Starting with: [/qa/home/mcimbora/radargun/RadarGun/plugins/process/bin/unix-start.sh, stcrbzfqxrsoloti, /home/mcimbora/temp/server/infinispan-server-8.2.1-SNAPSHOT/bin/clustered.sh, -Djboss.node.name=slave24, -Dsun.nio.ch.bugLevel=, -Djboss.bind.address=172.18.1.32, -Djboss.bind.address.management=172.18.1.32, -Djboss.default.multicast.address=234.99.54.15, -Djava.net.preferIPv4Stack=true, -Dsegments.count=512, -Dispn.log.level=INFO, -server-config, radargun-24.xml] JAVA_OPTS already set in environment; overriding default settings with values: -server -Xms24g -Xmx24g -XX:+UseLargePages -XX:MaxPermSize=512m -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps ========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /home/mcimbora/temp/server/infinispan-server-8.2.1-SNAPSHOT JAVA: /qa/tools/opt/amd64/jdk1.8.0_last/bin/java JAVA_OPTS: -server -Xms24g -Xmx24g -XX:+UseLargePages -XX:MaxPermSize=512m -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps ========================================================================= Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x00000001c0000000 bytes: 25769803776 (errno = 12). 12:13:55,967 INFO [org.jboss.modules] (main) JBoss Modules version 1.5.1.Final 12:13:56,591 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final 12:13:56,824 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: Infinispan Server 8.2.1-SNAPSHOT (WildFly Core 2.0.10.Final) starting 4.317: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(18874368K)] 2818602K(24536704K), 0.0909109 secs] [Times: user=0.24 sys=0.00, real=0.09 secs] 4.408: [CMS-concurrent-mark-start] 4.433: [CMS-concurrent-mark: 0.025/0.025 secs] [Times: user=0.09 sys=0.01, real=0.03 secs] 4.433: [CMS-concurrent-preclean-start] 4.585: [CMS-concurrent-preclean: 0.152/0.152 secs] [Times: user=0.21 sys=0.00, real=0.15 secs] 4.586: [CMS-concurrent-abortable-preclean-start] 12:13:59,879 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 14) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 12:14:00,869 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) 12:14:01,187 INFO [org.xnio] (MSC service thread 1-7) XNIO version 3.3.4.Final 12:14:01,209 INFO [org.xnio.nio] (MSC service thread 1-7) XNIO NIO Implementation Version 3.3.4.Final 12:14:01,391 INFO [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 4.0.18.Final 12:14:01,592 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem. 12:14:01,612 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique. 12:14:01,603 INFO [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem. 12:14:01,683 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors 12:14:01,689 INFO [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem 12:14:01,800 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem 12:14:01,871 INFO [org.jboss.as.security] (MSC service thread 1-5) WFLYSEC0001: Current PicketBox version=4.9.4.Final 7.166: [GC (Allocation Failure) 7.166: [ParNew: 5033216K->31421K(5662336K), 0.1836956 secs] 5033216K->31421K(24536704K), 0.1839743 secs] [Times: user=1.07 sys=0.08, real=0.18 secs] 12:14:02,219 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3) 12:14:02,273 INFO [org.jboss.as.connector] (MSC service thread 1-2) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.2.Final) 12:14:02,302 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-5) WFLYJCA0018: Started Driver service with driver-name = h2 12:14:02,309 INFO [org.jboss.as.naming] (MSC service thread 1-3) WFLYNAM0003: Starting Naming Service 12:14:03,009 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS] 12:14:03,273 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-7) WFLYDS0013: Started FileSystemDeploymentService for directory /home/mcimbora/temp/server/infinispan-server-8.2.1-SNAPSHOT/standalone/deployments 9.185: [CMS-concurrent-abortable-preclean: 4.132/4.599 secs] [Times: user=15.26 sys=0.54, real=4.60 secs] 9.186: [GC (CMS Final Remark) [YG occupancy: 2566296 K (5662336 K)]9.186: [Rescan (parallel) , 0.0411728 secs]9.227: [weak refs processing, 0.0000588 secs]9.227: [class unloading, 0.0166712 secs]9.244: [scrub symbol table, 0.0244433 secs]9.268: [scrub string table, 0.0023493 secs][1 CMS-remark: 0K(18874368K)] 2566296K(24536704K), 0.0886323 secs] [Times: user=0.36 sys=0.01, real=0.09 secs] 9.275: [CMS-concurrent-sweep-start] 9.275: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 9.275: [CMS-concurrent-reset-start] 9.428: [CMS-concurrent-reset: 0.153/0.153 secs] [Times: user=0.06 sys=0.09, real=0.15 secs] 12:14:20,544 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel clustered 12:14:20,570 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel clustered: [slave3|7] (8) [slave3, slave6, slave9, slave12, slave15, slave18, slave21, slave24] 12:14:20,580 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel clustered local address is slave24, physical addresses are [172.18.1.32:55200] 12:14:20,674 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-4) ISPN000128: Infinispan version: Infinispan 'Chakra' 8.2.1-SNAPSHOT 12:14:20,745 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10000: REST starting 12:14:21,413 INFO [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-5) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread 12:14:21,413 INFO [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-4) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread 12:14:21,415 INFO [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-2) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread 12:14:21,415 INFO [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-1) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread 12:14:21,651 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started transactionalCache cache from clustered container 12:14:21,653 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) DGISPN0001: Started default cache from clustered container 12:14:21,653 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) DGISPN0001: Started memcachedCache cache from clustered container 12:14:21,658 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: MemcachedServer starting 12:14:21,658 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: HotRodServer starting 12:14:21,658 INFO [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10000: WebSocketServer starting 12:14:21,660 INFO [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10001: WebSocketServer listening on 172.18.1.32:8181 12:14:21,660 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: MemcachedServer listening on 172.18.1.32:11211 12:14:21,660 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: HotRodServer listening on 172.18.1.32:11222 12:14:21,736 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) DGISPN0001: Started ___protobuf_metadata cache from clustered container 12:14:21,790 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) DGISPN0001: Started ___script_cache cache from clustered container 12:14:21,976 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0001: Started namedCache cache from clustered container 12:14:22,534 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started ___hotRodTopologyCache cache from clustered container 12:14:22,621 INFO [org.infinispan.rest.NettyRestServer] (MSC service thread 1-3) ISPN012003: REST server starting, listening on 172.18.1.32:8080 12:14:22,622 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10002: REST mapped to /rest 12:14:22,995 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://172.18.1.32:9990/management 12:14:22,997 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://172.18.1.32:9990 12:14:22,998 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 8.2.1-SNAPSHOT (WildFly Core 2.0.10.Final) started in 27781ms - Started 188 of 257 services (140 services are lazy, passive or on-demand) 12:14:23,878 INFO [org.radargun.stages.lifecycle.LifecycleHelper] (sc-main) Number of members=null is not the one expected: 8 12:14:25,999 INFO [org.radargun.stages.lifecycle.LifecycleHelper] (sc-main) Number of members is the one expected: 8 12:14:26,000 INFO [org.radargun.stages.lifecycle.ServiceStartStage] (sc-main) Successfully started cache service infinispan80/server on slave 24 12:14:26,036 INFO [org.radargun.Slave] (sc-main) Finished stage ServiceStart 12:14:26,044 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 12:14:26,068 INFO [org.radargun.Slave] (sc-main) Stage should not be executed 12:14:26,072 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 12:14:34,993 INFO [org.radargun.Slave] (sc-main) Starting stage Define 12:14:34,993 INFO [org.radargun.Slave] (sc-main) Finished stage Define 12:14:34,994 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 12:14:35,057 INFO [org.radargun.Slave] (sc-main) Stage should not be executed 12:14:35,058 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 54.518: [GC (Allocation Failure) 54.518: [ParNew: 5064637K->210152K(5662336K), 0.2531247 secs] 5064637K->210152K(24536704K), 0.2535328 secs] [Times: user=1.42 sys=0.18, real=0.26 secs] 127.452: [GC (Allocation Failure) 127.452: [ParNew: 5243368K->629120K(5662336K), 0.5961963 secs] 5243368K->858689K(24536704K), 0.5965111 secs] [Times: user=2.36 sys=0.58, real=0.60 secs] 205.554: [GC (Allocation Failure) 205.555: [ParNew: 5662336K->629120K(5662336K), 1.1117082 secs] 5891905K->1582733K(24536704K), 1.1120480 secs] [Times: user=3.79 sys=1.02, real=1.11 secs] 291.530: [GC (Allocation Failure) 291.530: [ParNew: 5662336K->629120K(5662336K), 0.9790970 secs] 6615949K->2300632K(24536704K), 0.9794457 secs] [Times: user=3.24 sys=0.69, real=0.98 secs] 373.169: [GC (Allocation Failure) 373.169: [ParNew: 5662336K->629120K(5662336K), 1.1863745 secs] 7333848K->3027438K(24536704K), 1.1867176 secs] [Times: user=3.19 sys=0.56, real=1.19 secs] 456.753: [GC (Allocation Failure) 456.753: [ParNew: 5662336K->629120K(5662336K), 0.9582534 secs] 8060654K->3755057K(24536704K), 0.9585910 secs] [Times: user=3.34 sys=0.68, real=0.95 secs] 546.275: [GC (Allocation Failure) 546.275: [ParNew: 5662336K->629120K(5662336K), 0.9942665 secs] 8788273K->4483074K(24536704K), 0.9946123 secs] [Times: user=3.34 sys=0.77, real=1.00 secs] 634.700: [GC (Allocation Failure) 634.700: [ParNew: 5662336K->629120K(5662336K), 0.9869951 secs] 9516290K->5222903K(24536704K), 0.9873186 secs] [Times: user=3.47 sys=0.70, real=0.98 secs] 723.964: [GC (Allocation Failure) 723.964: [ParNew: 5662336K->629120K(5662336K), 1.2186235 secs] 10256119K->5954213K(24536704K), 1.2221705 secs] [Times: user=2.94 sys=0.57, real=1.23 secs] 811.272: [GC (Allocation Failure) 811.273: [ParNew: 5662336K->629120K(5662336K), 1.0153856 secs] 10987429K->6685634K(24536704K), 1.0157180 secs] [Times: user=3.20 sys=0.66, real=1.02 secs] 899.796: [GC (Allocation Failure) 899.796: [ParNew: 5662336K->629120K(5662336K), 1.0714429 secs] 11718850K->7424318K(24536704K), 1.0717847 secs] [Times: user=3.07 sys=0.63, real=1.07 secs] 985.763: [GC (Allocation Failure) 985.763: [ParNew: 5662336K->629120K(5662336K), 0.9832098 secs] 12457534K->8157747K(24536704K), 0.9835602 secs] [Times: user=3.17 sys=0.73, real=0.98 secs] 1069.294: [GC (Allocation Failure) 1069.294: [ParNew: 5662336K->629120K(5662336K), 1.0070215 secs] 13190963K->8896931K(24536704K), 1.0073645 secs] [Times: user=3.16 sys=0.72, real=1.01 secs] 1155.930: [GC (Allocation Failure) 1155.930: [ParNew: 5662336K->629120K(5662336K), 1.0157988 secs] 13930147K->9645870K(24536704K), 1.0161418 secs] [Times: user=3.57 sys=0.74, real=1.01 secs] 1243.426: [GC (Allocation Failure) 1243.426: [ParNew: 5662336K->629120K(5662336K), 0.9878139 secs] 14679086K->10379440K(24536704K), 0.9881594 secs] [Times: user=3.52 sys=0.73, real=0.99 secs] 12:35:50,325 INFO [org.radargun.Slave] (sc-main) Stage should not be executed 12:35:50,326 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 1344.386: [GC (CMS Initial Mark) [1 CMS-initial-mark: 9750320K(18874368K)] 13087893K(24536704K), 0.0498993 secs] [Times: user=0.36 sys=0.00, real=0.05 secs] 1344.436: [CMS-concurrent-mark-start] 1345.260: [CMS-concurrent-mark: 0.822/0.823 secs] [Times: user=5.82 sys=0.52, real=0.83 secs] 1345.260: [CMS-concurrent-preclean-start] 1345.424: [CMS-concurrent-preclean: 0.159/0.164 secs] [Times: user=1.01 sys=0.24, real=0.16 secs] 1345.424: [CMS-concurrent-abortable-preclean-start] CMS: abort preclean due to time 1350.898: [CMS-concurrent-abortable-preclean: 4.810/5.474 secs] [Times: user=29.63 sys=10.95, real=5.47 secs] 1350.974: [GC (CMS Final Remark) [YG occupancy: 3388793 K (5662336 K)]1350.974: [Rescan (parallel) , 0.0745010 secs]1351.049: [weak refs processing, 0.0000727 secs]1351.049: [class unloading, 0.0340942 secs]1351.083: [scrub symbol table, 0.0253940 secs]1351.109: [scrub string table, 0.0022804 secs][1 CMS-remark: 9750320K(18874368K)] 13139113K(24536704K), 0.1415279 secs] [Times: user=0.66 sys=0.00, real=0.14 secs] 1351.116: [CMS-concurrent-sweep-start] 1351.543: [CMS-concurrent-sweep: 0.426/0.426 secs] [Times: user=1.06 sys=0.61, real=0.43 secs] 1351.543: [CMS-concurrent-reset-start] 1351.633: [CMS-concurrent-reset: 0.090/0.090 secs] [Times: user=0.09 sys=0.09, real=0.09 secs] 2039.204: [GC (Allocation Failure) 2039.204: [ParNew: 5662336K->629120K(5662336K), 2.8070624 secs] 15375647K->10987336K(24536704K), 2.8075103 secs] [Times: user=2.64 sys=2.20, real=2.80 secs] 2217.746: [GC (Allocation Failure) 2217.746: [ParNew: 5662336K->629120K(5662336K), 2.3462364 secs] 16020552K->11663486K(24536704K), 2.3468262 secs] [Times: user=6.02 sys=2.51, real=2.35 secs] 2240.158: [GC (Allocation Failure) 2240.158: [ParNew: 5662336K->629120K(5662336K), 0.6002145 secs] 16696702K->11835055K(24536704K), 0.6007773 secs] [Times: user=2.41 sys=0.45, real=0.60 secs] 2266.979: [GC (Allocation Failure) 2266.979: [ParNew: 5662336K->629120K(5662336K), 1.5591821 secs] 16868271K->12362999K(24536704K), 1.5596225 secs] [Times: user=2.52 sys=1.85, real=1.56 secs] 2296.171: [GC (Allocation Failure) 2296.171: [ParNew: 5662336K->383925K(5662336K), 0.7955232 secs] 17396215K->12400299K(24536704K), 0.7960954 secs] [Times: user=1.86 sys=0.94, real=0.79 secs] 2310.550: [GC (Allocation Failure) 2310.551: [ParNew: 5412594K->185197K(5662336K), 0.1583885 secs] 17428967K->12201570K(24536704K), 0.1589549 secs] [Times: user=1.20 sys=0.00, real=0.16 secs] 2324.846: [GC (Allocation Failure) 2324.846: [ParNew: 5218413K->214006K(5662336K), 0.1561199 secs] 17234786K->12230379K(24536704K), 0.1566761 secs] [Times: user=1.22 sys=0.01, real=0.15 secs] 2339.097: [GC (Allocation Failure) 2339.097: [ParNew: 5247222K->220245K(5662336K), 0.1539041 secs] 17263595K->12236618K(24536704K), 0.1544838 secs] [Times: user=1.21 sys=0.00, real=0.16 secs] 2353.476: [GC (Allocation Failure) 2353.477: [ParNew: 5250707K->274073K(5662336K), 0.2267871 secs] 17267080K->12290446K(24536704K), 0.2273498 secs] [Times: user=1.56 sys=0.01, real=0.23 secs] 2367.273: [GC (Allocation Failure) 2367.273: [ParNew: 5307289K->292682K(5662336K), 0.2060168 secs] 17323662K->12309055K(24536704K), 0.2066050 secs] [Times: user=1.43 sys=0.00, real=0.20 secs] 2381.658: [GC (Allocation Failure) 2381.658: [ParNew: 5325898K->245225K(5662336K), 0.2273045 secs] 17342271K->12286629K(24536704K), 0.2278782 secs] [Times: user=1.67 sys=0.04, real=0.23 secs] 2395.943: [GC (Allocation Failure) 2395.943: [ParNew: 5278441K->249249K(5662336K), 0.2146036 secs] 17319845K->12302195K(24536704K), 0.2150751 secs] [Times: user=1.63 sys=0.02, real=0.22 secs] 2409.990: [GC (Allocation Failure) 2409.991: [ParNew: 5277651K->279497K(5662336K), 0.2043486 secs] 17330597K->12344253K(24536704K), 0.2049436 secs] [Times: user=1.52 sys=0.02, real=0.20 secs] 2424.413: [GC (Allocation Failure) 2424.414: [ParNew: 5312713K->192789K(5662336K), 0.2128354 secs] 17377469K->12269137K(24536704K), 0.2133920 secs] [Times: user=1.51 sys=0.02, real=0.21 secs] 2438.790: [GC (Allocation Failure) 2438.790: [ParNew: 5226005K->212217K(5662336K), 0.2271850 secs] 17302353K->12301085K(24536704K), 0.2278386 secs] [Times: user=1.70 sys=0.02, real=0.23 secs] 2453.037: [GC (Allocation Failure) 2453.038: [ParNew: 5245433K->235756K(5662336K), 0.3195769 secs] 17334301K->12341519K(24536704K), 0.3201361 secs] [Times: user=1.71 sys=0.13, real=0.32 secs] 2467.366: [GC (Allocation Failure) 2467.367: [ParNew: 5268972K->236081K(5662336K), 0.2809685 secs] 17374735K->12358063K(24536704K), 0.2846354 secs] [Times: user=1.86 sys=0.04, real=0.29 secs] 2481.717: [GC (Allocation Failure) 2481.717: [ParNew: 5266930K->233440K(5662336K), 0.2590962 secs] 17388911K->12368860K(24536704K), 0.2596810 secs] [Times: user=1.96 sys=0.03, real=0.26 secs] 2496.237: [GC (Allocation Failure) 2496.237: [ParNew: 5266656K->320046K(5662336K), 0.3124429 secs] 17402076K->12468234K(24536704K), 0.3130345 secs] [Times: user=2.39 sys=0.01, real=0.31 secs] 2510.274: [GC (Allocation Failure) 2510.274: [ParNew: 5353262K->256181K(5662336K), 0.2598731 secs] 17501450K->12415754K(24536704K), 0.2603804 secs] [Times: user=1.98 sys=0.00, real=0.26 secs] 2524.731: [GC (Allocation Failure) 2524.731: [ParNew: 5289397K->276081K(5662336K), 0.2909141 secs] 17448970K->12447881K(24536704K), 0.2915500 secs] [Times: user=2.13 sys=0.01, real=0.29 secs] 2539.038: [GC (Allocation Failure) 2539.038: [ParNew: 5306724K->291029K(5662336K), 0.2985014 secs] 17478524K->12474588K(24536704K), 0.2989653 secs] [Times: user=2.27 sys=0.02, real=0.29 secs] 2553.435: [GC (Allocation Failure) 2553.435: [ParNew: 5323245K->265624K(5662336K), 0.2671491 secs] 17506805K->12461864K(24536704K), 0.2677189 secs] [Times: user=2.03 sys=0.01, real=0.27 secs] 2567.621: [GC (Allocation Failure) 2567.621: [ParNew: 5297641K->260760K(5662336K), 0.3576303 secs] 17493881K->12471091K(24536704K), 0.3582133 secs] [Times: user=2.02 sys=0.12, real=0.36 secs] 2582.050: [GC (Allocation Failure) 2582.051: [ParNew: 5293976K->290629K(5662336K), 0.2705256 secs] 17504307K->12523347K(24536704K), 0.2711645 secs] [Times: user=2.03 sys=0.04, real=0.27 secs] 2596.120: [GC (Allocation Failure) 2596.120: [ParNew: 5318627K->209808K(5662336K), 0.2346206 secs] 17551345K->12461296K(24536704K), 0.2352179 secs] [Times: user=1.73 sys=0.03, real=0.23 secs] 2610.616: [GC (Allocation Failure) 2610.616: [ParNew: 5243024K->183744K(5662336K), 0.2538925 secs] 17494512K->12451451K(24536704K), 0.2544597 secs] [Times: user=1.82 sys=0.01, real=0.26 secs] 2624.720: [GC (Allocation Failure) 2624.721: [ParNew: 5216960K->169027K(5662336K), 0.2286038 secs] 17484667K->12448907K(24536704K), 0.2291681 secs] [Times: user=1.74 sys=0.02, real=0.23 secs] 2639.256: [GC (Allocation Failure) 2639.256: [ParNew: 5202243K->238940K(5662336K), 0.2071556 secs] 17482123K->12530463K(24536704K), 0.2077279 secs] [Times: user=1.57 sys=0.02, real=0.21 secs] 2653.688: [GC (Allocation Failure) 2653.688: [ParNew: 5272156K->274342K(5662336K), 0.2757179 secs] 17563679K->12577256K(24536704K), 0.2762987 secs] [Times: user=2.10 sys=0.02, real=0.27 secs] 2668.053: [GC (Allocation Failure) 2668.054: [ParNew: 5307558K->269994K(5662336K), 0.2081257 secs] 17610472K->12584454K(24536704K), 0.2085476 secs] [Times: user=1.57 sys=0.02, real=0.21 secs] 2682.034: [GC (Allocation Failure) 2682.035: [ParNew: 5303210K->223241K(5662336K), 0.2504615 secs] 17617670K->12550230K(24536704K), 0.2510325 secs] [Times: user=1.91 sys=0.02, real=0.25 secs] 2696.233: [GC (Allocation Failure) 2696.233: [ParNew: 5256457K->206170K(5662336K), 0.2401046 secs] 17583446K->12544336K(24536704K), 0.2406828 secs] [Times: user=1.84 sys=0.01, real=0.24 secs] 2710.558: [GC (Allocation Failure) 2710.559: [ParNew: 5239386K->223907K(5662336K), 0.2249558 secs] 17577552K->12573840K(24536704K), 0.2255814 secs] [Times: user=1.65 sys=0.01, real=0.23 secs] 2725.035: [GC (Allocation Failure) 2725.035: [ParNew: 5251580K->250256K(5662336K), 0.3000335 secs] 17601512K->12611647K(24536704K), 0.3005921 secs] [Times: user=2.33 sys=0.01, real=0.30 secs] 2739.087: [GC (Allocation Failure) 2739.088: [ParNew: 5283472K->251499K(5662336K), 0.3881178 secs] 17644863K->12625270K(24536704K), 0.3886972 secs] [Times: user=2.31 sys=0.09, real=0.39 secs] 2753.920: [GC (Allocation Failure) 2753.920: [ParNew: 5284715K->330269K(5662336K), 0.3122374 secs] 17658486K->12715374K(24536704K), 0.3128181 secs] [Times: user=2.40 sys=0.02, real=0.32 secs] 2768.071: [GC (Allocation Failure) 2768.071: [ParNew: 5362955K->311035K(5662336K), 0.3291722 secs] 17748060K->12707930K(24536704K), 0.3296669 secs] [Times: user=2.45 sys=0.00, real=0.33 secs] 2782.383: [GC (Allocation Failure) 2782.383: [ParNew: 5344251K->285102K(5662336K), 0.3473762 secs] 17741146K->12694119K(24536704K), 0.3510415 secs] [Times: user=2.66 sys=0.02, real=0.35 secs] 2796.787: [GC (Allocation Failure) 2796.787: [ParNew: 5316680K->333768K(5662336K), 0.3057134 secs] 17725697K->12756185K(24536704K), 0.3062727 secs] [Times: user=2.36 sys=0.01, real=0.31 secs] 2811.092: [GC (Allocation Failure) 2811.093: [ParNew: 5363638K->339485K(5662336K), 0.4245521 secs] 17786056K->12776219K(24536704K), 0.4251113 secs] [Times: user=2.40 sys=0.10, real=0.42 secs] 2825.614: [GC (Allocation Failure) 2825.614: [ParNew: 5368934K->274062K(5662336K), 0.3251842 secs] 17805668K->12736212K(24536704K), 0.3258188 secs] [Times: user=2.32 sys=0.02, real=0.32 secs] 2840.263: [GC (Allocation Failure) 2840.264: [ParNew: 5307278K->315324K(5662336K), 0.3185047 secs] 17769428K->12805896K(24536704K), 0.3189883 secs] [Times: user=2.12 sys=0.02, real=0.32 secs] 2854.639: [GC (Allocation Failure) 2854.639: [ParNew: 5348536K->300250K(5662336K), 0.3003340 secs] 17839108K->12813928K(24536704K), 0.3009756 secs] [Times: user=2.27 sys=0.04, real=0.30 secs] 2868.763: [GC (Allocation Failure) 2868.764: [ParNew: 5333466K->281389K(5662336K), 0.2174934 secs] 17847144K->12811504K(24536704K), 0.2180617 secs] [Times: user=1.64 sys=0.02, real=0.22 secs] 2883.564: [GC (Allocation Failure) 2883.564: [ParNew: 5314605K->183418K(5662336K), 0.2221763 secs] 17844720K->12726035K(24536704K), 0.2228240 secs] [Times: user=1.68 sys=0.02, real=0.22 secs] 2897.888: [GC (Allocation Failure) 2897.889: [ParNew: 5211932K->211123K(5662336K), 0.2235930 secs] 17754549K->12765140K(24536704K), 0.2241331 secs] [Times: user=1.71 sys=0.02, real=0.22 secs] 2912.373: [GC (Allocation Failure) 2912.374: [ParNew: 5244339K->217057K(5662336K), 0.2320102 secs] 17798356K->12783211K(24536704K), 0.2326535 secs] [Times: user=1.75 sys=0.02, real=0.24 secs] 2926.653: [GC (Allocation Failure) 2926.653: [ParNew: 5244752K->188662K(5662336K), 0.2200619 secs] 17810905K->12765921K(24536704K), 0.2206434 secs] [Times: user=1.65 sys=0.02, real=0.22 secs] 2940.954: [GC (Allocation Failure) 2940.954: [ParNew: 5221878K->247315K(5662336K), 0.3572620 secs] 17799137K->12835878K(24536704K), 0.3579047 secs] [Times: user=2.03 sys=0.09, real=0.36 secs] 2955.446: [GC (Allocation Failure) 2955.447: [ParNew: 5280531K->239201K(5662336K), 0.2192311 secs] 17869094K->12839027K(24536704K), 0.2197975 secs] [Times: user=1.65 sys=0.01, real=0.22 secs] 2969.716: [GC (Allocation Failure) 2969.716: [ParNew: 5272417K->217129K(5662336K), 0.2187106 secs] 17872243K->12828278K(24536704K), 0.2192777 secs] [Times: user=1.66 sys=0.01, real=0.22 secs] 2984.009: [GC (Allocation Failure) 2984.010: [ParNew: 5250345K->262529K(5662336K), 0.2300621 secs] 17861494K->12884777K(24536704K), 0.2306042 secs] [Times: user=1.76 sys=0.01, real=0.23 secs] 2998.372: [GC (Allocation Failure) 2998.372: [ParNew: 5295745K->210528K(5662336K), 0.2284504 secs] 17917993K->12844095K(24536704K), 0.2289361 secs] [Times: user=1.77 sys=0.00, real=0.23 secs] 3012.456: [GC (Allocation Failure) 3012.457: [ParNew: 5242312K->242101K(5662336K), 0.2213412 secs] 17875880K->12886903K(24536704K), 0.2219142 secs] [Times: user=1.69 sys=0.01, real=0.22 secs] 13:04:11,012 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t15) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3026.746: [GC (Allocation Failure) 3026.746: [ParNew: 5272118K->273664K(5662336K), 0.3051981 secs] 17916919K->12929613K(24536704K), 0.3057667 secs] [Times: user=2.33 sys=0.03, real=0.30 secs] 13:04:28,269 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t20) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:04:28,346 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t19) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:04:28,360 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t3) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3040.981: [GC (Allocation Failure) 3040.981: [ParNew: 5306880K->292734K(5662336K), 0.3715612 secs] 17962829K->12959998K(24536704K), 0.3720488 secs] [Times: user=2.50 sys=0.00, real=0.37 secs] 3055.459: [GC (Allocation Failure) 3055.459: [ParNew: 5325950K->225959K(5662336K), 0.2358669 secs] 17993214K->12904506K(24536704K), 0.2364519 secs] [Times: user=1.80 sys=0.01, real=0.23 secs] 13:04:52,202 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t23) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:04:52,396 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t13) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:05:00,788 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t9) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3069.533: [GC (Allocation Failure) 3069.533: [ParNew: 5259175K->202471K(5662336K), 0.3551286 secs] 17937722K->12892216K(24536704K), 0.3556318 secs] [Times: user=1.93 sys=0.10, real=0.35 secs] 3084.045: [GC (Allocation Failure) 3084.046: [ParNew: 5235687K->175180K(5662336K), 0.2383337 secs] 17925432K->12876265K(24536704K), 0.2389207 secs] [Times: user=1.83 sys=0.01, real=0.24 secs] 3101.432: [GC (Allocation Failure) 3101.432: [ParNew: 5208396K->257066K(5662336K), 0.3810747 secs] 17909481K->12969309K(24536704K), 0.3816380 secs] [Times: user=2.59 sys=0.00, real=0.38 secs] 3116.088: [GC (Allocation Failure) 3116.088: [ParNew: 5290282K->228471K(5662336K), 0.3158944 secs] 18002525K->12951920K(24536704K), 0.3163565 secs] [Times: user=2.11 sys=0.02, real=0.31 secs] 13:05:58,056 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t16) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:06:02,264 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t11) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3130.505: [GC (Allocation Failure) 3130.505: [ParNew: 5261687K->250880K(5662336K), 0.2152874 secs] 17985136K->12985553K(24536704K), 0.2156491 secs] [Times: user=1.46 sys=0.00, real=0.22 secs] 3144.989: [GC (Allocation Failure) 3144.989: [ParNew: 5284096K->293523K(5662336K), 0.2611919 secs] 18018769K->13039451K(24536704K), 0.2617625 secs] [Times: user=1.83 sys=0.01, real=0.27 secs] 3159.051: [GC (Allocation Failure) 3159.051: [ParNew: 5326739K->230794K(5662336K), 0.2344971 secs] 18072667K->12988150K(24536704K), 0.2350794 secs] [Times: user=1.79 sys=0.01, real=0.24 secs] 13:06:41,038 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t16) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3173.452: [GC (Allocation Failure) 3173.453: [ParNew: 5264010K->286879K(5662336K), 0.3092317 secs] 18021366K->13055386K(24536704K), 0.3098080 secs] [Times: user=2.16 sys=0.01, real=0.31 secs] 13:06:58,349 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t2) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:06:58,484 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t6) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:06:58,562 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t7) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3187.622: [GC (Allocation Failure) 3187.622: [ParNew: 5320095K->223839K(5662336K), 0.2799959 secs] 18088602K->13003467K(24536704K), 0.2806433 secs] [Times: user=1.90 sys=0.01, real=0.28 secs] 3202.234: [GC (Allocation Failure) 3202.234: [ParNew: 5251487K->264236K(5662336K), 0.3006686 secs] 18031115K->13055277K(24536704K), 0.3012971 secs] [Times: user=2.03 sys=0.00, real=0.30 secs] 13:07:22,489 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t8) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:07:22,576 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t25) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3218.836: [GC (Allocation Failure) 3218.837: [ParNew: 5297452K->237622K(5662336K), 0.3139061 secs] 18088493K->13039896K(24536704K), 0.3145320 secs] [Times: user=2.12 sys=0.01, real=0.31 secs] 13:07:37,985 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t20) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3236.103: [GC (Allocation Failure) 3236.103: [ParNew: 5270838K->239163K(5662336K), 0.3078543 secs] 18073112K->13052491K(24536704K), 0.3084336 secs] [Times: user=2.07 sys=0.02, real=0.31 secs] 3253.816: [GC (Allocation Failure) 3253.816: [ParNew: 5272379K->233836K(5662336K), 0.3717023 secs] 18085707K->13058581K(24536704K), 0.3722842 secs] [Times: user=2.48 sys=0.05, real=0.38 secs] 3268.733: [GC (Allocation Failure) 3268.733: [ParNew: 5267052K->285139K(5662336K), 0.3787226 secs] 18091797K->13121060K(24536704K), 0.3792079 secs] [Times: user=2.94 sys=0.00, real=0.38 secs] 13:08:29,741 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t21) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:08:32,188 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t22) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3285.674: [GC (Allocation Failure) 3285.674: [ParNew: 5318355K->262153K(5662336K), 0.4029842 secs] 18154276K->13109561K(24536704K), 0.4035338 secs] [Times: user=3.13 sys=0.00, real=0.40 secs] 3301.841: [GC (Allocation Failure) 3301.841: [ParNew: 5295369K->317806K(5662336K), 0.3440296 secs] 18142777K->13177186K(24536704K), 0.3445745 secs] [Times: user=2.69 sys=0.00, real=0.35 secs] 3316.098: [GC (Allocation Failure) 3316.099: [ParNew: 5351022K->306660K(5662336K), 0.3298503 secs] 18210402K->13182274K(24536704K), 0.3303325 secs] [Times: user=2.57 sys=0.00, real=0.33 secs] 13:09:11,271 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t11) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3330.835: [GC (Allocation Failure) 3330.835: [ParNew: 5338981K->352608K(5662336K), 0.3454642 secs] 18214595K->13249442K(24536704K), 0.3461464 secs] [Times: user=2.65 sys=0.01, real=0.35 secs] 3344.960: [GC (Allocation Failure) 3344.961: [ParNew: 5385824K->355113K(5662336K), 0.3363832 secs] 18282658K->13279496K(24536704K), 0.3368849 secs] [Times: user=2.62 sys=0.01, real=0.34 secs] 3359.344: [GC (Allocation Failure) 3359.344: [ParNew: 5384701K->300476K(5662336K), 0.4124911 secs] 18309084K->13256918K(24536704K), 0.4130131 secs] [Times: user=2.52 sys=0.12, real=0.41 secs] 13:09:59,863 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t20) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave15 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3373.839: [GC (Allocation Failure) 3373.839: [ParNew: 5333692K->300414K(5662336K), 0.3692504 secs] 18290134K->13289997K(24536704K), 0.3697233 secs] [Times: user=2.49 sys=0.03, real=0.37 secs] 3389.531: [GC (Allocation Failure) 3389.531: [ParNew: 5333630K->292187K(5662336K), 0.3507781 secs] 18323213K->13309956K(24536704K), 0.3513463 secs] [Times: user=2.31 sys=0.02, real=0.35 secs] 3413.984: [GC (Allocation Failure) 3413.985: [ParNew: 5322142K->227074K(5662336K), 0.2713436 secs] 18339911K->13265887K(24536704K), 0.2719027 secs] [Times: user=2.04 sys=0.02, real=0.27 secs] 3447.456: [GC (Allocation Failure) 3447.457: [ParNew: 5257362K->225851K(5662336K), 0.2518938 secs] 18296176K->13279786K(24536704K), 0.2524566 secs] [Times: user=1.92 sys=0.02, real=0.25 secs] 3470.629: [GC (Allocation Failure) 3470.629: [ParNew: 5258712K->343752K(5662336K), 0.3461881 secs] 18312648K->13409509K(24536704K), 0.3466827 secs] [Times: user=2.37 sys=0.00, real=0.35 secs] 3490.871: [GC (Allocation Failure) 3490.871: [ParNew: 5376912K->627319K(5662336K), 0.6882618 secs] 18442668K->13876289K(24536704K), 0.6887296 secs] [Times: user=3.60 sys=0.82, real=0.69 secs] 3512.279: [GC (Allocation Failure) 3512.280: [ParNew: 5659215K->206447K(5662336K), 0.4154440 secs] 18908186K->13528514K(24536704K), 0.4160794 secs] [Times: user=2.16 sys=0.21, real=0.41 secs] 3553.103: [GC (Allocation Failure) 3553.103: [ParNew: 5239663K->299862K(5662336K), 0.1510875 secs] 18561730K->13621929K(24536704K), 0.1517227 secs] [Times: user=1.19 sys=0.00, real=0.15 secs] 3598.848: [GC (Allocation Failure) 3598.849: [ParNew: 5333078K->48425K(5662336K), 0.1251347 secs] 18655145K->13370492K(24536704K), 0.1257534 secs] [Times: user=0.86 sys=0.00, real=0.13 secs] 3695.652: [GC (Allocation Failure) 3695.652: [ParNew: 5074851K->628664K(5662336K), 0.3151331 secs] 18396918K->14053131K(24536704K), 0.3157810 secs] [Times: user=2.21 sys=0.11, real=0.32 secs] 13:16:05,484 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t13) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3738.858: [GC (Allocation Failure) 3738.859: [ParNew: 5661880K->116083K(5662336K), 0.1926649 secs] 19086347K->13575368K(24536704K), 0.1932342 secs] [Times: user=1.04 sys=0.06, real=0.20 secs] 13:16:43,171 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t19) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:16:43,184 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t20) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:16:43,184 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t11) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:16:44,148 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t14) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3925.583: [GC (Allocation Failure) 3925.584: [ParNew: 5147752K->402816K(5662336K), 0.1653487 secs] 18607037K->13862102K(24536704K), 0.1659214 secs] [Times: user=1.12 sys=0.00, real=0.17 secs] 13:19:51,287 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t6) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:19:51,339 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t23) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:19:56,922 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t10) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:19:57,029 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t24) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:19:57,048 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t16) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4140.374: [GC (Allocation Failure) 4140.374: [ParNew: 5433637K->626989K(5662336K), 0.3556715 secs] 18892922K->14135629K(24536704K), 0.3561471 secs] [Times: user=2.35 sys=0.35, real=0.35 secs] 4339.229: [GC (Allocation Failure) 4339.229: [ParNew: 5660190K->82917K(5662336K), 0.0843677 secs] 19168830K->13591802K(24536704K), 0.0849149 secs] [Times: user=0.65 sys=0.00, real=0.09 secs] 13:27:00,974 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t6) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:27:52,360 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t16) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:27:52,747 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t24) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:27:52,750 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t10) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4441.242: [GC (Allocation Failure) 4441.243: [ParNew: 5112368K->256816K(5662336K), 0.1308866 secs] 18621253K->13765701K(24536704K), 0.1315020 secs] [Times: user=0.89 sys=0.00, real=0.13 secs] 13:29:59,504 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t9) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:29:59,500 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t18) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4627.443: [GC (Allocation Failure) 4627.443: [ParNew: 5290032K->575003K(5662336K), 0.2012834 secs] 18798917K->14083888K(24536704K), 0.2019339 secs] [Times: user=1.38 sys=0.00, real=0.20 secs] 13:31:23,206 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t22) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:31:34,737 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t12) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:31:36,699 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t14) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:31:40,224 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t6) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:31:40,373 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t16) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:31:40,394 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t24) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:32:20,909 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t10) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:32:29,434 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t25) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4836.818: [GC (Allocation Failure) 4836.818: [ParNew: 5608219K->255042K(5662336K), 0.1310750 secs] 19117104K->13764463K(24536704K), 0.1316921 secs] [Times: user=0.99 sys=0.00, real=0.13 secs] 13:35:40,388 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t8) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4909.191: [GC (Allocation Failure) 4909.191: [ParNew: 5286094K->586948K(5662336K), 0.2002514 secs] 18795515K->14096369K(24536704K), 0.2007958 secs] [Times: user=1.59 sys=0.00, real=0.20 secs] 13:38:30,838 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t24) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:38:30,905 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t3) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:38:30,909 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t21) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5108.664: [GC (Allocation Failure) 5108.664: [ParNew: 5620164K->462170K(5662336K), 0.1743972 secs] 19129585K->13972115K(24536704K), 0.1750029 secs] [Times: user=1.37 sys=0.01, real=0.17 secs] 13:39:33,761 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t13) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:39:33,827 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t11) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:39:34,043 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t15) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:39:34,113 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t17) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:40:42,753 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t1) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5302.081: [GC (Allocation Failure) 5302.081: [ParNew: 5495386K->34700K(5662336K), 0.0684144 secs] 19005331K->13544793K(24536704K), 0.0690875 secs] [Times: user=0.52 sys=0.01, real=0.07 secs] 5342.095: [GC (Allocation Failure) 5342.096: [ParNew: 5067916K->622162K(5662336K), 0.2398535 secs] 18578009K->14156831K(24536704K), 0.2405225 secs] [Times: user=1.88 sys=0.02, real=0.24 secs] 13:43:24,404 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t10) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:43:30,889 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t16) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:43:31,384 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t25) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:43:35,923 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t11) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:44:01,393 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t17) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5568.989: [GC (Allocation Failure) 5568.990: [ParNew: 5652024K->276505K(5662336K), 0.1291327 secs] 19186693K->13811360K(24536704K), 0.1297906 secs] [Times: user=1.01 sys=0.00, real=0.14 secs] 13:49:37,082 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t19) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:49:41,913 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t18) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:50:01,986 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t3) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:50:02,027 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t10) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5773.604: [GC (Allocation Failure) 5773.605: [ParNew: 5308906K->444769K(5662336K), 0.1683066 secs] 18843761K->13979623K(24536704K), 0.1688588 secs] [Times: user=1.29 sys=0.00, real=0.17 secs] 13:50:39,060 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t16) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:50:39,115 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t24) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:50:39,127 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t17) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:50:39,127 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t11) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:50:44,332 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t25) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:50:44,422 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t22) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:51:19,334 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t12) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:51:20,801 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t4) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:51:35,333 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t9) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:51:37,868 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t5) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:51:55,875 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t23) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:52:50,325 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t14) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:53:51,329 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t7) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 6000.282: [GC (Allocation Failure) 6000.282: [ParNew: 5477364K->567438K(5662336K), 0.1776747 secs] 19012219K->14102915K(24536704K), 0.1782569 secs] [Times: user=1.37 sys=0.00, real=0.18 secs] 13:54:01,293 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t13) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:06,956 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t21) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:08,981 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t15) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:09,021 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t18) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:09,029 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t19) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:23,073 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t6) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:23,349 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t16) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:23,373 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t17) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:34,925 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t11) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:34,982 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t25) ISPN000071: Caught exception when handling command StreamRequestCommand{cacheName='default'}: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slave12 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$74(JGroupsTransport.java:633) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:54:51,691 INFO [org.radargun.Slave] (sc-main) Starting stage ScenarioDestroy 13:54:51,692 INFO [org.radargun.stages.ScenarioDestroyStage] (sc-main) Scenario finished, destroying... 13:54:51,697 INFO [org.radargun.stages.ScenarioDestroyStage] (sc-main) Memory before cleanup: Runtime free: 937,020 kb Runtime max:1,005,056 kb Runtime total:1,005,056 kb MX Code Cache(Non-heap memory): used: 6,288 kb, init: 2,496 kb, committed: 6,528 kb, max: 245,760 kb MX Metaspace(Non-heap memory): used: 17,662 kb, init: 0 kb, committed: 18,560 kb, max: 0 kb MX Compressed Class Space(Non-heap memory): used: 2,272 kb, init: 0 kb, committed: 2,432 kb, max: 1,048,576 kb MX PS Eden Space(Heap memory): used: 64,229 kb, init: 262,144 kb, committed: 262,144 kb, max: 262,144 kb MX PS Survivor Space(Heap memory): used: 0 kb, init: 43,520 kb, committed: 43,520 kb, max: 43,520 kb MX PS Old Gen(Heap memory): used: 3,806 kb, init: 699,392 kb, committed: 699,392 kb, max: 699,392 kb 13:54:53,904 INFO [org.radargun.stages.lifecycle.LifecycleHelper] (sc-main) Stopping service. /qa/home/mcimbora/radargun/RadarGun/plugins/process/bin/unix-start.sh: line 4: 20815 Terminated RADARGUN_TAG=$TAG $@ 13:54:56,213 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested.  13:55:23,905 WARN [org.radargun.stages.lifecycle.LifecycleHelper] (sc-main) Lifecycle did not finished withing timeout, killing instead. 13:55:27,355 INFO [org.radargun.stages.ScenarioDestroyStage] (sc-main) Service successfully stopped. 13:55:27,356 INFO [org.radargun.Slave] (sc-main) Finished stage ScenarioDestroy 13:55:27,363 INFO [org.radargun.RemoteMasterConnection] (sc-main) Response successfully sent to the master 13:55:27,590 INFO [org.radargun.Slave] (main) Starting stage ScenarioCleanup 13:55:27,592 WARN [org.radargun.stages.ScenarioCleanupStage] (main) Unfinished thread StopThread (id=25, state=WAITING) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396) at org.radargun.service.ProcessLifecycle.isRunning(ProcessLifecycle.java:194) at org.radargun.service.ProcessLifecycle.stopInternal(ProcessLifecycle.java:182) at org.radargun.service.InfinispanServerLifecycle.stop(InfinispanServerLifecycle.java:91) at org.radargun.stages.lifecycle.LifecycleHelper$1.run(LifecycleHelper.java:150) at java.lang.Thread.run(Thread.java:745) 13:55:27,593 INFO [org.radargun.stages.ScenarioCleanupStage] (main) Interrupting thread StopThread (id=25, state=WAITING) 13:55:27,595 ERROR [org.radargun.service.InfinispanServerLifecycle] (StopThread) Script interrupted java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396) at org.radargun.service.ProcessLifecycle.isRunning(ProcessLifecycle.java:194) at org.radargun.service.ProcessLifecycle.stopInternal(ProcessLifecycle.java:182) at org.radargun.service.InfinispanServerLifecycle.stop(InfinispanServerLifecycle.java:91) at org.radargun.stages.lifecycle.LifecycleHelper$1.run(LifecycleHelper.java:150) at java.lang.Thread.run(Thread.java:745) 13:55:29,680 INFO [org.radargun.stages.ScenarioCleanupStage] (main) Is thread StopThread (id=25, state=TERMINATED)) alive? false 13:55:29,813 INFO [org.radargun.stages.ScenarioCleanupStage] (main) Memory after cleanup: Runtime free: 996,805 kb Runtime max:1,005,056 kb Runtime total:1,005,056 kb MX Code Cache(Non-heap memory): used: 6,226 kb, init: 2,496 kb, committed: 6,528 kb, max: 245,760 kb MX Metaspace(Non-heap memory): used: 17,656 kb, init: 0 kb, committed: 18,560 kb, max: 0 kb MX Compressed Class Space(Non-heap memory): used: 2,270 kb, init: 0 kb, committed: 2,432 kb, max: 1,048,576 kb MX PS Eden Space(Heap memory): used: 11,796 kb, init: 262,144 kb, committed: 262,144 kb, max: 262,144 kb MX PS Survivor Space(Heap memory): used: 0 kb, init: 43,520 kb, committed: 43,520 kb, max: 43,520 kb MX PS Old Gen(Heap memory): used: 3,008 kb, init: 699,392 kb, committed: 699,392 kb, max: 699,392 kb 13:55:29,824 INFO [org.radargun.RemoteMasterConnection] (main) Response successfully sent to the master 13:55:33,138 INFO [org.radargun.RemoteMasterConnection] (main) Response successfully sent to the master 13:55:36,604 INFO [org.radargun.Slave] (main) Master shutdown! 13:55:36,607 INFO [org.radargun.ShutDownHook] (Thread-0) Slave process is being shutdown