Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-7487

Fix NPE in ClusterCacheStatus.java:147

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • 9.0.0.Final
    • 9.0.0.Final
    • Cloud, Core
    • None

      When I was performing Spring Session and Kubernetes Rolling Update demo I encountered a couple of problems.

      One of the is this:

      kubetail -l environment=infrastructure 
      Will tail 3 logs...
      transactions-repository-1-04x09
      transactions-repository-1-1f8dx
      transactions-repository-1-4z05w
      [transactions-repository-1-1f8dx] JAVA_OPTS already set in environment; overriding default settings with values: -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC
      [transactions-repository-1-1f8dx] =========================================================================
      [transactions-repository-1-1f8dx] 
      [transactions-repository-1-1f8dx] JBoss Bootstrap Environment
      [transactions-repository-1-1f8dx] 
      [transactions-repository-1-1f8dx] JBOSS_HOME: /opt/jboss/infinispan-server
      [transactions-repository-1-1f8dx] 
      [transactions-repository-1-1f8dx] JAVA: /usr/lib/jvm/java/bin/java
      [transactions-repository-1-1f8dx] 
      [transactions-repository-1-1f8dx] JAVA_OPTS:   -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC
      [transactions-repository-1-1f8dx] 
      [transactions-repository-1-1f8dx] =========================================================================
      [transactions-repository-1-1f8dx] 
      [transactions-repository-1-1f8dx] 17:54:15,634 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final
      [transactions-repository-1-1f8dx] 17:54:15,853 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
      [transactions-repository-1-1f8dx] 17:54:15,931 INFO  [org.jboss.as] (MSC service thread 1-8) WFLYSRV0049: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) starting
      [transactions-repository-1-1f8dx] 17:54:17,232 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 7) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
      [transactions-repository-1-1f8dx] [GC (CMS Initial Mark)  234913K(1013632K), 0.0473268 secs]
      [transactions-repository-1-1f8dx] 17:54:17,432 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
      [transactions-repository-1-1f8dx] 17:54:17,452 INFO  [org.xnio] (MSC service thread 1-2) XNIO version 3.4.0.Final
      [transactions-repository-1-1f8dx] 17:54:17,464 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.4.0.Final
      [transactions-repository-1-1f8dx] 17:54:17,516 INFO  [org.jboss.remoting] (MSC service thread 1-3) JBoss Remoting version 4.0.21.Final
      [transactions-repository-1-1f8dx] 17:54:17,553 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem.
      [transactions-repository-1-1f8dx] [GC (GCLocker Initiated GC)  279722K->29726K(1013632K), 0.0377206 secs]
      [transactions-repository-1-1f8dx] 17:54:17,611 INFO  [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem.
      [transactions-repository-1-1f8dx] 17:54:17,630 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
      [transactions-repository-1-1f8dx] 17:54:17,646 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem
      [transactions-repository-1-1f8dx] 17:54:17,646 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem
      [transactions-repository-1-1f8dx] 17:54:17,652 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors
      [transactions-repository-1-1f8dx] 17:54:17,653 INFO  [org.jboss.as.security] (MSC service thread 1-8) WFLYSEC0001: Current PicketBox version=4.9.6.Final
      [transactions-repository-1-1f8dx] 17:54:17,673 INFO  [org.jboss.as.connector] (MSC service thread 1-4) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)
      [transactions-repository-1-1f8dx] 17:54:17,701 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
      [transactions-repository-1-1f8dx] 17:54:17,703 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-6) WFLYJCA0018: Started Driver service with driver-name = h2
      [transactions-repository-1-1f8dx] 17:54:17,744 INFO  [org.jboss.as.naming] (MSC service thread 1-7) WFLYNAM0003: Starting Naming Service
      [transactions-repository-1-1f8dx] 17:54:18,268 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
      [transactions-repository-1-1f8dx] 17:54:18,425 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-1f8dx] 17:54:18,429 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-1f8dx] 17:54:18,779 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-5) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments
      [transactions-repository-1-04x09] JAVA_OPTS already set in environment; overriding default settings with values: -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC
      [transactions-repository-1-04x09] =========================================================================
      [transactions-repository-1-04x09] 
      [transactions-repository-1-04x09] JBoss Bootstrap Environment
      [transactions-repository-1-04x09] 
      [transactions-repository-1-04x09] JBOSS_HOME: /opt/jboss/infinispan-server
      [transactions-repository-1-04x09] 
      [transactions-repository-1-04x09] JAVA: /usr/lib/jvm/java/bin/java
      [transactions-repository-1-04x09] 
      [transactions-repository-1-04x09] JAVA_OPTS:   -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC
      [transactions-repository-1-04x09] 
      [transactions-repository-1-04x09] =========================================================================
      [transactions-repository-1-04x09] 
      [transactions-repository-1-04x09] 17:54:15,628 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final
      [transactions-repository-1-04x09] 17:54:15,849 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
      [transactions-repository-1-04x09] 17:54:15,960 INFO  [org.jboss.as] (MSC service thread 1-5) WFLYSRV0049: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) starting
      [transactions-repository-1-04x09] [GC (CMS Initial Mark)  218119K(1013632K), 0.0624692 secs]
      [transactions-repository-1-04x09] 17:54:17,475 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 13) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
      [transactions-repository-1-04x09] 17:54:17,647 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
      [transactions-repository-1-04x09] 17:54:17,664 INFO  [org.xnio] (MSC service thread 1-3) XNIO version 3.4.0.Final
      [transactions-repository-1-04x09] 17:54:17,683 INFO  [org.xnio.nio] (MSC service thread 1-3) XNIO NIO Implementation Version 3.4.0.Final
      [transactions-repository-1-04x09] 17:54:17,763 INFO  [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 4.0.21.Final
      [transactions-repository-1-04x09] [GC (Allocation Failure)  279616K->29652K(1013632K), 0.0690895 secs]
      [transactions-repository-1-04x09] 17:54:17,872 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem.
      [transactions-repository-1-04x09] 17:54:17,875 INFO  [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem.
      [transactions-repository-1-04x09] [GC (CMS Final Remark)  68259K(1013632K), 0.0485831 secs]
      [transactions-repository-1-04x09] 17:54:17,933 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors
      [transactions-repository-1-04x09] 17:54:17,960 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
      [transactions-repository-1-04x09] 17:54:17,972 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem
      [transactions-repository-1-04x09] 17:54:17,972 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem
      [transactions-repository-1-04x09] 17:54:17,984 INFO  [org.jboss.as.security] (MSC service thread 1-4) WFLYSEC0001: Current PicketBox version=4.9.6.Final
      [transactions-repository-1-04x09] 17:54:18,004 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
      [transactions-repository-1-04x09] 17:54:18,027 INFO  [org.jboss.as.naming] (MSC service thread 1-5) WFLYNAM0003: Starting Naming Service
      [transactions-repository-1-04x09] 17:54:18,053 INFO  [org.jboss.as.connector] (MSC service thread 1-8) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)
      [transactions-repository-1-04x09] 17:54:18,065 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-6) WFLYJCA0018: Started Driver service with driver-name = h2
      [transactions-repository-1-04x09] 17:54:18,543 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-04x09] 17:54:18,561 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-04x09] 17:54:18,584 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-6) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
      [transactions-repository-1-04x09] 17:54:18,795 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments
      [transactions-repository-1-4z05w] JAVA_OPTS already set in environment; overriding default settings with values: -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC
      [transactions-repository-1-4z05w] =========================================================================
      [transactions-repository-1-4z05w] 
      [transactions-repository-1-4z05w] JBoss Bootstrap Environment
      [transactions-repository-1-4z05w] 
      [transactions-repository-1-4z05w] JBOSS_HOME: /opt/jboss/infinispan-server
      [transactions-repository-1-4z05w] 
      [transactions-repository-1-4z05w] JAVA: /usr/lib/jvm/java/bin/java
      [transactions-repository-1-4z05w] 
      [transactions-repository-1-4z05w] JAVA_OPTS:   -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC
      [transactions-repository-1-4z05w] 
      [transactions-repository-1-4z05w] =========================================================================
      [transactions-repository-1-4z05w] 
      [transactions-repository-1-4z05w] 17:54:15,714 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final
      [transactions-repository-1-4z05w] 17:54:15,950 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
      [transactions-repository-1-4z05w] 17:54:16,016 INFO  [org.jboss.as] (MSC service thread 1-6) WFLYSRV0049: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) starting
      [transactions-repository-1-4z05w] 17:54:17,364 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 5) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
      [transactions-repository-1-4z05w] [GC (CMS Initial Mark)  234897K(1013632K), 0.0587177 secs]
      [transactions-repository-1-4z05w] 17:54:17,599 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
      [transactions-repository-1-4z05w] 17:54:17,620 INFO  [org.xnio] (MSC service thread 1-5) XNIO version 3.4.0.Final
      [transactions-repository-1-4z05w] 17:54:17,626 INFO  [org.xnio.nio] (MSC service thread 1-5) XNIO NIO Implementation Version 3.4.0.Final
      [transactions-repository-1-4z05w] 17:54:17,674 INFO  [org.jboss.remoting] (MSC service thread 1-5) JBoss Remoting version 4.0.21.Final
      [transactions-repository-1-4z05w] [GC (Allocation Failure)  279616K->29736K(1013632K), 0.0733498 secs]
      [transactions-repository-1-4z05w] 17:54:17,799 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem.
      [transactions-repository-1-4z05w] 17:54:17,802 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
      [transactions-repository-1-4z05w] 17:54:17,829 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem
      [transactions-repository-1-4z05w] 17:54:17,831 INFO  [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem.
      [transactions-repository-1-4z05w] 17:54:17,835 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors
      [transactions-repository-1-4z05w] 17:54:17,870 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem
      [transactions-repository-1-4z05w] 17:54:17,881 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
      [transactions-repository-1-4z05w] 17:54:17,887 INFO  [org.jboss.as.connector] (MSC service thread 1-2) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)
      [transactions-repository-1-4z05w] 17:54:17,895 INFO  [org.jboss.as.security] (MSC service thread 1-4) WFLYSEC0001: Current PicketBox version=4.9.6.Final
      [transactions-repository-1-4z05w] 17:54:17,900 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0018: Started Driver service with driver-name = h2
      [transactions-repository-1-4z05w] 17:54:18,037 INFO  [org.jboss.as.naming] (MSC service thread 1-6) WFLYNAM0003: Starting Naming Service
      [transactions-repository-1-4z05w] 17:54:18,842 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
      [transactions-repository-1-4z05w] 17:54:18,920 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-4z05w] 17:54:18,934 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-4z05w] 17:54:19,094 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-6) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments
      [transactions-repository-1-4z05w] 17:54:20,553 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-6) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-4z05w] 17:54:20,555 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-6) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-4z05w] [GC (CMS Final Remark)  171687K(1013632K), 0.0585468 secs]
      [transactions-repository-1-4z05w] 17:54:20,716 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel clustered
      [transactions-repository-1-4z05w] 17:54:20,732 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel clustered: [transactions-repository-1-4z05w|2] (3) [transactions-repository-1-4z05w, transactions-repository-1-04x09, transactions-repository-1-1f8dx]
      [transactions-repository-1-4z05w] 17:54:20,742 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079: Channel clustered local address is transactions-repository-1-4z05w, physical addresses are [172.17.0.3:7600]
      [transactions-repository-1-4z05w] 17:54:20,754 INFO  [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-6) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2017-02-18T17:54:20.748Z
      [transactions-repository-1-4z05w] 17:54:20,755 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-6) ISPN000128: Infinispan version: Infinispan 'Ruppaner' 9.0.0-SNAPSHOT
      [transactions-repository-1-4z05w] 17:54:20,933 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-6) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-4z05w] 17:54:20,933 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-6) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-4z05w] 17:54:20,936 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-5) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-4z05w] 17:54:20,939 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-4z05w] 17:54:20,943 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-4z05w] 17:54:20,954 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-4z05w] 17:54:20,950 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-5) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-4z05w] 17:54:20,956 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-5) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-1f8dx] [GC (CMS Final Remark)  172593K(1013632K), 0.0447491 secs]
      [transactions-repository-1-04x09] 17:54:20,995 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-04x09] 17:54:20,996 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-04x09] 17:54:21,109 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel clustered
      [transactions-repository-1-04x09] 17:54:21,129 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel clustered: [transactions-repository-1-4z05w|2] (3) [transactions-repository-1-4z05w, transactions-repository-1-04x09, transactions-repository-1-1f8dx]
      [transactions-repository-1-04x09] 17:54:21,138 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel clustered local address is transactions-repository-1-04x09, physical addresses are [172.17.0.4:7600]
      [transactions-repository-1-1f8dx] 17:54:21,158 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-1f8dx] 17:54:21,158 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-04x09] 17:54:21,160 INFO  [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-3) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2017-02-18T17:54:21.156Z
      [transactions-repository-1-04x09] 17:54:21,160 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-3) ISPN000128: Infinispan version: Infinispan 'Ruppaner' 9.0.0-SNAPSHOT
      [transactions-repository-1-1f8dx] 17:54:21,254 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups channel clustered
      [transactions-repository-1-1f8dx] 17:54:21,266 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000094: Received new cluster view for channel clustered: [transactions-repository-1-4z05w|2] (3) [transactions-repository-1-4z05w, transactions-repository-1-04x09, transactions-repository-1-1f8dx]
      [transactions-repository-1-1f8dx] 17:54:21,269 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel clustered local address is transactions-repository-1-1f8dx, physical addresses are [172.17.0.6:7600]
      [transactions-repository-1-1f8dx] 17:54:21,282 INFO  [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-8) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2017-02-18T17:54:21.279Z
      [transactions-repository-1-1f8dx] 17:54:21,283 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Ruppaner' 9.0.0-SNAPSHOT
      [transactions-repository-1-04x09] 17:54:21,297 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-04x09] 17:54:21,297 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-04x09] 17:54:21,300 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-6) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-04x09] 17:54:21,303 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-04x09] 17:54:21,305 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-6) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-04x09] 17:54:21,310 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-6) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-04x09] 17:54:21,310 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-04x09] 17:54:21,311 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-1f8dx] 17:54:21,443 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-1f8dx] 17:54:21,443 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
      [transactions-repository-1-1f8dx] 17:54:21,449 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-1f8dx] 17:54:21,450 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-1f8dx] 17:54:21,452 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-4) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-1f8dx] 17:54:21,462 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-4) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-1f8dx] 17:54:21,462 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [transactions-repository-1-1f8dx] 17:54:21,468 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-4) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
      [transactions-repository-1-04x09] [GC (CMS Initial Mark)  208341K(1013632K), 0.0101475 secs]
      [transactions-repository-1-4z05w] 17:54:23,974 ERROR [org.infinispan.topology.ClusterCacheStatus] (transport-thread--p4-t9) ISPN000228: Failed to recover cache transactions state after the current node became the coordinator: java.lang.NullPointerException
      [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterCacheStatus.updateAvailabilityMode(ClusterCacheStatus.java:147)
      [transactions-repository-1-4z05w] at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.updateMembersAndRebalance(PreferConsistencyStrategy.java:259)
      [transactions-repository-1-4z05w] at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.onPartitionMerge(PreferConsistencyStrategy.java:210)
      [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterCacheStatus.doMergePartitions(ClusterCacheStatus.java:554)
      [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$recoverClusterStatus$3(ClusterTopologyManagerImpl.java:499)
      [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
      [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
      [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
      [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [transactions-repository-1-4z05w] at java.lang.Thread.run(Thread.java:745)
      [transactions-repository-1-4z05w] 
      [transactions-repository-1-4z05w] 17:54:23,974 ERROR [org.infinispan.topology.ClusterCacheStatus] (transport-thread--p4-t7) ISPN000228: Failed to recover cache sessions state after the current node became the coordinator: java.lang.NullPointerException
      [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterCacheStatus.updateAvailabilityMode(ClusterCacheStatus.java:147)
      [transactions-repository-1-4z05w] at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.updateMembersAndRebalance(PreferConsistencyStrategy.java:259)
      [transactions-repository-1-4z05w] at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.onPartitionMerge(PreferConsistencyStrategy.java:210)
      [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterCacheStatus.doMergePartitions(ClusterCacheStatus.java:554)
      [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$recoverClusterStatus$3(ClusterTopologyManagerImpl.java:499)
      [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
      [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
      [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
      [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [transactions-repository-1-4z05w] at java.lang.Thread.run(Thread.java:745)
      [transactions-repository-1-4z05w] 
      [transactions-repository-1-4z05w] 17:54:24,005 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) DGISPN0001: Started transactions cache from clustered container
      [transactions-repository-1-4z05w] 17:54:24,005 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) DGISPN0001: Started default cache from clustered container
      [transactions-repository-1-04x09] 17:54:24,005 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0001: Started default cache from clustered container
      [transactions-repository-1-04x09] 17:54:24,006 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started memcachedCache cache from clustered container
      [transactions-repository-1-04x09] 17:54:24,006 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) DGISPN0001: Started transactions cache from clustered container
      [transactions-repository-1-4z05w] 17:54:24,008 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0001: Started memcachedCache cache from clustered container
      [transactions-repository-1-4z05w] 17:54:24,010 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0001: Started sessions cache from clustered container
      [transactions-repository-1-04x09] 17:54:24,011 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) DGISPN0001: Started sessions cache from clustered container
      [transactions-repository-1-1f8dx] 17:54:24,012 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) DGISPN0001: Started sessions cache from clustered container
      [transactions-repository-1-1f8dx] 17:54:24,012 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0001: Started memcachedCache cache from clustered container
      [transactions-repository-1-1f8dx] 17:54:24,012 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0001: Started default cache from clustered container
      [transactions-repository-1-1f8dx] 17:54:24,013 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0001: Started transactions cache from clustered container
      [transactions-repository-1-4z05w] 17:54:24,036 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10000: MemcachedServer starting
      [transactions-repository-1-4z05w] 17:54:24,039 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10000: HotRodServer starting
      [transactions-repository-1-4z05w] 17:54:24,039 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10001: HotRodServer listening on 172.17.0.3:11222
      [transactions-repository-1-4z05w] 17:54:24,040 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: WebSocketServer starting
      [transactions-repository-1-4z05w] 17:54:24,040 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: WebSocketServer listening on 172.17.0.3:8181
      [transactions-repository-1-1f8dx] 17:54:24,046 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10000: WebSocketServer starting
      [transactions-repository-1-1f8dx] 17:54:24,045 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: MemcachedServer starting
      [transactions-repository-1-1f8dx] 17:54:24,046 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: HotRodServer starting
      [transactions-repository-1-1f8dx] 17:54:24,047 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: HotRodServer listening on 172.17.0.6:11222
      [transactions-repository-1-1f8dx] 17:54:24,048 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: MemcachedServer listening on 172.17.0.6:11211
      [transactions-repository-1-04x09] 17:54:24,051 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10000: MemcachedServer starting
      [transactions-repository-1-04x09] 17:54:24,051 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: WebSocketServer starting
      [transactions-repository-1-04x09] 17:54:24,052 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: HotRodServer starting
      [transactions-repository-1-04x09] 17:54:24,055 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: HotRodServer listening on 172.17.0.4:11222
      [transactions-repository-1-04x09] 17:54:24,055 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10001: MemcachedServer listening on 172.17.0.4:11211
      [transactions-repository-1-04x09] 17:54:24,056 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: WebSocketServer listening on 172.17.0.4:8181
      [transactions-repository-1-4z05w] 17:54:24,063 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10001: MemcachedServer listening on 172.17.0.3:11211
      [transactions-repository-1-1f8dx] 17:54:24,059 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10001: WebSocketServer listening on 172.17.0.6:8181
      [transactions-repository-1-4z05w] 17:54:24,070 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: REST starting
      [transactions-repository-1-1f8dx] 17:54:24,079 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10000: REST starting
      [transactions-repository-1-04x09] 17:54:24,071 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-8) DGENDPT10000: REST starting
      [transactions-repository-1-4z05w] 17:54:24,193 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t20) ISPN000310: Starting cluster-wide rebalance for cache ___protobuf_metadata, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[transactions-repository-1-4z05w: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 126, transactions-repository-1-1f8dx: 130]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-1f8dx], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, ced31139-f1a1-4ee8-96af-f37c5386ecc0]}
      [transactions-repository-1-4z05w] 17:54:24,197 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=___protobuf_metadata][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance
      [transactions-repository-1-4z05w] 17:54:24,246 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t24) [Context=___protobuf_metadata][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,255 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000310: Starting cluster-wide rebalance for cache ___script_cache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[transactions-repository-1-4z05w: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 126, transactions-repository-1-1f8dx: 130]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-1f8dx], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, ced31139-f1a1-4ee8-96af-f37c5386ecc0]}
      [transactions-repository-1-4z05w] 17:54:24,256 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t22) [Context=___script_cache][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance
      [transactions-repository-1-4z05w] 17:54:24,272 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t2) [Context=___script_cache][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,317 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t2) ISPN000310: Starting cluster-wide rebalance for cache ___hotRodTopologyCache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[transactions-repository-1-4z05w: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 134, transactions-repository-1-04x09: 122]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-04x09], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, 5d246b27-021c-4425-b3c4-14017f378635]}
      [transactions-repository-1-4z05w] 17:54:24,318 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=___hotRodTopologyCache][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance
      [transactions-repository-1-4z05w] 17:54:24,321 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t11) [Context=___hotRodTopologyCache][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,364 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t8) [Context=___protobuf_metadata][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,366 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=___script_cache][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,382 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=___hotRodTopologyCache][Context=transactions-repository-1-04x09]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,385 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t8) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 2
      [transactions-repository-1-4z05w] 17:54:24,387 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t8) ISPN000310: Starting cluster-wide rebalance for cache ___protobuf_metadata, topology CacheTopology{id=4, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 126, transactions-repository-1-1f8dx: 130]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[transactions-repository-1-4z05w: 90, transactions-repository-1-1f8dx: 86, transactions-repository-1-04x09: 80]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-1f8dx, transactions-repository-1-04x09], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, ced31139-f1a1-4ee8-96af-f37c5386ecc0, 5d246b27-021c-4425-b3c4-14017f378635]}
      [transactions-repository-1-4z05w] 17:54:24,388 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t8) [Context=___protobuf_metadata][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance
      [transactions-repository-1-4z05w] 17:54:24,394 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t6) ISPN000336: Finished cluster-wide rebalance for cache ___hotRodTopologyCache, topology id = 2
      [transactions-repository-1-4z05w] 17:54:24,395 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000336: Finished cluster-wide rebalance for cache ___script_cache, topology id = 2
      [transactions-repository-1-4z05w] 17:54:24,397 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t19) [Context=___protobuf_metadata][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,397 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t7) [Context=___protobuf_metadata][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,398 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000310: Starting cluster-wide rebalance for cache ___script_cache, topology CacheTopology{id=4, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 126, transactions-repository-1-1f8dx: 130]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[transactions-repository-1-4z05w: 90, transactions-repository-1-1f8dx: 86, transactions-repository-1-04x09: 80]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-1f8dx, transactions-repository-1-04x09], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, ced31139-f1a1-4ee8-96af-f37c5386ecc0, 5d246b27-021c-4425-b3c4-14017f378635]}
      [transactions-repository-1-4z05w] 17:54:24,398 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=___script_cache][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance
      [transactions-repository-1-4z05w] 17:54:24,401 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t6) ISPN000310: Starting cluster-wide rebalance for cache ___hotRodTopologyCache, topology CacheTopology{id=4, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 134, transactions-repository-1-04x09: 122]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[transactions-repository-1-4z05w: 90, transactions-repository-1-04x09: 80, transactions-repository-1-1f8dx: 86]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-04x09, transactions-repository-1-1f8dx], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, 5d246b27-021c-4425-b3c4-14017f378635, ced31139-f1a1-4ee8-96af-f37c5386ecc0]}
      [transactions-repository-1-4z05w] 17:54:24,405 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=___hotRodTopologyCache][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance
      [transactions-repository-1-4z05w] 17:54:24,407 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t20) [Context=___hotRodTopologyCache][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,409 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t3) [Context=___script_cache][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,420 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t11) [Context=___script_cache][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,425 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t11) [Context=___hotRodTopologyCache][Context=transactions-repository-1-04x09]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,476 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t17) [Context=___hotRodTopologyCache][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,480 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t17) ISPN000336: Finished cluster-wide rebalance for cache ___hotRodTopologyCache, topology id = 4
      [transactions-repository-1-4z05w] 17:54:24,536 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t21) [Context=___protobuf_metadata][Context=transactions-repository-1-04x09]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,537 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t21) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 4
      [transactions-repository-1-4z05w] 17:54:24,544 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t24) [Context=___script_cache][Context=transactions-repository-1-04x09]ISPN100003: Finished local rebalance
      [transactions-repository-1-4z05w] 17:54:24,546 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t24) ISPN000336: Finished cluster-wide rebalance for cache ___script_cache, topology id = 4
      [transactions-repository-1-4z05w] 17:54:24,567 INFO  [org.infinispan.rest.NettyRestServer] (MSC service thread 1-1) ISPN012003: REST server starting, listening on 172.17.0.3:8080
      [transactions-repository-1-4z05w] 17:54:24,568 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10002: REST mapped to /rest
      [transactions-repository-1-1f8dx] 17:54:24,576 INFO  [org.infinispan.rest.NettyRestServer] (MSC service thread 1-6) ISPN012003: REST server starting, listening on 172.17.0.6:8080
      [transactions-repository-1-1f8dx] 17:54:24,577 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10002: REST mapped to /rest
      [transactions-repository-1-04x09] 17:54:24,639 INFO  [org.infinispan.rest.NettyRestServer] (MSC service thread 1-8) ISPN012003: REST server starting, listening on 172.17.0.4:8080
      [transactions-repository-1-04x09] 17:54:24,639 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-8) DGENDPT10002: REST mapped to /rest
      [transactions-repository-1-1f8dx] [GC (Allocation Failure)  309342K->63160K(1013632K), 0.1823599 secs]
      [transactions-repository-1-4z05w] [GC (Allocation Failure)  309352K->65174K(1013632K), 0.1616480 secs]
      [transactions-repository-1-04x09] [GC (Allocation Failure)  309268K->61649K(1013632K), 0.1507677 secs]
      [transactions-repository-1-1f8dx] 17:54:25,021 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management
      [transactions-repository-1-1f8dx] 17:54:25,022 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990
      [transactions-repository-1-1f8dx] 17:54:25,022 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) started in 9718ms - Started 197 of 282 services (164 services are lazy, passive or on-demand)
      [transactions-repository-1-4z05w] 17:54:25,021 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management
      [transactions-repository-1-4z05w] 17:54:25,024 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990
      [transactions-repository-1-4z05w] 17:54:25,027 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) started in 9643ms - Started 197 of 282 services (164 services are lazy, passive or on-demand)
      [transactions-repository-1-04x09] 17:54:25,063 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management
      [transactions-repository-1-04x09] 17:54:25,064 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990
      [transactions-repository-1-04x09] 17:54:25,064 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) started in 9746ms - Started 197 of 282 services (164 services are lazy, passive or on-demand)
      [transactions-repository-1-04x09] [GC (CMS Final Remark)  89552K(1013632K), 0.0310821 secs]
      

      Note that I applied multiplexed logs from 3 nodes. I hope this will be helpful for diagnosing.

      Steps to reproduce:

              slaskawi@redhat.com Sebastian Łaskawiec (Inactive)
              slaskawi@redhat.com Sebastian Łaskawiec (Inactive)
              Archiver:
              rhn-support-adongare Amol Dongare

                Created:
                Updated:
                Resolved:
                Archived: