Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-2970

TopologyAwareConsistentHashFactory.rebalance can fail with an IndexOutOfBoundsException

    XMLWordPrintable

Details

    Description

      In rare circumstances, TopologyAwareConsistentHashFactory can fail to allocate numOwners owners to each segment, resulting in an IndexOutOfBoundsException:

      09:41:23,009 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (undefined) Failed to start rebalance: Index: 1, Size: 1: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
      	at java.util.ArrayList.RangeCheck(ArrayList.java:547) [rt.jar:1.6.0_43]
      	at java.util.ArrayList.get(ArrayList.java:322) [rt.jar:1.6.0_43]
      	at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.doReplaceBackupOwnersSameMachine(TopologyAwareConsistentHashFactory.java:149) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.replaceBackupOwnerForMachineLevel(TopologyAwareConsistentHashFactory.java:131) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.addBackupOwners(TopologyAwareConsistentHashFactory.java:67) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalanceBuilder(DefaultConsistentHashFactory.java:125) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalance(DefaultConsistentHashFactory.java:106) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalance(DefaultConsistentHashFactory.java:45) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.topology.ClusterTopologyManagerImpl.startRebalance(ClusterTopologyManagerImpl.java:370) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.topology.ClusterTopologyManagerImpl.access$000(ClusterTopologyManagerImpl.java:66) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.topology.ClusterTopologyManagerImpl$1.call(ClusterTopologyManagerImpl.java:128) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_43]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_43]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
      	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
      	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2]
      

      This can happen only after a node leaves. When it happens, the rebalance is stopped and the some keys will remain with less than numOwners owners. If another node leaves, some keys may be lost. If another node joins, the next rebalance will succeed and the keys will be redistributed properly.

      Attachments

        Activity

          People

            dberinde@redhat.com Dan Berindei
            dberinde@redhat.com Dan Berindei
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: