Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-2970

TopologyAwareConsistentHashFactory.rebalance can fail with an IndexOutOfBoundsException

    XMLWordPrintable

    Details

      Description

      In rare circumstances, TopologyAwareConsistentHashFactory can fail to allocate numOwners owners to each segment, resulting in an IndexOutOfBoundsException:

      09:41:23,009 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (undefined) Failed to start rebalance: Index: 1, Size: 1: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
      	at java.util.ArrayList.RangeCheck(ArrayList.java:547) [rt.jar:1.6.0_43]
      	at java.util.ArrayList.get(ArrayList.java:322) [rt.jar:1.6.0_43]
      	at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.doReplaceBackupOwnersSameMachine(TopologyAwareConsistentHashFactory.java:149) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.replaceBackupOwnerForMachineLevel(TopologyAwareConsistentHashFactory.java:131) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.addBackupOwners(TopologyAwareConsistentHashFactory.java:67) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalanceBuilder(DefaultConsistentHashFactory.java:125) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalance(DefaultConsistentHashFactory.java:106) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalance(DefaultConsistentHashFactory.java:45) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.topology.ClusterTopologyManagerImpl.startRebalance(ClusterTopologyManagerImpl.java:370) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.topology.ClusterTopologyManagerImpl.access$000(ClusterTopologyManagerImpl.java:66) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at org.infinispan.topology.ClusterTopologyManagerImpl$1.call(ClusterTopologyManagerImpl.java:128) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_43]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_43]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
      	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
      	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2]
      

      This can happen only after a node leaves. When it happens, the rebalance is stopped and the some keys will remain with less than numOwners owners. If another node leaves, some keys may be lost. If another node joins, the next rebalance will succeed and the keys will be redistributed properly.

        Attachments

          Activity

            People

            Assignee:
            dan.berindei Dan Berindei
            Reporter:
            dan.berindei Dan Berindei
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: