Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-3738

Bad uneven segment distributions can happen after the rebalancing with SyncConsistentHashFactory

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: JDG 7.2.3 GA, RHDG 7.3.5 GA, RHDG 8.0 GA
    • Fix Version/s: RHDG 8.1 GA, RHDG 8.1 CD
    • Component/s: Server
    • Labels:
      None
    • Target Release:
    • Fix Build:
      CR1
    • Steps to Reproduce:
      Hide
      • Add a unit test "testRebalanceDistribution" in SyncConsistentHashFactoryKeyDistributionTest.

      https://github.com/infinispan/infinispan/compare/master...msfm:master_SyncCHFKeyDistTest

      • Execute unit test
      $ mvn -pl core test -Dtest=distribution.ch.SyncConsistentHashFactoryKeyDistributionTest#testRebalanceDistribution
      
      • Check the test result, especially "Primary: max", "Primary: min" and "Segments per node - max/min ratio". You will see "Primary: min" that indicates around half (around 0.500-0.600) and "Segments per node - max/min ratio" indicates around two times (around 1.7-2.0).
      Show
      Add a unit test "testRebalanceDistribution" in SyncConsistentHashFactoryKeyDistributionTest. https://github.com/infinispan/infinispan/compare/master...msfm:master_SyncCHFKeyDistTest Execute unit test $ mvn -pl core test -Dtest=distribution.ch.SyncConsistentHashFactoryKeyDistributionTest#testRebalanceDistribution Check the test result, especially "Primary: max", "Primary: min" and "Segments per node - max/min ratio". You will see "Primary: min" that indicates around half (around 0.500-0.600) and "Segments per node - max/min ratio" indicates around two times (around 1.7-2.0).

      Description

      Uneven segment distribution can happen after rebalancing with SyncConsistentHashFactory.

      Even if a larger number of segments than "20 * number of nodes" is configured, some nodes have only around 50-60% number of segments than other nodes after the rebalance in the worst case.

      I understand the documentation (like the following) states about the possibility of slight uneven distribution as a potential drawback of SyncConsistentHashFactory.

      Infinispan 10 configuration guide says:

      This implementation does have some negative points in that the load distribution is slightly uneven. It also moves more segments than strictly necessary on a join or leave.

      JDG 7 Developer Guide says:

      It has a potential drawback in that it can move a greater number of segments than necessary during re-balancing. This can be mitigated by using a larger number of segments.
      Another potential drawback is that the segments are not distributed as evenly as possible, and actually using a very large number of segments can make the distribution of segments worse.

      However, only half segments than others is not a slight uneven.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dan.berindei Dan Berindei
              Reporter:
              mmiura Masafumi Miura
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: