Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-7749

A node leaving during rebalance causes extra segment transfers

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Duplicate Issue
    • Affects Version/s: 9.0.0.Final
    • Fix Version/s: None
    • Component/s: Core, State Transfer
    • Labels:
      None

      Description

      If a node leaves during a rebalance, the only change is that other nodes will no longer request segments from that node. Otherwise the rebalance proceeds as usual, and at the end a node may delete its copy of a segment even if that leaves less than numOwners copies in the cluster.

      When the leaver is the joiner, a 2nd rebalance is very likely to return to the initial CH, but only after transferring some segments twice:

      Rebalance starts: current_owners(s) = AB, pending_owners(s) = AC
      C leaves: current_owners(s) = AB, pending_owners(s) = A
      Rebalance finishes: current_owners(s) = A, pending_owners(s) = A
      2nd rebalance starts: current_owners(s) = A, pending_owners(s) = AB
      

      Even if the leaver is one of the old owners and the 2 segment transfers are necessary, the cluster stays for too long with less than numOwners copies of the segment:

      Rebalance starts: current_owners(s) = AB, pending_owners(s) = AC
      A leaves: current_owners(s) = B, pending_owners(s) = C
      C transfers segment s from B
      Rebalance finishes: current_owners(s) = C, pending_owners(s) = C
      2nd rebalance starts: current_owners(s) = C, pending_owners(s) = DC
      

      We can fix this by making the pending CH a union of the current and pending CH whenever a node leavers, and only removing extra segment copies after the 2nd rebalance.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  dan.berindei Dan Berindei
                  Reporter:
                  dan.berindei Dan Berindei
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: