Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-6107

State transfer should not fetch segments that were added during a rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 8.2.0.Beta1
    • 8.1.0.Final
    • Core, Test Suite
    • None

    Description

      When the last owner of a segment leaves the cache, the coordinator will update the consistent hash and replace that owner with numOwners owners (so that a segment always has at least 1 owner). If there is a rebalance in progress, it could be that both the current and the pending CH lost all the owners of a segment, and the coordinator will assign new owners in both CHs (not necessarily the same).

      Sometimes, this causes tests that create clusters with many nodes to spend a lot of time shutting down the cluster. Here's an example:

      1. Cluster ABCDE, coordinator A, topology id = 0, currentCH = {0: CD, 1: BC}, pendingCH = null
      2. D leaves
      3. A broadcasts a REBALANCE_START command with topology id 1, members = ABCE, currentCH = {0: C, 1: BC}, pendingCH = {0: BC, 1: BC}
      4. A and E confirm that they finished the rebalance
      5. C leaves before sending the data for segment 0 to B
      6. A broadcasts a CH_UPDATE command with topology id 2, members = ABE, currentCH = {0: AE, 1: B}, pendingCH = {0: B, 1: B}
      7. A now owns segment 0 in the writeCH (which is the union of currentCH and pendingCH).
      8. A tries to request segment 0 from the other owner in the currentCH, E
      9. B confirms that it finished the rebalance
      10. A broadcasts a new topology: topology id 3, currentCH = {0: B, 1: B}, pendingCH = null
      11. E installs topology 3, and throws an IllegalArgumentException when handling A's request for segments
      12. A is not able to install topology 3, because it requests the transactions data while holding the lock on the LocalCacheStatus
      13. A receives the IllegalArgumentException from E and retries. But because it still has the old topology, it retries on E ad infinitum - using a lot of CPU in the process.

      A requesting segment 0 from E is not a problem in itself - normally E would just send back an empty set of transactions and entries. The problem is that the cluster is able to install a new topology, because A already confirmed receiving all the data, but A is stuck with the old topology.

      Attachments

        Activity

          People

            dberinde@redhat.com Dan Berindei (Inactive)
            dberinde@redhat.com Dan Berindei (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: