Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3744

Enabling group rebalancing with default / non-zero consumer-window-size can lead to out-of-order message consumption

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: AMQ 7.7.0.GA
    • Fix Version/s: AMQ 7.8.0.CR1
    • Component/s: broker-core
    • Labels:
      None
    • Target Release:
    • Steps to Reproduce:
      Hide

      Packaged reproducer WIP, but relatively easy to reproduce with a vanilla broker configuration with rebalancing enabled and a simple spring consumer with concurrentConsumers > 1.

      Show
      Packaged reproducer WIP, but relatively easy to reproduce with a vanilla broker configuration with rebalancing enabled and a simple spring consumer with concurrentConsumers > 1.
    • Workaround Description:
      Hide

      Explicitly set consumerWindowSize / Prefetch = 0 for consumers on the affected addresses

      Show
      Explicitly set consumerWindowSize / Prefetch = 0 for consumers on the affected addresses
    • Release Notes Text:
      Hide
      Previously, if some connected consumers were using a value of `consumerWindowSize` greater than zero (meaning that messages were pre-fetched into buffers on those clients) *and* you configured the broker to use group rebalancing (by setting `group-rebalance` or `default-group-rebalance` to `true`), this might result in out-of-order message consumption. This issue is now resolved.
      Show
      Previously, if some connected consumers were using a value of `consumerWindowSize` greater than zero (meaning that messages were pre-fetched into buffers on those clients) *and* you configured the broker to use group rebalancing (by setting `group-rebalance` or `default-group-rebalance` to `true`), this might result in out-of-order message consumption. This issue is now resolved.
    • Release Notes Docs Status:
      Documented as Resolved Issue
    • Verified:
      Verified in a release

      Description

      When group rebalancing is enabled on a queue with group-rebalance=true or globally with default-group-rebalance=true, then the broker shuffles group assignments for consumers every time the consumer count changes. When we combine this with a prefetch / consumerWindowSize > 0, here is what happens:

      As a multithreaded listener container starts up, it creates new consumer threads up to the concurrent consumer count. These consumers are generally not all created at once, so each time a new consumer starts, the broker re-balances the groups. This is all fine, except that existing consumers already have messages prefetched out to them. So the consumers get rebalanced, but the existing consumers continue to process messages that have already been prefetched to them - so now we potentially have concurrent processing going on within the same groups, until the prefetch buffer for each consumer is worked off.

      It seems like the broker should deal with this by either closing consumers or pausing dispatch until the delivering count is 0 before effecting the rebalance or by disabling prefetch for consumers on the address when group rebalancing is enabled. Alternatively, the documentation should contain a warning that enabling rebalancing with a non-zero consumer-window-size is not recommended when message ordering must be preserved.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jbertram Justin Bertram
              Reporter:
              hawkinsds Duane Hawkins
              Tester:
              Oleg Sushchenko Oleg Sushchenko
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: