Uploaded image for project: 'A-MQ Broker'
  1. A-MQ Broker
  2. ENTMQBR-3865

[LTS] Enabling group rebalancing with default / non-zero consumer-window-size can lead to out-of-order message consumption

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Explained
    • Affects Version/s: AMQ 7.7.0.GA
    • Fix Version/s: None
    • Component/s: broker-core
    • Target Release:
    • Steps to Reproduce:
      Hide

      Packaged reproducer WIP, but relatively easy to reproduce with a vanilla broker configuration with rebalancing enabled and a simple spring consumer with concurrentConsumers > 1.

      Show
      Packaged reproducer WIP, but relatively easy to reproduce with a vanilla broker configuration with rebalancing enabled and a simple spring consumer with concurrentConsumers > 1.
    • Workaround Description:
      Hide

      Explicitly set consumerWindowSize / Prefetch = 0 for consumers on the affected addresses

      Show
      Explicitly set consumerWindowSize / Prefetch = 0 for consumers on the affected addresses

      Description

      When group rebalancing is enabled on a queue with group-rebalance=true or globally with default-group-rebalance=true, then the broker shuffles group assignments for consumers every time the consumer count changes. When we combine this with a prefetch / consumerWindowSize > 0, here is what happens:

      As a multithreaded listener container starts up, it creates new consumer threads up to the concurrent consumer count. These consumers are generally not all created at once, so each time a new consumer starts, the broker re-balances the groups. This is all fine, except that existing consumers already have messages prefetched out to them. So the consumers get rebalanced, but the existing consumers continue to process messages that have already been prefetched to them - so now we potentially have concurrent processing going on within the same groups, until the prefetch buffer for each consumer is worked off.

      It seems like the broker should deal with this by either closing consumers or pausing dispatch until the delivering count is 0 before effecting the rebalance or by disabling prefetch for consumers on the address when group rebalancing is enabled. Alternatively, the documentation should contain a warning that enabling rebalancing with a non-zero consumer-window-size is not recommended when message ordering must be preserved.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  jbertram Justin Bertram
                  Reporter:
                  dbruscin Domenico Francesco Bruscino
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: