Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3820

Doc: Enabling group rebalancing with default / non-zero consumer-window-size can lead to out-of-order message consumption

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Do
    • Affects Version/s: AMQ 7.7.0.GA
    • Fix Version/s: None
    • Component/s: documentation
    • Labels:
      None

      Description

      When group rebalancing is enabled on a queue with group-rebalance=true or globally with default-group-rebalance=true, then the broker shuffles group assignments for consumers every time the consumer count changes. When we combine this with a prefetch / consumerWindowSize > 0, here is what happens:

      As a multithreaded listener container starts up, it creates new consumer threads up to the concurrent consumer count. These consumers are generally not all created at once, so each time a new consumer starts, the broker re-balances the groups. This is all fine, except that existing consumers already have messages prefetched out to them. So the consumers get rebalanced, but the existing consumers continue to process messages that have already been prefetched to them - so now we potentially have concurrent processing going on within the same groups, until the prefetch buffer for each consumer is worked off.

      It seems like the broker should deal with this by either closing consumers or pausing dispatch until the delivering count is 0 before effecting the rebalance or by disabling prefetch for consumers on the address when group rebalancing is enabled. Alternatively, the documentation should contain a warning that enabling rebalancing with a non-zero consumer-window-size is not recommended when message ordering must be preserved.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jbyrne1 John Byrne
              Reporter:
              jbyrne1 John Byrne
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: