Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-6960

[LTS] Multiple consumers on the same LV queue can starve one another of messages

XMLWordPrintable

    • False
    • None
    • False
    • Hide

      There are many ways to reproduce a problem that results in multiple consumers on the same LV queue starving one another of messages. I believe all these manifestations are related, so I will describe only two of them. All use the "amqutil" general-purpose test client, which is available here:

      https://github.com/kevinboone/amqutil

      All reproduce only with AMQP protocol. I'm sure that the client is not to blame, or even the runtime library, because the customer originally reported these problems with Proton C++ clients.

      In all cases, the result is that, although the producer is continuing to send messages, no consumer ever receives a message again. These problems only reproduce with last-value queues.

      Method 1 – stopping a consumer
      ===========================

      1. Start the producer, publishing messages who bodies are an ascending sequence of numbers, and with the same LV key:

      $ amqutil publish 10000 --qpid --destination SELECTOR --numbered --properties MESSAGE_KEY=1
       
      2. Start two consumers on the same LV queue, in different sessions:
       
      $ amqutil subscribe 10000 --qpid --destination SELECTOR::QUEUE.SEL (twice)
       
      3. Stop one of these consumers. Note that the other consumer also stops receiving messages, even though messages are still being produced.

       

      Method 2 – applying a non-matching selector

      =====================================

       1. In one session, produce using AMQP a stream of messages whose bodies are incrementing numbers, with LV key some fixed value (doesn't matter what the value is), and a property "foo=bar1" (which will be matched by one consumer's selector)

       $ amqutil publish 10000 --qpid --destination SELECTOR --numbered --properties foo=bar1,MESSAGE_KEY=1

      2. In another session, start to consume messages with a matching selector.

      $ amqutil subscribe 10000 --qpid --destination SELECTOR::QUEUE.SEL --selector "foo='bar1'"

      A constant stream of messages is received because, although this is a LV queue, the "last value" is constantly changing.

      3. In another session, start to consume messages with a non-matching selector.

      $ amqutil subscribe 10000 --qpid --destination SELECTOR::QUEUE.SEL --selector "foo='bar2'"

      This consumer receives no messages and – even more oddly – the first consumer (with the matching selector) receives no messages either.

      Preparation

      ==========

      Define the last-value queue in broker.xml like this:

          <address-setting match="SELECTOR">
                  <default-last-value-key>MESSAGE_KEY</default-last-value-key>
                  <default-non-destructive>true</default-non-destructive>
           </address-setting>
      
           <address name="SELECTOR">
                  <multicast>
                     <queue name="QUEUE.SEL" />
                  </multicast>
           </address> 

       

       

       

       

       

       

       

      Show
      There are many ways to reproduce a problem that results in multiple consumers on the same LV queue starving one another of messages. I believe all these manifestations are related, so I will describe only two of them. All use the "amqutil" general-purpose test client, which is available here: https://github.com/kevinboone/amqutil All reproduce only with AMQP protocol. I'm sure that the client is not to blame, or even the runtime library, because the customer originally reported these problems with Proton C++ clients. In all cases, the result is that, although the producer is continuing to send messages, no consumer ever receives a message again. These problems only reproduce with last-value queues. Method 1 – stopping a consumer =========================== 1. Start the producer, publishing messages who bodies are an ascending sequence of numbers, and with the same LV key: $ amqutil publish 10000 --qpid --destination SELECTOR --numbered --properties MESSAGE_KEY=1   2. Start two consumers on the same LV queue, in different sessions:   $ amqutil subscribe 10000 --qpid --destination SELECTOR::QUEUE.SEL (twice)   3. Stop one of these consumers. Note that the other consumer also stops receiving messages, even though messages are still being produced.   Method 2 – applying a non-matching selector =====================================  1. In one session, produce using AMQP a stream of messages whose bodies are incrementing numbers, with LV key some fixed value (doesn't matter what the value is), and a property "foo=bar1" (which will be matched by one consumer's selector)  $ amqutil publish 10000 --qpid --destination SELECTOR --numbered --properties foo=bar1,MESSAGE_KEY=1 2. In another session, start to consume messages with a matching selector. $ amqutil subscribe 10000 --qpid --destination SELECTOR::QUEUE.SEL --selector "foo='bar1'" A constant stream of messages is received because, although this is a LV queue, the "last value" is constantly changing. 3. In another session, start to consume messages with a non-matching selector. $ amqutil subscribe 10000 --qpid --destination SELECTOR::QUEUE.SEL --selector "foo='bar2'" This consumer receives no messages and – even more oddly – the first consumer (with the matching selector) receives no messages either. Preparation ========== Define the last-value queue in broker.xml like this: <address-setting match= "SELECTOR" > < default -last-value-key>MESSAGE_KEY</ default -last-value-key> < default -non-destructive> true </ default -non-destructive> </address-setting> <address name= "SELECTOR" > <multicast> <queue name= "QUEUE.SEL" /> </multicast> </address>              

      When multiple consumers are attached to the same last-value queue, the broker distributes messages between the consumers. It appears that the broker is rather zealous in the regularity of this distribution: if a consumer can't accept the message, then the broker won't try to send it somewhere else. The result is that nobody receives any messages.

      "Not accepting" the message can take many forms. I have investigated situations where consumers have different selectors but, in fact, simply shutting down one of the consumers will result in no further messages being sent to any consumer.

      I don't see any of these problems with non-LV queues – the broker seems to be able to re-route messages when consumers won't accept them. It's only with LV queues that I see what appears to be a dogged determination to send a message to a consumer that doesn't want it, to the detriment of other consumers.

      These problems reproduce with Java (Qpid JMS) and C++ (Qpid Proton) clients, using the AMQP protocol.

              rhn-support-jbertram Justin Bertram
              rhn-support-kboone Kevin Boone
              Samuel Gajdos Samuel Gajdos
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: