Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-7385

Message flops around the federation queue on slow consumers

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Normal
    • AMQ 7.11.1.GA
    • AMQ 7.10.2.GA
    • broker-core
    • None
    • False
    • None
    • False
    • Documentation (Ref Guide, User Guide, etc.)
    • Hide

      The behavior is evident with wither amqp / qpid clients or openwire clients. To reproduce:

      1. Use the configuration files from the attached broker.zip archive to configure a federated broker pair (note: the attached configuration is for 2 shared store ha pairs, but I verified after attaching that the issue occurs without the local cluster / ha configuration, as well). Important: Include the uncommented broker-logging plugin configuration, as in the attached files.
      2. Extract the attached threaded-artemis-jms-consumer-1.0-SNAPSHOT.zip and update jndi.properties to change the broker URL to point to the first federated node
      3. Extract the attached threaded-artemis-jms-consumer-1.0-SNAPSHOT.zip a second time and configure the jndi.properties to change the broker URL to point to the second node (opposite end of the federated pair)
      4. Extract the attached threaded-artemis-jms-producer-1.0-SNAPSHOT.zip and configure jndi.properties to change the broker URL to point to the first federated node
      5. Start the node1 and node2 brokers
      6. Run the consumer.sh in each of the consumers from steps 2 and 3 to create a consumer at each end of the federation
      7. Check the broker consoles and verify that each broker shows 2 consumers for TEST.Q.0 - one local and one from the opposite node
      8. Start the producer by running the producer.sh script and let it finish. It should produce 5000 messages.
      9. As the consumers slowly work off the load, periodically grep the logs for the number of deliveries made:

      cat log/artemis.log | grep delivered\ message | wc -l
      

      10. At first one broker shows 5000 and the other a slowly increasing count. Let it continue. After a few minutes, you will see the count on the second node jump as messages are pushed back to the first node. After a few more minutes, the count on both nodes starts jumping rapidly, hitting in the 10s of thousands and then millions. These numbers can also be verified by looking at the added and acked counts on the queue (TEST.Q.0).

      Show
      The behavior is evident with wither amqp / qpid clients or openwire clients. To reproduce: 1. Use the configuration files from the attached broker.zip archive to configure a federated broker pair (note: the attached configuration is for 2 shared store ha pairs, but I verified after attaching that the issue occurs without the local cluster / ha configuration, as well). Important: Include the uncommented broker-logging plugin configuration, as in the attached files. 2. Extract the attached threaded-artemis-jms-consumer-1.0-SNAPSHOT.zip and update jndi.properties to change the broker URL to point to the first federated node 3. Extract the attached threaded-artemis-jms-consumer-1.0-SNAPSHOT.zip a second time and configure the jndi.properties to change the broker URL to point to the second node (opposite end of the federated pair) 4. Extract the attached threaded-artemis-jms-producer-1.0-SNAPSHOT.zip and configure jndi.properties to change the broker URL to point to the first federated node 5. Start the node1 and node2 brokers 6. Run the consumer.sh in each of the consumers from steps 2 and 3 to create a consumer at each end of the federation 7. Check the broker consoles and verify that each broker shows 2 consumers for TEST.Q.0 - one local and one from the opposite node 8. Start the producer by running the producer.sh script and let it finish. It should produce 5000 messages. 9. As the consumers slowly work off the load, periodically grep the logs for the number of deliveries made: cat log/artemis.log | grep delivered\ message | wc -l 10. At first one broker shows 5000 and the other a slowly increasing count. Let it continue. After a few minutes, you will see the count on the second node jump as messages are pushed back to the first node. After a few more minutes, the count on both nodes starts jumping rapidly, hitting in the 10s of thousands and then millions. These numbers can also be verified by looking at the added and acked counts on the queue (TEST.Q.0).
    • Customer Facing

    Description

      In a two-broker symmetric queue federation setup, we have found that there is no way to limit the number of hops a message can take. So for example, if the local application consumers are all very slow or having some issues that prevents them from consuming, then a message can be sent back-and-forth over a federated connection a large number of times before it is finally consumed by an application consumer.

      Attachments

        Issue Links

          Activity

            People

              gtully@redhat.com Gary Tully
              Asouza@redhat.com Angelo Souza
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: