Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-4022

[LTS] Temporary Queue Leak With OpenWire Request-Reply Clients

    XMLWordPrintable

    Details

    • Target Release:
    • Steps to Reproduce:
      Hide

      Reproducer attached:

      1. Configure an AMQ 7.7.0 broker with default settings
      2. Build and run the response-consumer (this is the consumer for the producer that emits response messages) - configure application.properties for the correct broker-url. Run using mvn spring-boot:run.
      3. Build and run the replyto-producer (emits the original messages and creates the reply consumer) - configure application.properties for the correct broker-url. Run using mvn spring-boot:run.
      4. Allow the applications to run for 10 or so minutes
      5. Restart the replyto-producer
      6. If no issue is observed, let the apps run for another 10 or so minutes - repeat until the WARN messages are observed in the response-consumer log
      7. Take a heap dump of the broker for examination:

      SELECT toString(tq.physicalName) FROM org.apache.activemq.command.ActiveMQTempQueue tq

      Show
      Reproducer attached: 1. Configure an AMQ 7.7.0 broker with default settings 2. Build and run the response-consumer (this is the consumer for the producer that emits response messages) - configure application.properties for the correct broker-url. Run using mvn spring-boot:run . 3. Build and run the replyto-producer (emits the original messages and creates the reply consumer) - configure application.properties for the correct broker-url. Run using mvn spring-boot:run . 4. Allow the applications to run for 10 or so minutes 5. Restart the replyto-producer 6. If no issue is observed, let the apps run for another 10 or so minutes - repeat until the WARN messages are observed in the response-consumer log 7. Take a heap dump of the broker for examination: SELECT toString(tq.physicalName) FROM org.apache.activemq.command.ActiveMQTempQueue tq
    • Release Notes Text:
      Hide
      Cause: When using OpenWire and creating/closing multiple different producers on the same temporary queue the broker will store (i.e. duplicate) the same temporary queue in its internal data structures., Consequence: Duplicate objects will accumulate in heap memory., Fix: Detect duplicate temporary queues and don't store them., Result: Duplicate objects no longer accumulate in heap memory.
      Show
      Cause: When using OpenWire and creating/closing multiple different producers on the same temporary queue the broker will store (i.e. duplicate) the same temporary queue in its internal data structures., Consequence: Duplicate objects will accumulate in heap memory., Fix: Detect duplicate temporary queues and don't store them., Result: Duplicate objects no longer accumulate in heap memory.
    • QE Test Coverage:
      +
    • Verified:
      Verified in a release

      Description

      When using the camel-activiemq component with an InOut exchange pattern and the default settings (use a temporary queue for replyTo address), a build-up of temporary reply queues with the same name occurs.

      When the InOut producers is restarted, I can see orphan temporary queues corresponding to the original producer client ID remain stranded in the broker heap left behind.

      Sometimes when this occurs the consumer trying to respond to the producer begins emitting WARN messages like the following that do not stop until the broker or the replying consumer are restarted:

      18:32:34.409 [Camel (camel-1) thread #2 - JmsConsumer[replyto-producer-v1]] WARN  o.a.c.c.jms.EndpointMessageListener - Execution of JMS message listener failed. Caused by: [org.apache.camel.RuntimeCamelException - org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1]
      org.apache.camel.RuntimeCamelException: org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1
      	at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1826)
      	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:134)
      	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)
      	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)
      	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:testremote.csb-43698-1600554750891-1:1:1
      	at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:280)
      	at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:185)
      	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:507)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:556)
      	at org.apache.camel.component.jms.EndpointMessageListener.sendReply(EndpointMessageListener.java:370)
      	at org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback.done(EndpointMessageListener.java:223)
      	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:117)
      	... 11 common frames omitted
      Caused by: javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1
      	at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1911)
      	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:288)
      	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:223)
      	at org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
      	at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:634)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSend(JmsConfiguration.java:634)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:573)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.access$100(JmsConfiguration.java:515)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate$4.doInJms(JmsConfiguration.java:558)
      	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:504)
      	... 15 common frames omitted
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jbertram Justin Bertram
              Reporter:
              hawkinsds Duane Hawkins
              Tester:
              Tiago Bueno Tiago Bueno
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: