Uploaded image for project: 'A-MQ Broker'
  1. A-MQ Broker
  2. ENTMQBR-4022

[LTS] Temporary Queue Leak With OpenWire Request-Reply Clients

    Details

    • Target Release:
    • Steps to Reproduce:
      Hide

      Reproducer attached:

      1. Configure an AMQ 7.7.0 broker with default settings
      2. Build and run the response-consumer (this is the consumer for the producer that emits response messages) - configure application.properties for the correct broker-url. Run using mvn spring-boot:run.
      3. Build and run the replyto-producer (emits the original messages and creates the reply consumer) - configure application.properties for the correct broker-url. Run using mvn spring-boot:run.
      4. Allow the applications to run for 10 or so minutes
      5. Restart the replyto-producer
      6. If no issue is observed, let the apps run for another 10 or so minutes - repeat until the WARN messages are observed in the response-consumer log
      7. Take a heap dump of the broker for examination:

      SELECT toString(tq.physicalName) FROM org.apache.activemq.command.ActiveMQTempQueue tq

      Show
      Reproducer attached: 1. Configure an AMQ 7.7.0 broker with default settings 2. Build and run the response-consumer (this is the consumer for the producer that emits response messages) - configure application.properties for the correct broker-url. Run using mvn spring-boot:run . 3. Build and run the replyto-producer (emits the original messages and creates the reply consumer) - configure application.properties for the correct broker-url. Run using mvn spring-boot:run . 4. Allow the applications to run for 10 or so minutes 5. Restart the replyto-producer 6. If no issue is observed, let the apps run for another 10 or so minutes - repeat until the WARN messages are observed in the response-consumer log 7. Take a heap dump of the broker for examination: SELECT toString(tq.physicalName) FROM org.apache.activemq.command.ActiveMQTempQueue tq

      Description

      When using the camel-activiemq component with an InOut exchange pattern and the default settings (use a temporary queue for replyTo address), a build-up of temporary reply queues with the same name occurs.

      When the InOut producers is restarted, I can see orphan temporary queues corresponding to the original producer client ID remain stranded in the broker heap left behind.

      Sometimes when this occurs the consumer trying to respond to the producer begins emitting WARN messages like the following that do not stop until the broker or the replying consumer are restarted:

      18:32:34.409 [Camel (camel-1) thread #2 - JmsConsumer[replyto-producer-v1]] WARN  o.a.c.c.jms.EndpointMessageListener - Execution of JMS message listener failed. Caused by: [org.apache.camel.RuntimeCamelException - org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1]
      org.apache.camel.RuntimeCamelException: org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1
      	at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1826)
      	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:134)
      	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)
      	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)
      	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:testremote.csb-43698-1600554750891-1:1:1
      	at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:280)
      	at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:185)
      	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:507)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:556)
      	at org.apache.camel.component.jms.EndpointMessageListener.sendReply(EndpointMessageListener.java:370)
      	at org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback.done(EndpointMessageListener.java:223)
      	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:117)
      	... 11 common frames omitted
      Caused by: javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:test.remote.csb-43698-1600554750891-1:1:1
      	at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1911)
      	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:288)
      	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:223)
      	at org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
      	at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:634)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSend(JmsConfiguration.java:634)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:573)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.access$100(JmsConfiguration.java:515)
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate$4.doInJms(JmsConfiguration.java:558)
      	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:504)
      	... 15 common frames omitted
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  jbertram Justin Bertram
                  Reporter:
                  hawkinsds Duane Hawkins
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated: