Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-7463

AMQP Large Messages in Clustering Broker setup can fail to be consumed

XMLWordPrintable

      tl;dr of what happens:

      I have an openshift-based setup of Clustered AMQ Broker

      cli-qpid-sender is executed to send message to broker #1

      then with small pause (less than 5-10 seconds) cli-qpid-receiver is executed to receive messages from broker #0.

      In this case, what is described in code block below is manifested, with a long wait period (about ~60+seconds)

       

      In case when pause between executions of sender/receive is >10-20 seconds, cli-qpid-sender is able to receive the message just fine in under 5 seconds.

       

      There are 2 problems with this:

      1) timeout time is way too long. (that probably is on cli-java side, given the error message).

      2) error message is uninformative - it just states that client failed to connect to broker, not that target queue on broker is empty (which might happen due to synchronization issue between 2 broker instances).

       

      Log from cli-java client image:

       

      sh-5.1$ cli-qpid-sender --broker-uri amqp://ex-aao-amqp-1-svc.mkr-large.svc.cluster.local:5672 --address /second --msg-content-from-file=test.img
sh-5.1$ cli-qpid-receiver --broker amqp://ex-aao-amqp-0-svc.mkr-large.svc.cluster.local:5672 --address /second --log-msgs json > 2Exception while consuming message!
org.apache.qpid.jms.JmsOperationTimedOutException: Remote did not respond to a drain request in time
        at org.apache.qpid.jms.provider.exceptions.ProviderOperationTimedOutException.toJMSException(ProviderOperationTimedOutException.java:39)
        at org.apache.qpid.jms.provider.exceptions.ProviderOperationTimedOutException.toJMSException(ProviderOperationTimedOutException.java:25)
        at org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:80)
        at org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:112)
        at org.apache.qpid.jms.JmsConnection.pull(JmsConnection.java:915)
        at org.apache.qpid.jms.JmsConnection.pull(JmsConnection.java:899)
        at org.apache.qpid.jms.JmsMessageConsumer.performPullIfRequired(JmsMessageConsumer.java:732)
        at org.apache.qpid.jms.JmsMessageConsumer.dequeue(JmsMessageConsumer.java:332)
        at org.apache.qpid.jms.JmsMessageConsumer.receive(JmsMessageConsumer.java:213)
        at com.redhat.mqe.lib.ReceiverClient.consumeMessage(ReceiverClient.java:209)
        at com.redhat.mqe.lib.ReceiverClient.startClient(ReceiverClient.java:149)
        at com.redhat.mqe.lib.Main.main(Main.java:46)
        at com.redhat.mqe.jms.Main.main(Main.java:80)
        at com.redhat.mqe.jms.Main.main(Main.java:84)
Caused by: org.apache.qpid.jms.provider.exceptions.ProviderOperationTimedOutException: Remote did not respond to a drain request in time
        at org.apache.qpid.jms.provider.amqp.AmqpConsumer.lambda$stop$1(AmqpConsumer.java:184)
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:829)

              csuconic@redhat.com Clebert Suconic
              mkrutov Mikhail Krutov
              Mikhail Krutov Mikhail Krutov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: