Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3389

NMS / Openwire Client Runs Out of Credits Even though Broker Shows All Messages Acked

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • AMQ 7.6.0.GA, AMQ 7.7.0.CR1
    • None
    • None
    • +
    • Verified in a release
    • Hide

      Reproducer attached.

      Extract and edit jndi.properties to correct the broker URL for host / port and run with included consumer.sh script. Reproducer was created by adding an intentiional delay to the ActiveMQSession clearMessagesInProgress async task to force the consumer to be recreated as a pull consumer and subsequently updated with prefetch.

      I reproduced the issue by installing the client on one host configuring with a prefetch of 10 in the URI, starting the broker on another host (I used 7.4.1), sending 100 or so messages to the TEST.Q address, then interrupting the network connection to force a reconnect.

      To interrupt the connection, I used this script on the broker host:

      #!/bin/bash
        sleep 2
        echo "Interrupting network"
        ip link set eth0 down
        sleep 35
        ip link set eth0 up

      Consumer will recover, process a few (prefech) messages, then will stall. If a heap is taken of the broker we can see that currentWindow is 0, no messages are in delivery and the pullHandler value is non-null.

      Show
      Reproducer attached. Extract and edit jndi.properties to correct the broker URL for host / port and run with included consumer.sh script. Reproducer was created by adding an intentiional delay to the ActiveMQSession clearMessagesInProgress async task to force the consumer to be recreated as a pull consumer and subsequently updated with prefetch. I reproduced the issue by installing the client on one host configuring with a prefetch of 10 in the URI, starting the broker on another host (I used 7.4.1), sending 100 or so messages to the TEST.Q address, then interrupting the network connection to force a reconnect. To interrupt the connection, I used this script on the broker host: #!/bin/bash sleep 2 echo "Interrupting network" ip link set eth0 down sleep 35 ip link set eth0 up Consumer will recover, process a few (prefech) messages, then will stall. If a heap is taken of the broker we can see that currentWindow is 0, no messages are in delivery and the pullHandler value is non-null.

      In an environment using OpenWire NMS clients and broker version 7.4.1, we see that messages get prefetched out to the consumer up to its default limit (1000); however, after the client acks all the messages (and the broker shows 1000 messages acked), the broker still reports the consumer is waiting on credits and reports current credits as NULL:

      2020-02-26 12:44:06,228 DEBUG [org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl] ServerConsumerImpl [id=0, filter=null, binding=LocalQueueBinding [address=XXXX, queue=QueueImpl[name=XXXX, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=739b6b51-f061-11e9-98c9-000f536b0260], temp=false]@376f976c, filter=null, name=XXXX, clusterName=XXXX739b6b51-f061-11e9-98c9-000f536b0260]] is busy for the lack of credits. Current credits = null Can't receive reference Reference[45099938419]:RELIABLE:CoreMessage[messageID=45099938419,durable=true,userID=ace4f982-5895-11ea-ab83-000f536b0260,priority=4, timestamp=Wed Feb 26 12:44:06 GMT 2020,expiration=Wed Feb 26 12:44:16 GMT 2020, durable=true, address=XXXX,size=1996,properties=...

      Looking at a heap from the affected broker, the currentWindow for the consumer is "0" though there are no message references in deliveringRefs.

              dbruscin Domenico Francesco Bruscino
              dbruscin Domenico Francesco Bruscino
              Tiago Bueno Tiago Bueno
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: