Inconsistent and negative address size

Cause-Consequence-Fix-Result ?

Messages are reference counted in Artemis and the first binding to cause a message to get stored increments the reference count and tracks the message size on its pageStore and on the address message counters.

This is problematic when there are durable subscribers or cluster connections and the order of acknowledgement can be different than the dispatch order. If the binding and page store that tracked the message size on first use is not the last binding to acknowledge the message some other binding and page store gets to discount the message size in error.

The consequence was pageStores reporting negative message size bytes usage and potentially delaying the onset of subsequent paging.

The fix is to always use the page store corresponding to the target address, not the binding, when accounting for the message size increment and decrement of the page store counters. Each subsequent reference increment would just track 64bytes for a java message reference.

One implication that can be counter intuitive, in a case where there is only a store and forward queue from a cluster connection on a broker, paging will only kick in when the global-max-size bytes is reached, because it is only the global-max-size that can combine the message size on the paging store with the references on the store and forward queue binding.

If only a individual page store max-size-bytes is configured for the target address, it won't ever be triggered b/c the messages are only stored on the store and forward queue in this scenario.

Set up a fully-connected mesh with two brokers (they can be on the same host). I will refer to the brokers as Broker 1 and Broker 2.

Establish a durable consumer on address t1, on Broker 1. using AMQP

Establish a durable consumer on address t1, on Broker 2, with a different client ID, using AMQP

Publish messages to address t1, on either broker, using AMQP. Messages will be received by both subscribers, as expected, but the warning message will be shown on the broker that first received the message.

                       +-----------+
                       |           |
                       |  Broker 1 |
Publisher on t1 -----> |  Topic t1 |-------> Durable subscriber on t1, client ID 'a'
                       |           |
                       +-----------+
                           ^   |
                           |   |
                           |   v
                       +-----------+
                       |           |
                       |  Broker 2 |
Publisher on t1 ---->  |  Topic t1 |-------> Durable subscriber on t1, client ID 'b'
                       |           |
                       +-----------+