Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-2068

some messages received but not delivered during HA fail-over, fail-back scenario

    XMLWordPrintable

Details

    • Hide

      reproducer available

      Show
      reproducer available
    • Release Notes
    • Hide

      Use either an AMQP client or the Core JMS Client

      Show
      Use either an AMQP client or the Core JMS Client
    • Hide
      Currently, if a broker fails over to its slave while an OpenWire client is sending messages, messages being delivered to the broker when failover occurs could be lost. To work around this issue, ensure that the broker persists the messages before acknowledging them.
      Show
      Currently, if a broker fails over to its slave while an OpenWire client is sending messages, messages being delivered to the broker when failover occurs could be lost. To work around this issue, ensure that the broker persists the messages before acknowledging them.
    • Documented as Known Issue

    Description

      In a high througput producer/consumer scenario, replicated HA configuration, a small number of messages are lost during fail-over and subsequent fail-back. The messages are received by the broker but never delivered to the consumer. Sample "lost" message trace (queue names have been scrubbed). These logs are all from the slave, the master never sees the message:

      2018-10-19 20:10:11,553 DEBUG [org.apache.activemq.artemis.core.server.plugin.impl] AMQ843008: beforeSend called with message: CoreMessage[messageID=-1,durable=true,userID=null,priority=4, timestamp=Fri Oct 19 20:10:11 UTC 2018,expiration=0, durable=true, address=q.test,size=14882,properties=TypedProperties[__AMQ_CID=ID:localhost.localdomain-45939-1539979753396-0:3,__HDR_BROKER_IN_TIME=1539979811553,_AMQ_ROUTING_TYPE=1,__HDR_ARRIVAL=0,__HDR_GROUP_SEQUENCE=0,__HDR_COMMAND_ID=493,__HDR_PRODUCER_ID=[0000 0044 7B01 0030 4944 3A6C 6F63 616C 686F 7374 2E6C 6F63 616C 646F 6D61  ...  3939 3739 3735 3333 3936 2D31 3A33 0000 0000 0000 0001 0000 0000 0000 0002),_AMQ_DUPL_ID=ID:localhost.localdomain-45939-1539979753396-1:3:2:1:233,__HDR_MESSAGE_ID=[0000 0057 6E00 017B 0100 3049 443A 6C6F 6361 6C68 6F73 742E 6C6F 6361 6C64  ... 00 0000 0000 0100 0000 0000 0000 0200 0000 0000 0000 E900 0000 0000 0000 00),__HDR_DROPPABLE=false]]@1604091705, tx: null, session: ServerSessionImpl(), direct: false, noAutoCreateQueue: false
      
      2018-10-19 20:10:11,553 DEBUG [org.apache.activemq.artemis.core.server.plugin.impl] AMQ843010: beforeMessageRoute called with message: CoreMessage[messageID=2147615336,durable=true,userID=null,priority=4, timestamp=Fri Oct 19 20:10:11 UTC 2018,expiration=0, durable=true, address=q.test,size=14882,properties=TypedProperties[__AMQ_CID=ID:localhost.localdomain-45939-1539979753396-0:3,__HDR_BROKER_IN_TIME=1539979811553,_AMQ_ROUTING_TYPE=1,__HDR_ARRIVAL=0,__HDR_GROUP_SEQUENCE=0,__HDR_COMMAND_ID=493,__HDR_PRODUCER_ID=[0000 0044 7B01 0030 4944 3A6C 6F63 616C 686F 7374 2E6C 6F63 616C 646F 6D61  ...  3939 3739 3735 3333 3936 2D31 3A33 0000 0000 0000 0001 0000 0000 0000 0002),_AMQ_DUPL_ID=ID:localhost.localdomain-45939-1539979753396-1:3:2:1:233,__HDR_MESSAGE_ID=[0000 0057 6E00 017B 0100 3049 443A 6C6F 6361 6C68 6F73 742E 6C6F 6361 6C64  ... 00 0000 0000 0100 0000 0000 0000 0200 0000 0000 0000 E900 0000 0000 0000 00),__HDR_DROPPABLE=false]]@1604091705, context: org.apache.activemq.artemis.core.server.impl.RoutingContextImpl@340ab8f0, direct: false, rejectDuplicates: true
      
      2018-10-19 20:10:11,553 DEBUG [org.apache.activemq.artemis.core.server.plugin.impl] AMQ843011: afterMessageRoute message: CoreMessage[messageID=2147615336,durable=true,userID=null,priority=4, timestamp=Fri Oct 19 20:10:11 UTC 2018,expiration=0, durable=true, address=q.test,size=14882,properties=TypedProperties[__AMQ_CID=ID:localhost.localdomain-45939-1539979753396-0:3,__HDR_BROKER_IN_TIME=1539979811553,_AMQ_ROUTING_TYPE=1,__HDR_ARRIVAL=0,__HDR_GROUP_SEQUENCE=0,__HDR_COMMAND_ID=493,__HDR_PRODUCER_ID=[0000 0044 7B01 0030 4944 3A6C 6F63 616C 686F 7374 2E6C 6F63 616C 646F 6D61  ...  3939 3739 3735 3333 3936 2D31 3A33 0000 0000 0000 0001 0000 0000 0000 0002),_AMQ_DUPL_ID=ID:localhost.localdomain-45939-1539979753396-1:3:2:1:233,__HDR_MESSAGE_ID=[0000 0057 6E00 017B 0100 3049 443A 6C6F 6361 6C68 6F73 742E 6C6F 6361 6C64  ... 00 0000 0000 0100 0000 0000 0000 0200 0000 0000 0000 E900 0000 0000 0000 00),__HDR_DROPPABLE=false]]@1604091705, with context: org.apache.activemq.artemis.core.server.impl.RoutingContextImpl@340ab8f0, direct: false, rejectDuplicates: true
      
      2018-10-19 20:10:11,553 DEBUG [org.apache.activemq.artemis.core.server.plugin.impl] AMQ843009: message ID: 2147615336, message CoreMessage[messageID=2147615336,durable=true,userID=null,priority=4, timestamp=Fri Oct 19 20:10:11 UTC 2018,expiration=0, durable=true, address=q.test,size=14882,properties=TypedProperties[__AMQ_CID=ID:localhost.localdomain-45939-1539979753396-0:3,__HDR_BROKER_IN_TIME=1539979811553,_AMQ_ROUTING_TYPE=1,__HDR_ARRIVAL=0,__HDR_GROUP_SEQUENCE=0,__HDR_COMMAND_ID=493,__HDR_PRODUCER_ID=[0000 0044 7B01 0030 4944 3A6C 6F63 616C 686F 7374 2E6C 6F63 616C 646F 6D61  ...  3939 3739 3735 3333 3936 2D31 3A33 0000 0000 0000 0001 0000 0000 0000 0002),_AMQ_DUPL_ID=ID:localhost.localdomain-45939-1539979753396-1:3:2:1:233,__HDR_MESSAGE_ID=[0000 0057 6E00 017B 0100 3049 443A 6C6F 6361 6C68 6F73 742E 6C6F 6361 6C64  ... 00 0000 0000 0100 0000 0000 0000 0200 0000 0000 0000 E900 0000 0000 0000 00),__HDR_DROPPABLE=false]]@1604091705, session name: ID:localhost.localdomain-45939-1539979753396-1:3:2 with tx: null, session: ServerSessionImpl(), direct: false, noAutoCreateQueue: false
      

      This is the last log for this message; other messages in the test are successfully delivered.

      Attachments

        Issue Links

          Activity

            People

              ataylor@redhat.com Andy Taylor
              rhn-support-shiggs Stephen Higgs
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: