Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-1583

Some messages cannot be received in network of brokers

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • JBoss A-MQ 6.3
    • JBoss A-MQ 6.2.1
    • broker, fabric8
    • None
    • Hide
      1. setup three networked broker within fabric 
        fabric:create --wait-for-provisioning
container-create-child root broker 3
mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker1 broker1
mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker2 broker2
mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker3 broker3
      2. run attached testcase 
        mvn clean test -Dclient.threads=125 -Dmessage.count=1000000 -Ddestination.name="mesh.3.failover.1m" -Dbroker.url="failover:(tcp://host1:42270,tcp://host1:37617,tcp://host1:46291)?randomize=true"

      I am able to reproduce the issue only with large number of threads/connections (I can reliably reproduce with 125 consumer and 125 producer threads and 1000000 messages).

      Show
      setup three networked broker within fabric fabric:create --wait- for -provisioning container-create-child root broker 3 mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker1 broker1 mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker2 broker2 mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker3 broker3 run attached testcase mvn clean test -Dclient.threads=125 -Dmessage.count=1000000 -Ddestination.name= "mesh.3.failover.1m" -Dbroker.url= "failover:(tcp: //host1:42270,tcp://host1:37617,tcp://host1:46291)?randomize= true " I am able to reproduce the issue only with large number of threads/connections (I can reliably reproduce with 125 consumer and 125 producer threads and 1000000 messages).

      There are three child containers in fabric8 with brokers. All brokers are in the same group. The brokers are set up to form network of brokers (brokers are connected to each other).

      Then I run test case which attaches producers in consumers in following way:

      • each consumer/producer has its own connection
      • each consumer/producer disconnects after receiving 100 messages and connects again (to random broker)
      • Sessions are not transacted ( AUTO_ACKNOWLEDGE is used)
      • the producers send large number of messages

      The test case counts number of sent and received messages in total. When the test finishes the number of message received is not equal to number of messages sent. The number of messagess received is usually less than number of sent messages

      The queues show no messages:

      dstat output on the brokers
      broker1:
Name                                                Queue Size  Producer #  Consumer #   Enqueue #   Dequeue #   Forward #
ActiveMQ.DLQ                                                 1           0           0           1           0           0
mesh.3.failover.1m                                           0           0           0      362098      362098       17347
broker2:
Name                                                Queue Size  Producer #  Consumer #   Enqueue #   Dequeue #   Forward #
mesh.3.failover.1m                                           0           0           0      383995      383995       16195
broker3:
Name                                                Queue Size  Producer #  Consumer #   Enqueue #   Dequeue #   Forward #
ActiveMQ.DLQ                                                 1           0           0           1           0           0
mesh.3.failover.1m                                           0           0           0      302459      302459       15045

      but when I try browse mesh.3.failover.1m queue using hawtio I can see messages there. But these messages cannot be consumed.

      The following warning appears in the log of brokers multiple times:

      Warning in the logs
      2016-03-16 13:02:22,391 | WARN  | .125:42829@35073 | AbstractStoreCursor              | 109 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-621090 | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@22d82c12:mesh.3.failover.1m,batchResetNeeded=false,size=0,cacheEnabled=true,maxBatchSize:1,hasSpace:true,pendingCachedIds.size:553,lastSyncCachedId:ID:soaperf22.mw.lab.eng.bos.redhat.com-60962-1458147522872-1:2283:1:1:49,lastSyncCachedId-seq:4969,lastAsyncCachedId:ID:soaperf22.mw.lab.eng.bos.redhat.com-60962-1458147522872-1:2283:1:1:49,lastAsyncCachedId-seq:4969,store=permits:9446,sd=nextSeq:4970,lastRet:MessageOrderCursor:[def:4968, low:4966, high:4966],pending:0 - cursor got duplicate send ID:soaperf22.mw.lab.eng.bos.redhat.com-60962-1458147522872-1:2529:1:1:99 seq: org.apache.activemq.store.kahadb.KahaDBStore$StoreQueueTask$InnerFutureTask@1ee69e9a

        1. 171-failover-less-messages.tar.gz
          76 kB
        2. broker-1.log
          164 kB
        3. broker-2.log
          148 kB
        4. broker-3.log
          158 kB
        5. mesh-client-reproducer.tar.gz
          4 kB

              gtully@redhat.com Gary Tully
              knetl.j@gmail.com Jakub Knetl (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: