Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-10131

[Artemis 2.x upgrade] Missing message JMS bridge with DUPS_OK quality of service

    Details

    • Steps to Reproduce:
      Hide

      Steps to reproduce (not 100% reproducer):

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
      cd eap-tests-hornetq/scripts/
      groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/273//artifact/jboss-eap.zip PrepareServers7.groovy
      
      export WORKSPACE=$PWD
      export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
      export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
      export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
      export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
      cd ../jboss-hornetq-testsuite/
      
      mvn clean test -Dtest=DedicatedFailoverCoreBridges#testFailoverKillWithBridgeWithStaticConnectors -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1521544306-SNAPSHOT -DfailIfNoTests=false -Deap=7x | tee log
      
      Show
      Steps to reproduce (not 100% reproducer) : git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/273//artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=DedicatedFailoverCoreBridges#testFailoverKillWithBridgeWithStaticConnectors -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1521544306-SNAPSHOT -DfailIfNoTests= false -Deap=7x | tee log

      Description

      There is missing one message in scenario where JMS bridge (with DUPS_OK) resending messages from one server to another. Durint resending target server is killed and bridge does failover to backup.

      After failover in server logs there are lots of warns/errors:

      05:39:01,697 WARN  [org.apache.activemq.artemis.jms.bridge] (Thread-144) AMQ342009: JMS Bridge N/A failed to send + acknowledge batch, closing JMS objects: javax.jms.JMSException: Duplicate message detected - me
      ssage will not be routed. Message information:LargeServerMessage[messageID=2147485500,durable=true,userID=d8ea8e0a-326b-11e8-b74b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:39:01 EDT 2018,expiration=0, dur
      able=true, address=jms.queue.OutQueue,size=410263,properties=TypedProperties[__AMQ_CID=bd95a355-326b-11e8-b74b-fa163e64a220,_AMQ_ROUTING_TYPE=1,AMQ_BRIDGE_MSG_ID_LIST=ID:c13a41dc-326b-11e8-949b-fa163e64a220,JMSX
      DeliveryCount=10,count=989,color=RED,counter=990,_AMQ_DUPL_ID=fde672e0-e858-4c50-a148-a0b7a4a223591522229901951,_AMQ_LARGE_SIZE=409615]]@1304406481
              at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:423) [artemis-core-client-2.5.0.jar:2.5.0]
              at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:319) [artemis-core-client-2.5.0.jar:2.5.0]
              at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.simpleCommit(ActiveMQSessionContext.java:381) [artemis-core-client-2.5.0.jar:2.5.0]
              at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:862) [artemis-core-client-2.5.0.jar:2.5.0]
              at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:835) [artemis-core-client-2.5.0.jar:2.5.0]
              at org.apache.activemq.artemis.jms.client.ActiveMQSession.commit(ActiveMQSession.java:231) [artemis-jms-client-2.5.0.jar:2.5.0]
              at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatchNonTransacted(JMSBridgeImpl.java:1338) [artemis-jms-server-2.5.0.jar:2.5.0]
              at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatch(JMSBridgeImpl.java:1298) [artemis-jms-server-2.5.0.jar:2.5.0]
              at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.access$1700(JMSBridgeImpl.java:74) [artemis-jms-server-2.5.0.jar:2.5.0]
              at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl$SourceReceiver.run(JMSBridgeImpl.java:1694) [artemis-jms-server-2.5.0.jar:2.5.0]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
              at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
      Caused by: ActiveMQDuplicateIdException[errorType=DUPLICATE_ID_REJECTED message=Duplicate message detected - message will not be routed. Message information:LargeServerMessage[messageID=2147485500,durable=true,u
      serID=d8ea8e0a-326b-11e8-b74b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:39:01 EDT 2018,expiration=0, durable=true, address=jms.queue.OutQueue,size=410263,properties=TypedProperties[__AMQ_CID=bd95a355-326b
      -11e8-b74b-fa163e64a220,_AMQ_ROUTING_TYPE=1,AMQ_BRIDGE_MSG_ID_LIST=ID:c13a41dc-326b-11e8-949b-fa163e64a220,JMSXDeliveryCount=10,count=989,color=RED,counter=990,_AMQ_DUPL_ID=fde672e0-e858-4c50-a148-a0b7a4a2235915
      22229901951,_AMQ_LARGE_SIZE=409615]]@1304406481]
              ... 13 more
      
      05:39:01,699 WARN  [org.apache.activemq.artemis.core.server] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@2076a02)) AMQ222149: Message Reference[3056]:RELIABLE:Co
      reMessage[messageID=3056,durable=true,userID=c1347573-326b-11e8-949b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:38:21 EDT 2018,expiration=0, durable=true, address=jms.queue.InQueue,size=10673,properties=Ty
      pedProperties[__AMQ_CID=be528a5c-326b-11e8-949b-fa163e64a220,_AMQ_ROUTING_TYPE=1,count=980,counter=981,_AMQ_DUPL_ID=13adf1ec-581f-42e4-835b-0e6e33a4e1191522229901913,color=GREEN]]@511432790 has reached maximum d
      elivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.InQueue
      05:39:01,700 WARN  [org.apache.activemq.artemis.core.server] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@2076a02)) AMQ222149: Message Reference[3058]:RELIABLE:CoreMessage[messageID=3058,durable=true,userID=c1349c84-326b-11e8-949b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:38:21 EDT 2018,expiration=0, durable=true, address=jms.queue.InQueue,size=20914,properties=TypedProperties[__AMQ_CID=be528a5c-326b-11e8-949b-fa163e64a220,_AMQ_ROUTING_TYPE=1,count=981,counter=982,_AMQ_DUPL_ID=6feb66cd-8616-4dff-8775-3f85a4b9bdbb1522229901914,color=RED]]@1150646585 has reached maximum delivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.InQueue
      ....
      05:39:01,706 WARN  [org.apache.activemq.artemis.core.server] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@2076a02)) AMQ222149: Message Reference[3082]:RELIABLE:CoreMessage[messageID=3082,durable=true,userID=c13d4f1d-326b-11e8-949b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:38:21 EDT 2018,expiration=0, durable=true, address=jms.queue.InQueue,size=10673,properties=TypedProperties[__AMQ_CID=be528a5c-326b-11e8-949b-fa163e64a220,_AMQ_ROUTING_TYPE=1,count=990,counter=991,_AMQ_DUPL_ID=f802f4b7-e0a1-4dee-acc7-0b11963bec401522229901971,color=GREEN]]@2708478 has reached maximum delivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.InQueue
      

      So the missing message ended up in DLQ as it could not be delivered to target queue. It looks like that _AMQ_DUPL_ID might trigger dup message detection in target queueu however such message did not get there.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  martyn-taylor Martyn Taylor
                  Reporter:
                  mnovak Miroslav Novak
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: