Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-10131

[Artemis 2.x upgrade] Missing message JMS bridge with DUPS_OK quality of service

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • None
    • JMS
    • Hide

      Steps to reproduce (not 100% reproducer):

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/273//artifact/jboss-eap.zip PrepareServers7.groovy

export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=DedicatedFailoverCoreBridges#testFailoverKillWithBridgeWithStaticConnectors -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1521544306-SNAPSHOT -DfailIfNoTests=false -Deap=7x | tee log
      Show
      Steps to reproduce (not 100% reproducer) : git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/273//artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=DedicatedFailoverCoreBridges#testFailoverKillWithBridgeWithStaticConnectors -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1521544306-SNAPSHOT -DfailIfNoTests= false -Deap=7x | tee log

    Description

      There is missing one message in scenario where JMS bridge (with DUPS_OK) resending messages from one server to another. Durint resending target server is killed and bridge does failover to backup.

      After failover in server logs there are lots of warns/errors:

      05:39:01,697 WARN  [org.apache.activemq.artemis.jms.bridge] (Thread-144) AMQ342009: JMS Bridge N/A failed to send + acknowledge batch, closing JMS objects: javax.jms.JMSException: Duplicate message detected - me
ssage will not be routed. Message information:LargeServerMessage[messageID=2147485500,durable=true,userID=d8ea8e0a-326b-11e8-b74b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:39:01 EDT 2018,expiration=0, dur
able=true, address=jms.queue.OutQueue,size=410263,properties=TypedProperties[__AMQ_CID=bd95a355-326b-11e8-b74b-fa163e64a220,_AMQ_ROUTING_TYPE=1,AMQ_BRIDGE_MSG_ID_LIST=ID:c13a41dc-326b-11e8-949b-fa163e64a220,JMSX
DeliveryCount=10,count=989,color=RED,counter=990,_AMQ_DUPL_ID=fde672e0-e858-4c50-a148-a0b7a4a223591522229901951,_AMQ_LARGE_SIZE=409615]]@1304406481
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:423) [artemis-core-client-2.5.0.jar:2.5.0]
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:319) [artemis-core-client-2.5.0.jar:2.5.0]
        at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.simpleCommit(ActiveMQSessionContext.java:381) [artemis-core-client-2.5.0.jar:2.5.0]
        at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:862) [artemis-core-client-2.5.0.jar:2.5.0]
        at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:835) [artemis-core-client-2.5.0.jar:2.5.0]
        at org.apache.activemq.artemis.jms.client.ActiveMQSession.commit(ActiveMQSession.java:231) [artemis-jms-client-2.5.0.jar:2.5.0]
        at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatchNonTransacted(JMSBridgeImpl.java:1338) [artemis-jms-server-2.5.0.jar:2.5.0]
        at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatch(JMSBridgeImpl.java:1298) [artemis-jms-server-2.5.0.jar:2.5.0]
        at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.access$1700(JMSBridgeImpl.java:74) [artemis-jms-server-2.5.0.jar:2.5.0]
        at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl$SourceReceiver.run(JMSBridgeImpl.java:1694) [artemis-jms-server-2.5.0.jar:2.5.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
Caused by: ActiveMQDuplicateIdException[errorType=DUPLICATE_ID_REJECTED message=Duplicate message detected - message will not be routed. Message information:LargeServerMessage[messageID=2147485500,durable=true,u
serID=d8ea8e0a-326b-11e8-b74b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:39:01 EDT 2018,expiration=0, durable=true, address=jms.queue.OutQueue,size=410263,properties=TypedProperties[__AMQ_CID=bd95a355-326b
-11e8-b74b-fa163e64a220,_AMQ_ROUTING_TYPE=1,AMQ_BRIDGE_MSG_ID_LIST=ID:c13a41dc-326b-11e8-949b-fa163e64a220,JMSXDeliveryCount=10,count=989,color=RED,counter=990,_AMQ_DUPL_ID=fde672e0-e858-4c50-a148-a0b7a4a2235915
22229901951,_AMQ_LARGE_SIZE=409615]]@1304406481]
        ... 13 more

05:39:01,699 WARN  [org.apache.activemq.artemis.core.server] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@2076a02)) AMQ222149: Message Reference[3056]:RELIABLE:Co
reMessage[messageID=3056,durable=true,userID=c1347573-326b-11e8-949b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:38:21 EDT 2018,expiration=0, durable=true, address=jms.queue.InQueue,size=10673,properties=Ty
pedProperties[__AMQ_CID=be528a5c-326b-11e8-949b-fa163e64a220,_AMQ_ROUTING_TYPE=1,count=980,counter=981,_AMQ_DUPL_ID=13adf1ec-581f-42e4-835b-0e6e33a4e1191522229901913,color=GREEN]]@511432790 has reached maximum d
elivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.InQueue
05:39:01,700 WARN  [org.apache.activemq.artemis.core.server] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@2076a02)) AMQ222149: Message Reference[3058]:RELIABLE:CoreMessage[messageID=3058,durable=true,userID=c1349c84-326b-11e8-949b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:38:21 EDT 2018,expiration=0, durable=true, address=jms.queue.InQueue,size=20914,properties=TypedProperties[__AMQ_CID=be528a5c-326b-11e8-949b-fa163e64a220,_AMQ_ROUTING_TYPE=1,count=981,counter=982,_AMQ_DUPL_ID=6feb66cd-8616-4dff-8775-3f85a4b9bdbb1522229901914,color=RED]]@1150646585 has reached maximum delivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.InQueue
....
05:39:01,706 WARN  [org.apache.activemq.artemis.core.server] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@2076a02)) AMQ222149: Message Reference[3082]:RELIABLE:CoreMessage[messageID=3082,durable=true,userID=c13d4f1d-326b-11e8-949b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:38:21 EDT 2018,expiration=0, durable=true, address=jms.queue.InQueue,size=10673,properties=TypedProperties[__AMQ_CID=be528a5c-326b-11e8-949b-fa163e64a220,_AMQ_ROUTING_TYPE=1,count=990,counter=991,_AMQ_DUPL_ID=f802f4b7-e0a1-4dee-acc7-0b11963bec401522229901971,color=GREEN]]@2708478 has reached maximum delivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.InQueue

      So the missing message ended up in DLQ as it could not be delivered to target queue. It looks like that _AMQ_DUPL_ID might trigger dup message detection in target queueu however such message did not get there.

      Attachments

        Issue Links

          blocks

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. WFLY-10320 Upgrade artemis from 1.5.x to 2.x.x

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-1183 Missing message JMS bridge with DUPS_OK quality of service

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              mtaylor1@redhat.com Martyn Taylor (Inactive)
              mnovak1@redhat.com Miroslav Novak
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: