Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-11059

Clients do not failover to backup even it if backup started and keep retrying connection

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • 14.0.0.Final
    • JMS
    • None
    • Hide

      Issue is intermittent - not 100% reproducer:

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout eap7.2.x-artemis2.x-ha-jdbc
groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/362/artifact/jboss-eap.zip PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=NettyColocatedClusterFailoverTestCase#testFailbackClientAckQueueNIO -Deap7.clients.version=7.1531981138-SNAPSHOT | tee log
      Show
      Issue is intermittent - not 100% reproducer : git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout eap7.2.x-artemis2.x-ha-jdbc groovy -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/362/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=NettyColocatedClusterFailoverTestCase#testFailbackClientAckQueueNIO -Deap7.clients.version=7.1531981138-SNAPSHOT | tee log

      Test Scenario:

      • Start 2 servers in colocated topology with shared store
      • Start procuder and consumer sending/receiving messages to/from queue on 1st server(to live)
      • Kill 1st server and wait for backup on 2nd server to start
      • Clients failover to backup to 2nd server and wait for them to finish
      • Check that number of sent/received messages is the same

      Result:
      Sometimes happens that clients do not failover to backup to 2nd server and retry connection. I can see that backup on 2nd server started and created cluster with colocated live server but clients did not connect to it.

      Clients are retrying connection:

      07:32:16,676 Thread-36 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:794] Trying reconnection attempt 163/-1
07:32:16,676 Thread-39 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:446] Connector NettyConnector [host=rhel7-large-42748, port=5445, httpEnabled=fa
lse, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using native epoll
07:32:16,676 Thread-36 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1080] Trying to connect with connectorFactory = org.apache.activemq.artemis.co
re.remoting.impl.netty.NettyConnectorFactory@6568a014, connectorConfig=TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&useNio=true
&host=rhel7-large-42748&useNioGlobalWorkerPool=true&blockOnNonDurableSend=false&retryIntervalMultiplier=1-0&maxRetryInterval=2000&producerMaxRate=-1&blockOnDurableSend=true&connectionTTL=60000&compressLargeMessa
ge=false&reconnectAttempts=-1&ignoreJTA=false&cacheLargeMessagesClient=false&scheduledThreadPoolMaxSize=5&useGlobalPools=true&callFailoverTimeout=60000&initialConnectAttempts=1&clientFailureCheckPeriod=30000&blo
ckOnAcknowledge=true&consumerWindowSize=1048576&minLargeMessageSize=102400&finalizeChecks=false&autoGroup=false&threadPoolMaxSize=30&confirmationWindowSize=-1&transactionBatchSize=1048576&callTimeout=30000&preAc
knowledge=false&enable1xPrefixes=true&cacheDestinations=false&connectionLoadBalancingPolicyClassName=org-apache-activemq-artemis-api-core-client-loadbalance-RoundRobinConnectionLoadBalancingPolicy&dupsOKBatchSiz
e=1048576&incomingInterceptorList=&initialMessagePacketSize=1500&consumerMaxRate=-1&enableSharedClientID=true&HA=true&retryInterval=2000&factoryType=0&failoverOnInitialConnection=false&outgoingInterceptorList=&p
roducerWindowSize=65536
07:32:16,676 Thread-39 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:600] AMQ211002: Started EPOLL Netty Connector version 4.1.16.Final to rhel7-larg
e-42748:5445
07:32:16,676 Thread-36 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:446] Connector NettyConnector [host=rhel7-large-42748, port=5445, httpEnabled=fa
lse, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using native epoll
07:32:16,676 Thread-38 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:794] Trying reconnection attempt 163/-1
07:32:16,676 Thread-38 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1080] Trying to connect with connectorFactory = org.apache.activemq.artemis.co
re.remoting.impl.netty.NettyConnectorFactory@43d7f9fb, connectorConfig=TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&useNio=true
&host=rhel7-large-42748&useNioGlobalWorkerPool=true&blockOnNonDurableSend=false&retryIntervalMultiplier=1-0&maxRetryInterval=2000&producerMaxRate=-1&blockOnDurableSend=true&connectionTTL=60000&compressLargeMessa
ge=false&reconnectAttempts=-1&ignoreJTA=false&cacheLargeMessagesClient=false&scheduledThreadPoolMaxSize=5&useGlobalPools=true&callFailoverTimeout=60000&initialConnectAttempts=1&clientFailureCheckPeriod=30000&blo
ckOnAcknowledge=true&consumerWindowSize=1048576&minLargeMessageSize=102400&finalizeChecks=false&autoGroup=false&threadPoolMaxSize=30&confirmationWindowSize=-1&transactionBatchSize=1048576&callTimeout=30000&preAc
knowledge=false&enable1xPrefixes=true&cacheDestinations=false&connectionLoadBalancingPolicyClassName=org-apache-activemq-artemis-api-core-client-loadbalance-RoundRobinConnectionLoadBalancingPolicy&dupsOKBatchSiz
e=1048576&incomingInterceptorList=&initialMessagePacketSize=1500&consumerMaxRate=-1&enableSharedClientID=true&HA=true&retryInterval=2000&factoryType=0&failoverOnInitialConnection=false&outgoingInterceptorList=&p
roducerWindowSize=65536

              mtaylor1@redhat.com Martyn Taylor (Inactive)
              mnovak1@redhat.com Miroslav Novak
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: