Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-8285

Clients are not able to do failback in colocated HA topology with HTTP connectors

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • 7.1.0.DR12
    • 7.1.0.DR10, 7.1.0.DR11
    • JMS
    • None
    • Workaround Exists
    • Hide

      Use remote connectors/acceptors

      Show
      Use remote connectors/acceptors
    • Hide 
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout cc737cb50d1477d292a814cfc920835506e345a0
groovy -DEAP_VERSION=7.1.0.DR11 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

 mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackTransAckTopic -DfailIfNoTests=false -Deap=7x -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1.0.DR11  | tee log

      Tip for debug: Trace logs of client's side are stored in file eap-tests-hornetq/jboss-hornetq-testsuite/common-tests/target/test-suite.log

      Note: This is not 100% reproducer. However I'm able to reproduce the issue locally on my laptop.

      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout cc737cb50d1477d292a814cfc920835506e345a0 groovy -DEAP_VERSION=7.1.0.DR11 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackTransAckTopic -DfailIfNoTests= false -Deap=7x -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1.0.DR11 | tee log Tip for debug: Trace logs of client's side are stored in file eap-tests-hornetq/jboss-hornetq-testsuite/common-tests/target/test-suite.log Note: This is not 100% reproducer. However I'm able to reproduce the issue locally on my laptop.

    Description

      Scenario:

      • There are two EAPs configured in old way colocated HA topology
      • Start both servers
      • Connect clients to the first server
      • Kill first server
      • Wait until clients do failover
      • Start first server
      • Wait until clients do failback

      Description of problem:
      Sometimes happens that after the first server is started again, clients don't do failback. Producers are blocked because they don't have credits for sending. Receivers don't receive any messages.

      In trace logs I can see following messages:

      10:22:52,915 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1
10:22:52,915 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1
10:22:53,241 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@6f7ca1c1, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost
10:22:53,241 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
10:22:53,242 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080
10:22:52,743 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1
10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@25572f4, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost
10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080
10:22:53,242 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET  HTTP/1.1
Host: localhost
Upgrade: activemq-remoting
Connection: Upgrade
activemqServerName: backup
httpUpgradeEndpoint: acceptor-backup
Sec-ActiveMQRemoting-Key: bvTu4ZbJbAqWX5n3TwgNGA==
10:22:53,241 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@1c5a232f, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost
10:22:53,242 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
10:22:53,242 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080
10:22:53,241 main INFO  [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:178] Producer: Thread[Thread-22,5,main] is still alive.
10:22:53,243 main INFO  [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:178] Producer: Thread[Thread-23,5,main] is still alive.
10:22:53,243 main INFO  [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:191] Reciever: Thread[Thread-24,5,] is finished.
10:22:53,243 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET  HTTP/1.1
Host: localhost
Upgrade: activemq-remoting
Connection: Upgrade
activemqServerName: backup
httpUpgradeEndpoint: acceptor-backup
Sec-ActiveMQRemoting-Key: vxNO5uMO4KK6etjAlAtorA==
10:22:53,243 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET  HTTP/1.1
Host: localhost
Upgrade: activemq-remoting
Connection: Upgrade
activemqServerName: backup
httpUpgradeEndpoint: acceptor-backup
Sec-ActiveMQRemoting-Key: aUC+bE6WaMmU1A6KGcssww==
10:22:53,243 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost
10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080
10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
10:22:53,244 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET  HTTP/1.1
Host: localhost
Upgrade: activemq-remoting
Connection: Upgrade
activemqServerName: backup
httpUpgradeEndpoint: acceptor-backup
Sec-ActiveMQRemoting-Key: plpRSFUDmbA3Xky/5BWx6g==
10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost
10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080
10:22:53,244 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost
10:22:53,245 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
10:22:53,245 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080
10:22:53,245 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
10:22:53,245 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET  HTTP/1.1
Host: localhost
Upgrade: activemq-remoting
Connection: Upgrade
activemqServerName: backup
httpUpgradeEndpoint: acceptor-backup
Sec-ActiveMQRemoting-Key: f2nehkPE0VDmGlWAcx3Ltg==
10:22:53,246 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET  HTTP/1.1
Host: localhost
Upgrade: activemq-remoting
Connection: Upgrade
activemqServerName: backup
httpUpgradeEndpoint: acceptor-backup
Sec-ActiveMQRemoting-Key: RIaHZ2gy6JEm9nz6XOdyVA==
10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1104] Backup is not active.
10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1258] Neither backup or live were active, will just give up now
10:22:53,245 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1104] Backup is not active.
10:22:53,246 Thread-2 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1258] Neither backup or live were active, will just give up now
10:22:53,246 Thread-2 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:815] Waiting 2000 milliseconds before next retry. RetryInterval=2000 and multiplier=1.0

      You can notice in trace logs that clients try to reconnect only to Artemis backup instances. They don't try any default (live) instance. At the time both Artemis Backup instances are inactive.

      I think if clients are initially connected to Artemis default instance on first server, they should try to reconnect only to this instance or to its backup.

      Note: The same scenario works with remote connectors/acceptors.

      Customer impact: Clients may hang after failback in colocated HA topology with HTTP connectors. If it happens, client's application must be restarted.

      Attachments

        Issue Links

          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. WFLY-8033 Clients are not able to do failback in colocated HA topology with HTTP connectors

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          relates to

          ARTEMIS-931 Loading...

          Activity

            People

              jmesnil1@redhat.com Jeff Mesnil
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: