Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-8033

Clients are not able to do failback in colocated HA topology with HTTP connectors

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Done
    • 10.1.0.Final
    • 11.0.0.Alpha1
    • JMS
    • None
    • Hide
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
      cd eap-tests-hornetq/scripts/
      git checkout cc737cb50d1477d292a814cfc920835506e345a0
      groovy -DEAP_VERSION=7.1.0.DR11 PrepareServers7.groovy
      export WORKSPACE=$PWD
      export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
      export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
      export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
      export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
      
      cd ../jboss-hornetq-testsuite/
      
       mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackTransAckTopic -DfailIfNoTests=false -Deap=7x -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1.0.DR11  | tee log
      

      Tip for debug: Trace logs of client's side are stored in file eap-tests-hornetq/jboss-hornetq-testsuite/common-tests/target/test-suite.log

      Note: This is not 100% reproducer. However I'm able to reproduce the issue locally on my laptop.

      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout cc737cb50d1477d292a814cfc920835506e345a0 groovy -DEAP_VERSION=7.1.0.DR11 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackTransAckTopic -DfailIfNoTests= false -Deap=7x -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1.0.DR11 | tee log Tip for debug: Trace logs of client's side are stored in file eap-tests-hornetq/jboss-hornetq-testsuite/common-tests/target/test-suite.log Note: This is not 100% reproducer. However I'm able to reproduce the issue locally on my laptop.
    • Workaround Exists
    • Hide

      Use remote connectors/acceptors

      Show
      Use remote connectors/acceptors

    Description

      Scenario:

      • There are two EAPs configured in old way colocated HA topology
      • Start both servers
      • Connect clients to the first server
      • Kill first server
      • Wait until clients do failover
      • Start first server
      • Wait until clients do failback

      Description of problem:
      Sometimes happens that after the first server is started again, clients don't do failback. Producers are blocked because they don't have credits for sending. Receivers don't receive any messages.

      In trace logs I can see following messages:

      10:22:52,915 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1
      10:22:52,915 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1
      10:22:53,241 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@6f7ca1c1, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost
      10:22:53,241 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
      10:22:53,242 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080
      10:22:52,743 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1
      10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@25572f4, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost
      10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
      10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080
      10:22:53,242 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
      GET  HTTP/1.1
      Host: localhost
      Upgrade: activemq-remoting
      Connection: Upgrade
      activemqServerName: backup
      httpUpgradeEndpoint: acceptor-backup
      Sec-ActiveMQRemoting-Key: bvTu4ZbJbAqWX5n3TwgNGA==
      10:22:53,241 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@1c5a232f, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost
      10:22:53,242 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
      10:22:53,242 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080
      10:22:53,241 main INFO  [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:178] Producer: Thread[Thread-22,5,main] is still alive.
      10:22:53,243 main INFO  [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:178] Producer: Thread[Thread-23,5,main] is still alive.
      10:22:53,243 main INFO  [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:191] Reciever: Thread[Thread-24,5,] is finished.
      10:22:53,243 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
      GET  HTTP/1.1
      Host: localhost
      Upgrade: activemq-remoting
      Connection: Upgrade
      activemqServerName: backup
      httpUpgradeEndpoint: acceptor-backup
      Sec-ActiveMQRemoting-Key: vxNO5uMO4KK6etjAlAtorA==
      10:22:53,243 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
      GET  HTTP/1.1
      Host: localhost
      Upgrade: activemq-remoting
      Connection: Upgrade
      activemqServerName: backup
      httpUpgradeEndpoint: acceptor-backup
      Sec-ActiveMQRemoting-Key: aUC+bE6WaMmU1A6KGcssww==
      10:22:53,243 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
      10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost
      10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
      10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080
      10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
      10:22:53,244 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
      10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
      GET  HTTP/1.1
      Host: localhost
      Upgrade: activemq-remoting
      Connection: Upgrade
      activemqServerName: backup
      httpUpgradeEndpoint: acceptor-backup
      Sec-ActiveMQRemoting-Key: plpRSFUDmbA3Xky/5BWx6g==
      10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost
      10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
      10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080
      10:22:53,244 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost
      10:22:53,245 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
      10:22:53,245 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080
      10:22:53,245 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
      10:22:53,245 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
      GET  HTTP/1.1
      Host: localhost
      Upgrade: activemq-remoting
      Connection: Upgrade
      activemqServerName: backup
      httpUpgradeEndpoint: acceptor-backup
      Sec-ActiveMQRemoting-Key: f2nehkPE0VDmGlWAcx3Ltg==
      10:22:53,246 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
      GET  HTTP/1.1
      Host: localhost
      Upgrade: activemq-remoting
      Connection: Upgrade
      activemqServerName: backup
      httpUpgradeEndpoint: acceptor-backup
      Sec-ActiveMQRemoting-Key: RIaHZ2gy6JEm9nz6XOdyVA==
      10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
      10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1104] Backup is not active.
      10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1258] Neither backup or live were active, will just give up now
      10:22:53,245 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1104] Backup is not active.
      10:22:53,246 Thread-2 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1258] Neither backup or live were active, will just give up now
      10:22:53,246 Thread-2 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:815] Waiting 2000 milliseconds before next retry. RetryInterval=2000 and multiplier=1.0
      

      You can notice in trace logs that clients try to reconnect only to Artemis backup instances. They don't try any default (live) instance. At the time both Artemis Backup instances are inactive.

      I think if clients are initially connected to Artemis default instance on first server, they should try to reconnect only to this instance or to its backup.

      Note: The same scenario works with remote connectors/acceptors.

      Customer impact: Clients may hang after failback in colocated HA topology with HTTP connectors. If it happens, client's application must be restarted.

      Attachments

        Issue Links

          Activity

            People

              jmesnil1@redhat.com Jeff Mesnil
              jmesnil1@redhat.com Jeff Mesnil
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: