Uploaded image for project: 'WildFly WIP'
  1. WildFly WIP
  2. WFWIP-11

[Artemis upgrade] Regression in replicated HA tests

    XMLWordPrintable

Details

    • Hide 
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-JMS/view/early-testing/view/tooling/job/early-testing-messaging-prepare/125/artifact/jboss-eap.zip PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=ReplicatedDedicatedFailoverTestCase#testFailbackClientAckTopicOnShutdown -DfailIfNoTests=false -Deap=7x -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1517218654-SNAPSHOT | tee log
      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-JMS/view/early-testing/view/tooling/job/early-testing-messaging-prepare/125/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ReplicatedDedicatedFailoverTestCase#testFailbackClientAckTopicOnShutdown -DfailIfNoTests= false -Deap=7x -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1517218654-SNAPSHOT | tee log

    Description

      Scenario

      • There are two servers configured as replicated Live-Backup pair
      • Live server is killed
      • Test waits until Backup server activates
      • Live server is restarted
      • Test expects that Backup server deactivates and Live becomes active

      Reality:
      Sometimes happens that Live server doesn't become active. In the log I can see that it was synchronized with Backup, but based on quorum vote, it was restarted as Backup.

      Customer impact: Failback feature in replicated HA is unstable.

      Live
      12:26:51,011 INFO  [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.5.0-SNAPSHOT [null] started, waiting live to fail before it gets active
12:26:51,013 INFO  [org.apache.activemq.artemis.core.client] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ211002: Started EPOLL Netty Connector version unknown to localhost:9080
12:26:52,344 INFO  [org.apache.activemq.artemis.core.server] (Thread-6 (ActiveMQ-client-netty-threads)) AMQ221024: Backup server ActiveMQServerImpl::serverUUID=a9e2f7d1-0742-11e8-bd92-54ee7553e6a7 is synchronized with live-server.
12:26:52,386 INFO  [org.apache.activemq.artemis.core.client] (Thread-5 (ActiveMQ-client-netty-threads)) AMQ211002: Started EPOLL Netty Connector version unknown to localhost:9080
12:26:52,423 INFO  [org.apache.activemq.artemis.core.server] (Thread-5 (ActiveMQ-client-netty-threads)) AMQ221070: Restarting as backup based on quorum vote results.
      Backup
      12:26:51,435 INFO  [org.apache.activemq.artemis.core.server] (Thread-152) AMQ221025: Replication: sending AIOSequentialFile:/home/eduda/Projects/messaging-testsuite/server2/jboss-eap/standalone/data/../../../../
hornetq-journal-B/journal/activemq-data-11.amq (size=10 485 760) to replica.
12:26:51,766 INFO  [org.apache.activemq.artemis.core.server] (Thread-152) AMQ221025: Replication: sending AIOSequentialFile:/home/eduda/Projects/messaging-testsuite/server2/jboss-eap/standalone/data/../../../../
hornetq-journal-B/journal/activemq-data-4.amq (size=10 485 760) to replica.
12:26:51,884 INFO  [org.apache.activemq.artemis.core.server] (Thread-152) AMQ221025: Replication: sending NIOSequentialFile /home/eduda/Projects/messaging-testsuite/server2/jboss-eap/standalone/data/../../../../
hornetq-journal-B/bindings/activemq-bindings-4.bindings (size=1 048 576) to replica.
12:26:51,890 INFO  [org.apache.activemq.artemis.core.server] (Thread-152) AMQ221025: Replication: sending NIOSequentialFile /home/eduda/Projects/messaging-testsuite/server2/jboss-eap/standalone/data/../../../../
hornetq-journal-B/bindings/activemq-bindings-2.bindings (size=1 048 576) to replica.
12:26:52,388 WARN  [org.apache.activemq.artemis.core.client] (Thread-4 (ActiveMQ-client-global-threads)) AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of ser
ver shutdown [code=DISCONNECTED]
12:26:52,391 WARN  [org.jboss.activemq.artemis.wildfly.integration.recovery] (Thread-4 (ActiveMQ-client-global-threads)) being disconnected for server shutdown: ActiveMQDisconnectedException[errorType=DISCONNECT
ED message=AMQ119015: The connection was disconnected because of server shutdown]
        at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:996) [artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151]

12:26:52,396 WARN  [org.apache.activemq.artemis.core.server] (default I/O-9) AMQ222061: Client connection failed, clearing up resources for session c12b0cb8-0742-11e8-8a8a-54ee7553e6a7
12:26:52,397 WARN  [org.apache.activemq.artemis.core.server] (default I/O-9) AMQ222107: Cleared up resources for session c12b0cb8-0742-11e8-8a8a-54ee7553e6a7
12:26:52,397 WARN  [org.apache.activemq.artemis.core.server] (default I/O-9) AMQ222061: Client connection failed, clearing up resources for session c12da4cb-0742-11e8-8a8a-54ee7553e6a7
12:26:52,397 WARN  [org.apache.activemq.artemis.core.server] (default I/O-9) AMQ222107: Cleared up resources for session c12da4cb-0742-11e8-8a8a-54ee7553e6a7
12:26:52,412 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-1) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
12:26:52,412 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
12:26:52,412 INFO  [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 74) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
12:26:52,413 INFO  [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 107) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
12:26:52,416 INFO  [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 108) AMQ151003: resource adaptor stopped
12:26:52,458 WARN  [org.apache.activemq.artemis.core.server] (default I/O-13) AMQ222092: Connection to the backup node failed, removing replication now: ActiveMQRemoteDisconnectException[errorType=REMOTE_DISCONNECT message=null]
        at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:553)
        at org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor$Listener.connectionDestroyed(NettyAcceptor.java:768)
        at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelInactive(ActiveMQChannelHandler.java:78)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377)
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917)
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
        at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612) [xnio-nio-3.5.4.Final.jar:3.5.4.Final]
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:479) [xnio-nio-3.5.4.Final.jar:3.5.4.Final]

      Attachments

        Issue Links

          blocks

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. WFLY-10320 Upgrade artemis from 1.5.x to 2.x.x

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-1077 [Artemis upgrade] Regression in replicated HA tests

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-1164 [Artemis upgrade] Regression in replicated HA tests

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-1466 [3 HA pairs] Slave does not become live after master is killed and isolates itself

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              mtaylor1@redhat.com Martyn Taylor (Inactive)
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: