Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-1536

Misconfigured HA Shared store shows proper cluster topology, unless started to use properly

    XMLWordPrintable

Details

    • Story
    • Resolution: Unresolved
    • Major
    • None
    • AMQ 7.2.0.GA
    • high-availability
    • None

    Description

      UPDATE:

      <ataylor> mtoth: still raise an enhancement to show misconfigured backups and passive backups in topology correctly
      

      Original issue - slightly invalid:
      When HA Shared store is set up, the topology is not updated properly.
      It still uses "original" from cluster. There are no backups announced via Topology calls.

      So effectively User has no means to figure out, who is going to be active slave, when master goes down. It can be only seen from logs.
      This also prevents QE to write any tests for HA shared store, because we don't know who is going to be the taking-over slave.

      For HA replication, one can easily see, that Pairs (in topology) are formed correctly and we know which broker is going to be active, when master goes down.

      This also gives more sense why HA Shared Store is not capable to show slaves, only master nodes. See ENTMQBR-1498 & ENTMQBR-1478 for details.

      Examples of topology - observe Pair[] parts:
      Cluster of 6 nodes (only A or B in pair is filled -> expected OK)

      ====== 6 nodes Cluster 
      {"request":{"mbean":"org.apache.activemq.artemis:broker=\"amq-10-37-145-202\",component=cluster-connections,name=\"my-cluster\"","attribute":"Topology","type":"read"},"value":"topology on Topology@14af7052[owner=ClusterConnectionImpl@1812823171[nodeUUID=e6b35213-5836-11e8-a025-52540045bbb8, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-202, address=, server=ActiveMQServerImpl::serverUUID=e6b35213-5836-11e8-a025-52540045bbb8]]:
      
        e80c5303-5836-11e8-8036-5254004e96d5 => TopologyMember[id = e80c5303-5836-11e8-8036-5254004e96d5, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-203, b=null], backupGroupName=null, scaleDownGroupName=null]
      
        e6d3d301-5836-11e8-85c6-525400b22d8e => TopologyMember[id = e6d3d301-5836-11e8-85c6-525400b22d8e, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-205, b=null], backupGroupName=null, scaleDownGroupName=null]
      
        e6e6e5f5-5836-11e8-b4b0-5254007f6e6f => TopologyMember[id = e6e6e5f5-5836-11e8-b4b0-5254007f6e6f, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-204, b=null], backupGroupName=null, scaleDownGroupName=null]
      
        e8cdcd9d-5836-11e8-bea5-5254003da80d => TopologyMember[id = e8cdcd9d-5836-11e8-bea5-5254003da80d, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-214, b=null], backupGroupName=null, scaleDownGroupName=null]
      
        e6b35213-5836-11e8-a025-52540045bbb8 => TopologyMember[id = e6b35213-5836-11e8-a025-52540045bbb8, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-202, b=null], backupGroupName=null, scaleDownGroupName=null]
      
        e7c9cb11-5836-11e8-849a-525400f1f81a => TopologyMember[id = e7c9cb11-5836-11e8-849a-525400f1f81a, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-223, b=null], backupGroupName=null, scaleDownGroupName=null]
      	nodes=6	members=6","timestamp":1526469962,"status":200}
      

      HA Replication (both A and B in pair are filled -> expected OK)

      ====== 6 nodes HA Replication
      {"request":{"mbean":"org.apache.activemq.artemis:broker=\"amq-10-37-145-105\",component=cluster-connections,name=\"my-cluster-ha\"","attribute":"Topology","type":"read"},"value":"topology on Topology@3f991a0d[owner=ClusterConnectionImpl@67749199[nodeUUID=090785ed-5900-11e8-954f-525400bf24d7, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-105, address=, server=ActiveMQServerImpl::serverUUID=090785ed-5900-11e8-954f-525400bf24d7]]:
      
        1f4e4cb5-58ff-11e8-bfc5-525400907622 => TopologyMember[id = 1f4e4cb5-58ff-11e8-bfc5-525400907622, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-109, b=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-106], backupGroupName=dog, scaleDownGroupName=null]
      
        265c547d-58ff-11e8-ba86-525400546f1d => TopologyMember[id = 265c547d-58ff-11e8-ba86-525400546f1d, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-104, b=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-97], backupGroupName=dog, scaleDownGroupName=null]
      
        090785ed-5900-11e8-954f-525400bf24d7 => TopologyMember[id = 090785ed-5900-11e8-954f-525400bf24d7, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-105, b=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-37-145-107], backupGroupName=dog, scaleDownGroupName=null]
      	nodes=12	members=9","timestamp":1526472510,"status":200}
      

      HA Shared Store (only A or B in pair are filled -> expected both - HA Pair -> Not OK)

      ====== 2 nodes HA Shared Store
      {"request":{"mbean":"org.apache.activemq.artemis:broker=\"amq-172-0-0-5\",component=cluster-connections,name=\"my-cluster\"","attribute":"Topology","type":"read"},"value":"topology on Topology@73eb4f33[owner=ClusterConnectionImpl@220695851[nodeUUID=665ea2df-58f5-11e8-9795-fa163e4614e7, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=172-0-0-5, address=, server=ActiveMQServerImpl::serverUUID=665ea2df-58f5-11e8-9795-fa163e4614e7]]:
      
      	664b6986-58f5-11e8-a696-fa163ec4c59b => TopologyMember[id = 664b6986-58f5-11e8-a696-fa163ec4c59b, connector=Pair[a=null, b=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=172-0-0-15], backupGroupName=null, scaleDownGroupName=null]
      
      	665ea2df-58f5-11e8-9795-fa163e4614e7 => TopologyMember[id = 665ea2df-58f5-11e8-9795-fa163e4614e7, connector=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=172-0-0-5, b=null], backupGroupName=null, scaleDownGroupName=null]
      
      	nodes=2	members=2","timestamp":1526469828,"status":200}
      

      Another example using MBeans (172.0.0.5 is live, rest are backups):

      ActiveMQServerControl serverControl = getResolver(ArtemisResolver.class).getBrokerView();
              logger.info("topology="  + serverControl.listNetworkTopology());
      

      on all nodes produces

      $ java -jar amqx-2.0.0.amq-700002-redhat-1.jar broker --host 172.0.0.27  --broker-name amq-172-0-0-27 --action connectors
      topology=[{"nodeID":"4921e104-59c0-11e8-9a21-fa163e4614e7","live":"172.0.0.5:61616"},{},{},{}]
      
      $ java -jar amqx-2.0.0.amq-700002-redhat-1.jar broker --host 172.0.0.5  --broker-name amq-172-0-0-5 --action connectors
      topology=[{"nodeID":"4921e104-59c0-11e8-9a21-fa163e4614e7","live":"172.0.0.5:61616"},{},{},{}]
      
      $ java -jar amqx-2.0.0.amq-700002-redhat-1.jar broker  --broker-name amq-172-0-0-15 --action connectors
      topology=[{"nodeID":"4921e104-59c0-11e8-9a21-fa163e4614e7","live":"172.0.0.5:61616"},{},{},{}]
      

      Notice, that there are three empty {} -> most probably missing info for connected three backups/slaves.
      After a quick test and starting live and one slave after another, I have confirmed that those empty "{}" are slaves.

      Start live + slave
      [root@172 node_data]# java -jar amqx-2.0.0.amq-700002-redhat-1.jar broker --host 172.0.0.5  --broker-name amq-172-0-0-5 --action connectors
      topology=[{"nodeID":"4921e104-59c0-11e8-9a21-fa163e4614e7","live":"172.0.0.5:61616"},{}]
      
      Start another slave 
      [root@172 node_data]# java -jar amqx-2.0.0.amq-700002-redhat-1.jar broker --host 172.0.0.5  --broker-name amq-172-0-0-5 --action connectors
      topology=[{"nodeID":"4921e104-59c0-11e8-9a21-fa163e4614e7","live":"172.0.0.5:61616"},{},{}]
      

      Attachments

        Issue Links

          Activity

            People

              rh-ee-ataylor Andy Taylor
              mtoth@redhat.com Michal Toth
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: