Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-2546

Repeated WARN: [AMQ212034: There are more than one servers on the network broadcasting the same node id] after restarting master broker in 3-node multicast cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Critical Critical
    • None
    • AMQ 7.3.0.GA, AMQ 7.2.4.GA, AMQ 7.4.1.GA
    • clustering, high-availability
    • None
    • Workaround Exists
    • Hide

      The workaround is to add a small delay before the service is started in systemd service file:

      [Service]
ExecStartPre=/bin/sleep 30
      Show
      The workaround is to add a small delay before the service is started in systemd service file: [Service] ExecStartPre=/bin/sleep 30
    • Hide

      I used the attached 3-node (6 brokers) configuration to reproduce the issue, along with the attached camel application to create a load on the cluster.

      1. Install and configure 6 brokers using the attached configuration (I installed a master and a slave on each of 3 virtual hosts)

      2. Start the brokers

      3. Build and start the attached camel spring boot example to push messages to the cluster

      4. Wait some time for a few thousand messages to accumulate on the broker

      5. Find the broker that has the local vs. SAF consumer (the one with the TEST.QUEUE.1-3 addresses)

      6. Stop and quickly restart this broker

      7. Tail the other broker logs in the cluster and look for repeated duplicate node warnings.

      Show
      I used the attached 3-node (6 brokers) configuration to reproduce the issue, along with the attached camel application to create a load on the cluster. 1. Install and configure 6 brokers using the attached configuration (I installed a master and a slave on each of 3 virtual hosts) 2. Start the brokers 3. Build and start the attached camel spring boot example to push messages to the cluster 4. Wait some time for a few thousand messages to accumulate on the broker 5. Find the broker that has the local vs. SAF consumer (the one with the TEST.QUEUE.1-3 addresses) 6. Stop and quickly restart this broker 7. Tail the other broker logs in the cluster and look for repeated duplicate node warnings.

      When restarting a master node under a load in a 3-master/slave pair cluster configured for udp multicast discovery, the cluster nodes all begin issuing duplicate node id warnings and do not recover after the node is restarted and rejoins the cluster.

      2019-05-25 18:57:26,451 WARN  [org.apache.activemq.artemis.core.client] AMQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=be93b547-7f3b-11e9-b540-080027990b69

      This seems to happen most often in testing when restarting the node to which the load is attached and after some backlog is allowed to develop in the addresses / queues.

        1. artemis.log_failed_host-10-0-132-24
          44 kB
        2. artemis.log_failed_host-10-0-133-83
          55 kB
        3. logs.zip
          4 kB
        4. reproducer.tar.gz
          24 kB

            fnigro Francesco Nigro
            rhn-support-dhawkins Duane Hawkins
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: