I have two separate Fabric environment and a broker container on each of the two fabrics. Both broker were configured to use JDBC persistent adapter and connect to the same MS SQLServer database to form a master/slave cluster.
It happens quite often when the master broker container was stopped but the failover to the slave broker just failed. Here was what happened:
The slave broker container was re-started:
Jul 17, 2019 12:30:10 PM org.apache.karaf.main.SimpleFileLock lock
During the startup process the mq-fabric bundle was refreshed by the fabric deployment agent as part of the fabric container start process. But the mq-fabric bundle failed to stop due to the concurrent CancellationException
2019-07-17 12:32:36,839 | WARN | 6.0.1-1-thread-1 | Deployer | 176 - io.fabric8.fabric-agent - 1.2.0.redhat-630396 | Error while trying to stop bundle io.fabric8.mq.mq-fabric
Because the mq-fabric bundle refresh failed, the relevant MBean "org.apache.activemq:type=Broker,brokerName=MyNotificationBroker" wasn't unregistered properly. Therefore, JMX always threw the "javax.management.InstanceAlreadyExistsException" exception and it had prevented the broker from starting up:
2019-07-17 12:33:38,539 | ERROR | AMQ-1-thread-1 | BrokerService | 219 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630396 | Failed to start Apache ActiveMQ (MyNotificationBroker, null)
Therefore, the master/slave cluster failover failed because the slave broker can not be started up due to above "javax.management.InstanceAlreadyExistsException" on the broker MBean.