The slave after the configuration reload cannot be a live server

The slave after the configuration reload cannot be a live server

[Problem]

• When I replaced broker.xml on the master and the slave to add new queue configuration, the master broker deploys and reload the queue as expected, but the slave broker deploys the queue, not only reload queue configuration[1]. And then when after stopping the master broker, the slave broker output error[2] or NPE[3] and is not started as live server.

[Note]

• When only replacing master configuration, There are no any errors and problems. But broker.xml files are not synchronized on master and slave.

[Reproducer]

• I attached a reproducer. Please see ReplicatedFailbackStaticExample.java #L54

1. Unzip the “reproducer.zip" to amq-broker-7.1.0/examples/features/ha/replicated-failback-static
2. Execute "mvn clean verify" on the above directory.
3. Reproduce error[2] or NPE[3]. (Probably error[2] is only the first time. NPE are the second time or more)

[1] Deploy the queue,  not only reload queue configuration on the slave
~~~
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:34,970 WARN  [org.apache.activemq.artemis.core.server] AMQ222269: Please use a fixed value for "journal-pool-files". Default changed per https://issues.apache.org/jira/browse/ARTEMIS-1628
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:34,972 INFO  [org.apache.activemq.artemis.core.server] AMQ221056: Reloading configuration ...security
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:34,981 INFO  [org.apache.activemq.artemis.core.server] AMQ221056: Reloading configuration ...address settings
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:34,982 INFO  [org.apache.activemq.artemis.core.server] AMQ221056: Reloading configuration ...diverts
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:34,982 INFO  [org.apache.activemq.artemis.core.server] AMQ221056: Reloading configuration ...addresses
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:35,023 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue exampleQueue on address exampleQueue
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:35,045 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue exampleQueue in AddressSettings
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:35,046 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue exampleQueue in AddressSettings
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:35,098 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue exampleQueue1 on address exampleQueue1
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:35,104 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue exampleQueue1 in AddressSettings
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:35,105 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue exampleQueue1 in AddressSettings
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:35,111 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue exampleQueue2 on address exampleQueue2
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:35,112 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue exampleQueue2 in AddressSettings
ReplicatedFailbackStaticExample1-out:2018-03-26 18:30:35,113 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue exampleQueue2 in AddressSettings
~~~

[2] Error
~~~
WARN: AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,032 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,039 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,038 INFO  [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,046 INFO  [org.apache.activemq.artemis.core.server] AMQ221067: Waiting 30 seconds for quorum vote results.
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,046 INFO  [org.apache.activemq.artemis.core.server] AMQ221068: Received all quorum votes.
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,047 INFO  [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,055 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,128 INFO  [org.apache.activemq.artemis.core.server] AMQ221037: ActiveMQServerImpl::serverUUID=2721d949-30df-11e8-abc3-c85b76bd6ea0 to become 'live'
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,137 WARN  [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,168 ERROR [org.apache.activemq.artemis.core.server] AMQ224008: Failed to store id: java.lang.IllegalStateException: Cannot find add info 1 on compactor or current records
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkKnownRecordID(JournalImpl.java:1019) [artemis-journal-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendDeleteRecord(JournalImpl.java:897) [artemis-journal-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.journal.impl.JournalBase.appendDeleteRecord(JournalBase.java:155) [artemis-journal-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendDeleteRecord(JournalImpl.java:90) [artemis-journal-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.deleteID(AbstractJournalStorageManager.java:797) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.persistence.impl.journal.BatchingIDGenerator.deleteID(BatchingIDGenerator.java:155) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.persistence.impl.journal.BatchingIDGenerator.cleanup(BatchingIDGenerator.java:78) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1491) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2539) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2322) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:343) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:3044) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,173 WARN  [org.apache.activemq.artemis.core.server] AMQ222173: Queue exampleQueue is duplicated during reload. This queue will be renamed as exampleQueue-0
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,174 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue exampleQueue-0 in AddressSettings
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,174 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue exampleQueue-0 in AddressSettings
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,183 WARN  [org.apache.activemq.artemis.core.server] AMQ222173: Queue exampleQueue1 is duplicated during reload. This queue will be renamed as exampleQueue1-1
ReplicatedFailbackStaticExample1-out:2018-03-26 19:20:17,184 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: java.lang.IllegalStateException: java.lang.IllegalStateException: Cursor 8 had already been created
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.QueueConfig$Builder.build(QueueConfig.java:151) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.initQueues(PostOfficeJournalLoader.java:154) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2549) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2322) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:343) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:3044) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:Caused by: java.lang.IllegalStateException: Cursor 8 had already been created
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl.createSubscription(PageCursorProviderImpl.java:100) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.server.QueueConfig$Builder.build(QueueConfig.java:149) [artemis-server-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   ... 5 more
ReplicatedFailbackStaticExample1-out:
~~~

[3] NPE
~~~
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,116 INFO  [org.apache.activemq.artemis.core.server] AMQ221068: Received all quorum votes.
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,116 INFO  [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,126 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,108 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,289 INFO  [org.apache.activemq.artemis.core.server] AMQ221037: ActiveMQServerImpl::serverUUID=c481e2a2-30cf-11e8-bb1a-c85b76bd6ea0 to become 'live'
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,331 WARN  [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,396 WARN  [org.apache.activemq.artemis.core.server] AMQ222173: Queue exampleQueue is duplicated during reload. This queue will be renamed as exampleQueue-0
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,397 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue exampleQueue-0 in AddressSettings
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,398 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue exampleQueue-0 in AddressSettings
ReplicatedFailbackStaticExample1-out:2018-03-26 17:30:09,396 ERROR [org.apache.activemq.artemis.core.journal.impl.JournalImpl] appendDeleteRecord:java.lang.NullPointerException: java.lang.NullPointerException
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.core.journal.impl.JournalImpl$3.run(JournalImpl.java:920) [artemis-journal-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at org.apache.activemq.artemis.utils.actors.ProcessorBase$ExecutorTask.run(ProcessorBase.java:53) [artemis-commons-2.4.0.amq-710004-redhat-1.jar:2.4.0.amq-710004-redhat-1]
ReplicatedFailbackStaticExample1-out:   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_144]
ReplicatedFailbackStaticExample1-out:   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_144]
ReplicatedFailbackStaticExample1-out:   at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_144]
ReplicatedFailbackStaticExample1-out:
~~~