-
Bug
-
Resolution: Done
-
Blocker
-
A-MQ 7.0.0.ER15
-
None
-
Replication is critical for us and these stability issues are very troubling. I see below error on the master broker, looks like the replication failed.
18:56:01,034 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending JournalFileImpl: (activemq-data-2.amq id = 2, recordID = 2) (size=10,485,760) to backup. AIOSequentialFile:/PerfTest/AMQ7.0/instance/run/data/journal/activemq-data-2.amq 18:56:01,357 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending JournalFileImpl: (activemq-bindings-4.bindings id = 1, recordID = 1) (size=1,048,576) to backup. NIOSequentialFile /PerfTest/AMQ7.0/instance/run/data/bindings/activemq-bindings-4.bindings 18:56:01,390 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending JournalFileImpl: (activemq-bindings-2.bindings id = 2, recordID = 2) (size=1,048,576) to backup. NIOSequentialFile /PerfTest/AMQ7.0/instance/run/data/bindings/activemq-bindings-2.bindings 18:56:02,214 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://h0064.ldn.labs.ubs.net:8161 18:56:02,214 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://h0064.ldn.labs.ubs.net:8161/jolokia 18:56:59,774 WARN [org.apache.activemq.artemis.journal] AMQ142012: Couldnt find tx={0} to merge after compacting 18:57:46,137 WARN [org.apache.activemq.artemis.core.server] AMQ222055: Error on deleting duplicate cache: java.lang.IllegalStateException: Cannot find add info 449312 on compactor or current records at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkKnownRecordID(JournalImpl.java:937) [artemis-journal-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendDeleteRecord(JournalImpl.java:822) [artemis-journal-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.replication.ReplicatedJournal.appendDeleteRecord(ReplicatedJournal.java:216) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.deleteDuplicateID(AbstractJournalStorageManager.java:454) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl.addToCacheInMemory(DuplicateIDCacheImpl.java:263) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl.access$000(DuplicateIDCacheImpl.java:41) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl$AddDuplicateIDOperation.process(DuplicateIDCacheImpl.java:340) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl$AddDuplicateIDOperation.afterCommit(DuplicateIDCacheImpl.java:348) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:546) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.access$100(TransactionImpl.java:38) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$3.done(TransactionImpl.java:321) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:236) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101) [artemis-commons-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_112] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_112]
Replication is critical for us and these stability issues are very troubling. I see below error on the master broker, looks like the replication failed. 18:56:01,034 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending JournalFileImpl: (activemq-data-2.amq id = 2, recordID = 2) (size=10,485,760) to backup. AIOSequentialFile:/PerfTest/AMQ7.0/instance/run/data/journal/activemq-data-2.amq 18:56:01,357 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending JournalFileImpl: (activemq-bindings-4.bindings id = 1, recordID = 1) (size=1,048,576) to backup. NIOSequentialFile /PerfTest/AMQ7.0/instance/run/data/bindings/activemq-bindings-4.bindings 18:56:01,390 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending JournalFileImpl: (activemq-bindings-2.bindings id = 2, recordID = 2) (size=1,048,576) to backup. NIOSequentialFile /PerfTest/AMQ7.0/instance/run/data/bindings/activemq-bindings-2.bindings 18:56:02,214 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http: //h0064.ldn.labs.ubs.net:8161 18:56:02,214 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http: //h0064.ldn.labs.ubs.net:8161/jolokia 18:56:59,774 WARN [org.apache.activemq.artemis.journal] AMQ142012: Couldnt find tx={0} to merge after compacting 18:57:46,137 WARN [org.apache.activemq.artemis.core.server] AMQ222055: Error on deleting duplicate cache: java.lang.IllegalStateException: Cannot find add info 449312 on compactor or current records at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkKnownRecordID(JournalImpl.java:937) [artemis-journal-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendDeleteRecord(JournalImpl.java:822) [artemis-journal-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.replication.ReplicatedJournal.appendDeleteRecord(ReplicatedJournal.java:216) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.deleteDuplicateID(AbstractJournalStorageManager.java:454) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl.addToCacheInMemory(DuplicateIDCacheImpl.java:263) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl.access$000(DuplicateIDCacheImpl.java:41) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl$AddDuplicateIDOperation.process(DuplicateIDCacheImpl.java:340) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl$AddDuplicateIDOperation.afterCommit(DuplicateIDCacheImpl.java:348) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:546) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.access$100(TransactionImpl.java:38) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$3.done(TransactionImpl.java:321) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:236) [artemis-server-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101) [artemis-commons-1.5.1.amq-700002-redhat-1.jar:1.5.1.amq-700002-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_112] at java.lang. Thread .run( Thread .java:745) [rt.jar:1.8.0_112]
We are using AMQ7 ER15 broker and AMQ6.3 java client, the issue occurs randomly, hard to send out a reproducer.
- is related to
-
JBEAP-10062 Lost paged messages if journal is located on NFS
-
- Closed
-