Details
-
Bug
-
Resolution: Done
-
Major
-
AMQ 7.2.3.GA
Description
Intermittently getting error "AMQ222010: Critical IO Error, shutting down the server" and causing production broker shutdown. Here is sample error stack trace:
2019-02-14 05:30:09,714 WARN [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=AIOSequentialFile:/opt/broker01/data/journal/activemq-data-33124.amq, message=Timeout on close: java.io.IOException: Timeout on close at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.close(AIOSequentialFile.java:126) [artemis-journal-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.close(AIOSequentialFile.java:103) [artemis-journal-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.closeFile(JournalFilesRepository.java:495) [artemis-journal-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.moveNextFile(JournalImpl.java:3069) [artemis-journal-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.switchFileIfNecessary(JournalImpl.java:3008) [artemis-journal-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendRecord(JournalImpl.java:2727) [artemis-journal-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.access$200(JournalImpl.java:91) [artemis-journal-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.core.journal.impl.JournalImpl$4.run(JournalImpl.java:997) [artemis-journal-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.6.3.redhat-00013.jar:2.6.3.redhat-00013] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_172] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.6.3.redhat-00013.jar:2.6.3.redhat-00013]
Artemis was trying to append a record to one of journal files "activemq-data-33124.amq" but found that the size of the record does not fit into the journal file (not enough space left), therefore, it needs to switch to next file in order to write the record. But JournalFilesRepository needs to close the file of type AIOSequentialFile before opening a new one. However, the invocation "pendingCallbacks.await(10, TimeUnit.SECONDS)" failed due to 10 seconds timeout elapsed.
