Loading...

Type: Bug
Resolution: Done
Priority: Undefined
Fix Version/s: None
Affects Version/s: AMQ 7.8.5.EAP.GA
Component/s: journal
Labels:

Blocked:
False
Blocked Reason:
None
Ready:
False
GSS Priority:
Target Release:

AMQ 7.10.6.GA
Upstream Jira:
https://issues.apache.org/jira/browse/ARTEMIS-4302
Git Pull Request:
https://github.com/rh-messaging/activemq-artemis/pull/620
Intelligence Requested:
Market:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

In a situation where the broker is restarting after a file system error, we see an unhandled exception in org.apache.activemq.artemis.core.journal.impl.JournalTransaction::forget upon restart, resulting in a failed boot:

05/24/23 14:30:41,569 ERROR [] [] [org.jboss.msc.service.fail] (ServerService Thread Pool -- 76) MSC000001: Failed to start service jboss.messaging-activemq.active.jms.manager: org.jboss.msc.service.StartException in service jboss.messaging-activemq.active.jms.manager: java.lang.Exception
	at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService.lambda$doStart$0(JMSService.java:147)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callActivationFailureListeners(ActiveMQServerImpl.java:2434)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.core.server.impl.SharedStoreLiveActivation.onActivationFailure(SharedStoreLiveActivation.java:118)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.core.server.impl.SharedStoreLiveActivation.run(SharedStoreLiveActivation.java:111)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:639)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:558)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:374)
	at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:211)
	at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65)
	at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: java.lang.Exception
	at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:866)
	at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:2028)
	at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:2319)
	at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1629)
	at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.core.journal.Journal.load(Journal.java:259)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:874)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:3518)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:3183)
	at org.apache.activemq.artemis@2.16.0.redhat-00034//org.apache.activemq.artemis.core.server.impl.SharedStoreLiveActivation.run(SharedStoreLiveActivation.java:94)
	... 14 more
Caused by: java.lang.NullPointerException
	at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.core.journal.impl.JournalTransaction.forget(JournalTransaction.java:358)
	at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.core.journal.impl.JournalImpl$12.onReadCommitRecord(JournalImpl.java:2214)
	at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:835)
	... 22 more

The context of this issue was a broker cluster running on a replicated NFS setup where the NFS file system was failed over and we observed Critical IO Errors on the brokers due to stale file handles - presumably due to some replication lag.

Upon restart, several of the brokers failed to start, logging the errors above. In this instance, the brokers were embedded brokers running in JBoss EAP 7.4.4 (broker version 2.16.0.redhat-00034)

clones

ENTMQBR-8122 [7.8.EAP] Unhandled NullPointerException in JournalTransaction::forget

Closed

is depended on by

JBEAP-25252 [GSS](8.0.z) ENTMQBR-8489 - Unhandled NullPointerException in JournalTransaction::forget

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates