Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: JBoss A-MQ 6.3.x
Affects Version/s: None
Component/s: kahadb
Labels:
None

GSS Priority:
Release Note Text:

Hide
see some detail on: kahaDBIOExceptionHandler at:
http://activemq.apache.org/configurable-ioexception-handling.html

also https://issues.apache.org/jira/browse/AMQ-6625

Show
see some detail on: kahaDBIOExceptionHandler at: http://activemq.apache.org/configurable-ioexception-handling.html also https://issues.apache.org/jira/browse/AMQ-6625
Release Note Status:
Not Yet Documented

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

After the broker runs out of disk space (intended in the disaster recovery testing) intermittently the next start results in many "message not found in sequence id index:" and failed to start with a NullPointerException.

initial "no space on device" :

2017-03-08 11:02:21,058 | DEBUG | heckpoint Worker | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1896 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | Checkpoint done.
2017-03-08 11:02:26,106 | DEBUG | heckpoint Worker | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1663 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | Checkpoint started.
2017-03-08 11:02:26,148 | DEBUG | heckpoint Worker | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1896 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | Checkpoint done.
2017-03-08 11:02:30,251 | ERROR | .0.1:45603@61616 | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1117 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | KahaDB failed to store to Journal
java.io.FileNotFoundException: /XXXXX/jboss-a-mq-6.3.0.redhat-250/data/amqdb/kahadb/db-32.log (No space left on device)
	at java.io.RandomAccessFile.open0(Native Method)[:1.8.0_91]
	at java.io.RandomAccessFile.open(RandomAccessFile.java:316)[:1.8.0_91]
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)[:1.8.0_91]
	at org.apache.activemq.util.RecoverableRa2017-03-08 11:03:16,121 | INFO  | FelixStartLevel  | fileinstall                      | ?                                   ? | 9 - org.apache.felix.fileinstall - 3.5.8 | Updating configuration from io.fabric8.mq.fabric.server-broker.cfg

Subsequent restart has many

2017-03-08 11:03:30,822 | DEBUG | AMQ-1-thread-1   | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1516 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | message not found in sequence id index: ID:XXXXX:16:1704

and fails to start with the following NullPointer

2017-03-08 11:03:34,903 | INFO  | AMQ-1-thread-1   | ActiveMQServiceFactory           | Factory$ClusteredConfiguration$1  502 | 177 - io.fabric8.mq.mq-fabric - 1.2.0.redhat-630250 | Broker amq failed to start.  Will try again in 10 seconds                                                                                                                            
2017-03-08 11:03:34,903 | ERROR | AMQ-1-thread-1   | ActiveMQServiceFactory           | Factory$ClusteredConfiguration$1  503 | 177 - io.fabric8.mq.mq-fabric - 1.2.0.redhat-630250 | Exception on start: java.lang.NullPointerException
java.lang.NullPointerException
	at org.apache.activemq.store.kahadb.disk.index.BTreeNode$BTreeIterator.findNextPage(BTreeNode.java:118)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.disk.index.BTreeNode$BTreeIterator.hasNext(BTreeNode.java:129)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$2.execute(KahaDBStore.java:528)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$2.execute(KahaDBStore.java:521)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:802)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.getMessageCount(KahaDBStore.java:521)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:100)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.Queue.initialize(Queue.java:386)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:87)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:541)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:135)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:348)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:199)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:118)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:177)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:208)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:120)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:119)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerService$6.start(BrokerService.java:2325)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:719)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:705)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerService.start(BrokerService.java:611)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.doStart(ActiveMQServiceFactory.java:549)[177:io.fabric8.mq.mq-fabric:1.2.0.redhat-630250]
	at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.access$400(ActiveMQServiceFactory.java:359)[177:io.fabric8.mq.mq-fabric:1.2.0.redhat-630250]
	at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration$1.run(ActiveMQServiceFactory.java:490)[177:io.fabric8.mq.mq-fabric:1.2.0.redhat-630250]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_91]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_91]
	at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]

is related to

ENTMQ-2112 [A-MQ 6, kahadb] kahaDBIOExceptionHandler cannot be used when using lease locker

Closed

relates to

ENTMQ-2065 [A-MQ 6, kahadb, recovery] partial last write in journal is blocking broker restart

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates