Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-2073

[A-MQ, kahadb] NullPointerException raised at org.apache.activemq.store.kahadb.disk.index.BTreeNode$BTreeIterator.findNextPage()

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: JBoss A-MQ 6.3.x
    • Component/s: kahadb
    • Labels:
      None

      Description

      After the broker runs out of disk space (intended in the disaster recovery testing) intermittently the next start results in many "message not found in sequence id index:" and failed to start with a NullPointerException.

      initial "no space on device" :

      2017-03-08 11:02:21,058 | DEBUG | heckpoint Worker | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1896 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | Checkpoint done.
      2017-03-08 11:02:26,106 | DEBUG | heckpoint Worker | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1663 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | Checkpoint started.
      2017-03-08 11:02:26,148 | DEBUG | heckpoint Worker | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1896 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | Checkpoint done.
      2017-03-08 11:02:30,251 | ERROR | .0.1:45603@61616 | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1117 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | KahaDB failed to store to Journal
      java.io.FileNotFoundException: /XXXXX/jboss-a-mq-6.3.0.redhat-250/data/amqdb/kahadb/db-32.log (No space left on device)
      	at java.io.RandomAccessFile.open0(Native Method)[:1.8.0_91]
      	at java.io.RandomAccessFile.open(RandomAccessFile.java:316)[:1.8.0_91]
      	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)[:1.8.0_91]
      	at org.apache.activemq.util.RecoverableRa2017-03-08 11:03:16,121 | INFO  | FelixStartLevel  | fileinstall                      | ?                                   ? | 9 - org.apache.felix.fileinstall - 3.5.8 | Updating configuration from io.fabric8.mq.fabric.server-broker.cfg
      
      

      Subsequent restart has many

      2017-03-08 11:03:30,822 | DEBUG | AMQ-1-thread-1   | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1516 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | message not found in sequence id index: ID:XXXXX:16:1704
      
      

      and fails to start with the following NullPointer

      2017-03-08 11:03:34,903 | INFO  | AMQ-1-thread-1   | ActiveMQServiceFactory           | Factory$ClusteredConfiguration$1  502 | 177 - io.fabric8.mq.mq-fabric - 1.2.0.redhat-630250 | Broker amq failed to start.  Will try again in 10 seconds                                                                                                                            
      2017-03-08 11:03:34,903 | ERROR | AMQ-1-thread-1   | ActiveMQServiceFactory           | Factory$ClusteredConfiguration$1  503 | 177 - io.fabric8.mq.mq-fabric - 1.2.0.redhat-630250 | Exception on start: java.lang.NullPointerException
      java.lang.NullPointerException
      	at org.apache.activemq.store.kahadb.disk.index.BTreeNode$BTreeIterator.findNextPage(BTreeNode.java:118)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.store.kahadb.disk.index.BTreeNode$BTreeIterator.hasNext(BTreeNode.java:129)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$2.execute(KahaDBStore.java:528)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$2.execute(KahaDBStore.java:521)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:802)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.getMessageCount(KahaDBStore.java:521)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:100)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.region.Queue.initialize(Queue.java:386)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:87)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:541)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:135)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:348)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:199)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:118)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:177)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:208)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:120)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:119)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerService$6.start(BrokerService.java:2325)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:719)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:705)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at org.apache.activemq.broker.BrokerService.start(BrokerService.java:611)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
      	at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.doStart(ActiveMQServiceFactory.java:549)[177:io.fabric8.mq.mq-fabric:1.2.0.redhat-630250]
      	at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.access$400(ActiveMQServiceFactory.java:359)[177:io.fabric8.mq.mq-fabric:1.2.0.redhat-630250]
      	at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration$1.run(ActiveMQServiceFactory.java:490)[177:io.fabric8.mq.mq-fabric:1.2.0.redhat-630250]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_91]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_91]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_91]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_91]
      	at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]
      
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  garytully Gary Tully
                  Reporter:
                  pfox Patrick Fox
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: