Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-10704

AMQ142032: Error reading journal file arises in scenarios when server is killed during journal compacting

XMLWordPrintable

    • Workaround Exists
    • Hide

      Remove the truncated control file.

      Show
      Remove the truncated control file.
    • Hide 
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout 1abcd9b69f11aeec8007786f43c4c207a70eb9b6
groovy -DEAP_VERSION=7.1.0.ER1 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=DedicatedFailoverTestCase#testKillAtWriteControlFile -DfailIfNoTests=false -Deap=7x | tee log
      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout 1abcd9b69f11aeec8007786f43c4c207a70eb9b6 groovy -DEAP_VERSION=7.1.0.ER1 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=DedicatedFailoverTestCase#testKillAtWriteControlFile -DfailIfNoTests= false -Deap=7x | tee log
    • AMQ Sprint 3

      In test scenarios where EAP is killed during journal compacting it may happen that backup fails to load the journal.

      Customer impact: During the failover backup server may fail to start because of this issue. It leads to unavailability of the service and breaks meaning of HA.

      You can see the same exception even if you copy the corrupted journal in attachment into the fresh EAP installation and run the server with the standalone-full-ha profile.

      The stack trace of the exception [1] shows that the error arises when Backup reads JournalCompactor control file. In this case the Live only created the file but it didn't write any record to it. The Backup doesn't expect that the file could have 0 length.

      [1]

      15:32:43,529 WARN  [org.apache.activemq.artemis.journal] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=4b1f5a18-2c49-11e7-9fdb-001b217d6d93) AMQ142032: Error reading journal file: java.lang.Il
legalArgumentException
        at java.nio.Buffer.position(Buffer.java:244) [rt.jar:1.8.0_121]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:418) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalCompactor.readControlFile(JournalCompactor.java:79) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkControlFile(JournalImpl.java:2720) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1666) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1326) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1310) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:837) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redh
at-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2213) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2067) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.SharedStoreBackupActivation.run(SharedStoreBackupActivation.java:90) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2491) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]

15:32:43,530 ERROR [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=4b1f5a18-2c49-11e7-9fdb-001b217d6d93) AMQ224000: Failure in initialisation: java.lang
.Exception
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:702) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalCompactor.readControlFile(JournalCompactor.java:79) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkControlFile(JournalImpl.java:2720) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1666) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1326) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1310) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:837) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redh
at-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2213) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2067) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.SharedStoreBackupActivation.run(SharedStoreBackupActivation.java:90) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2491) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
Caused by: java.lang.IllegalArgumentException
        at java.nio.Buffer.position(Buffer.java:244) [rt.jar:1.8.0_121]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:418) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        ... 10 more

        1. test.log
          3.94 MB
        2. node-2-logs.zip
          186 kB
        3. node-1-logs.zip
          939 kB
        4. journal.zip
          292 kB

              csuconic@redhat.com Clebert Suconic
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: