Uploaded image for project: 'A-MQ Broker'
  1. A-MQ Broker
  2. ENTMQBR-1123

[Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)

    Details

    • Type: Bug
    • Status: Done
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: AMQ 7.2.0.GA
    • Component/s: journal
    • Labels:
    • Target Release:
    • Steps to Reproduce:
      Hide

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
      cd eap-tests-hornetq/scripts/
      git checkout f3de9baf1f8b39b810bf3d55d45f4341e3b2aa87

      groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/217/artifact/jboss-eap.zip PrepareServers7.groovy
      export WORKSPACE=$PWD
      export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
      export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
      export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
      export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
      cd ../jboss-hornetq-testsuite/
      ulimit -n 8196
      mvn clean test -Dtest=DurableSubscriptionsTestCase#largeByteMessagesTest -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1520950239-SNAPSHOT -DfailIfNoTests=false | tee log

      Show
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout f3de9baf1f8b39b810bf3d55d45f4341e3b2aa87 groovy -DEAP_ZIP_URL= https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/217/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ ulimit -n 8196 mvn clean test -Dtest=DurableSubscriptionsTestCase#largeByteMessagesTest -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1520950239-SNAPSHOT -DfailIfNoTests=false | tee log

      Description

      There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.

      Test Scenario:

      • Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
      • create durable subscriptions on testTopic
      • start sending large messages on testTopic
      • subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions

      Pass Criteria: Subscribers must receive correct number of messages.

      Actual Result:
      Artemis server shutdown/crashes itself on (ulimit for max number of open files was set to 8196):

      15:21:54,717 WARN  [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
              at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
      Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
              at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
              at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
              at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
              at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
              ... 15 more
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  ataylor Andy Taylor
                  Reporter:
                  mnovak Miroslav Novak
                  Tester:
                  Roman Vais
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: