Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-1123

[Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • AMQ 7.2.0.GA
    • None
    • journal
    • Hide

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
      cd eap-tests-hornetq/scripts/
      git checkout f3de9baf1f8b39b810bf3d55d45f4341e3b2aa87

      groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/217/artifact/jboss-eap.zip PrepareServers7.groovy
      export WORKSPACE=$PWD
      export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
      export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
      export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
      export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
      cd ../jboss-hornetq-testsuite/
      ulimit -n 8196
      mvn clean test -Dtest=DurableSubscriptionsTestCase#largeByteMessagesTest -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1520950239-SNAPSHOT -DfailIfNoTests=false | tee log

      Show
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout f3de9baf1f8b39b810bf3d55d45f4341e3b2aa87 groovy -DEAP_ZIP_URL= https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/217/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ ulimit -n 8196 mvn clean test -Dtest=DurableSubscriptionsTestCase#largeByteMessagesTest -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1520950239-SNAPSHOT -DfailIfNoTests=false | tee log

    Description

      There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.

      Test Scenario:

      • Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
      • create durable subscriptions on testTopic
      • start sending large messages on testTopic
      • subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions

      Pass Criteria: Subscribers must receive correct number of messages.

      Actual Result:
      Artemis server shutdown/crashes itself on (ulimit for max number of open files was set to 8196):

      15:21:54,717 WARN  [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
        at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
        at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
        at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
        ... 15 more

      Attachments

        Issue Links

          blocks

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. WFLY-10320 Upgrade artemis from 1.5.x to 2.x.x

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          clones

          Bug - A problem that impairs or prevents the functions of the product. WFLY-10022 [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-2670 Too many opened FDs after server stops

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          relates to

          ARTEMIS-1754 Loading...

          Activity

            People

              rh-ee-ataylor Andy Taylor
              mnovak1@redhat.com Miroslav Novak
              Roman Vais Roman Vais
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: