Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-2563

Broker shuts down because of file handle exhaustion during replication

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Major Major
    • None
    • AMQ 7.3.0.GA
    • broker-core
    • None

      A master-slave pair of brokers is running normally, under unexceptional load. For no clear reason, the slave broker reports "Too many open files" whilst receiving a file, and shuts down.

      2019-05-29 10:47:56,926 WARN  [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /opt/projects/jboss_amq_instances/foo/./data/large-messages/439266171135.msg, message=/opt/projects/jboss_amq_instances/foo/./data/large-messages/439266171135.msg: Too many open files: ActiveMQIOErrorException[errorType=IO_ERROR message=/opt/projects/jboss_amq_instances/foo/./data/large-messages/439266171135.msg: Too many open files]
        at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91) [artemis-journal-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:500) [artemis-server-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:479) [artemis-server-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.addBytes(LargeServerMessageImpl.java:150) [artemis-server-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.handleLargeMessageWrite(ReplicationEndpoint.java:602) [artemis-server-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:217) [artemis-server-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:715) [artemis-core-client-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:407) [artemis-core-client-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:384) [artemis-core-client-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]
        at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1191) [artemis-core-client-2.7.0.redhat-00056.jar:2.7.0.redhat-00056]

      At the time of the failure, the large-messages/ directory contained approximately 200 000 files. The problem was not seen on AMQ 7.1 or 7.2.2.

            rh-ee-ataylor Andy Taylor
            rhn-support-kboone Kevin Boone
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: