Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-9212

AMQ 7: ClosedChannelException with NFS

XMLWordPrintable

    • Critical

      This is an active/passive AMQ 7 set-up with NFS shared store.

      After an upgrade from 7.10.2 to 7.11.6, at unpredictable times, but under high production load, AMQ 7 emits this exception:

      2024-06-16 03:18:26,534 WARN  [org.apache.activemq.artemis.core.io.AbstractSequentialFile] null
      java.nio.channels.ClosedChannelException: null
              at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:159) ~[?:?]
              at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:459) ~[?:?]
              at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.sync(NIOSequentialFile.java:322) ~[artemis-journal-2.28.0.redhat-00019.jar:2.28.0.redhat-00019]
              at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.checkSync(AbstractSequentialFile.java:304) ~[artemis-journal-2.28.0.redhat-00019.jar:2.28.0.redhat-00019]
              at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:362) ~[artemis-journal-2.28.0.redhat-00019.jar:2.28.0.redhat-00019]
              at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:306) ~[artemis-journal-2.28.0.redhat-00019.jar:2.28.0.redhat-00019]
              at org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:482) ~[artemis-journal-2.28.0.redhat-00019.jar:2.28.0.redhat-00019]
              at java.lang.Thread.run(Thread.java:840) ~[?:?]

      Thereafter, it continues to emit this warning, several times a second, indefinitely:

      2024-06-18 12:01:24,868 WARN  [org.apache.activemq.artemis.core.server] AMQ222104: IO Error completing the transaction, code = 6, message = null
      2024-06-18 12:01:24,868 WARN  [org.apache.activemq.artemis.core.server] AMQ222063: Error processing IOCallback code = 6 message = null

      The broker may be paging on a few addresses when this happens, but that's normal in this set-up. There's nothing else in the log to indicate a problem, even going back hours. There's nothing in the log of the broker host system, or in the log of the NFS server.

      Once this problem arises, the broker does seem to be able to provide a service, but with severely reduced throughput. It takes a broker restart to clear the problem.

              csuconic@redhat.com Clebert Suconic
              rhn-support-kboone Kevin Boone
              Tiago Bueno Tiago Bueno
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: