Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-9900

JVM does not fully exit on storage loss

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • AMQ 7.13.0.GA
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      The following test was performed with an external drive used by the broker as the data directory.

      • Configure the broker to use an external drive for the data directory (journal)
      • Start the broker
      • Allow the broker to initialize and run successfully
      • Disconnect the external storage
      Show
      The following test was performed with an external drive used by the broker as the data directory. Configure the broker to use an external drive for the data directory (journal) Start the broker Allow the broker to initialize and run successfully Disconnect the external storage
    • Moderate

      The JVM fails to fully exit when the broker loses its storage.  The log will show the broker shutdown, however there are remaining threads that prevent the JVM from exiting.

      The following is captured from the broker:

      2025-07-23 13:21:09,904 WARN  [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] Lost the lock according to the monitor, notifying listeners
2025-07-23 13:21:11,910 ERROR [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] java.io.IOException: Input/output error
org.apache.activemq.artemis.core.server.NodeManager$NodeManagerException: java.io.IOException: Input/output error
    at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.getState(FileLockNodeManager.java:375) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager$MonitorLock.run(FileLockNodeManager.java:539) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForExecutor(ActiveMQScheduledComponent.java:326) ~[artemis-    commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.bookedRunForScheduler(ActiveMQScheduledComponent.java:341) ~[artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForScheduler(ActiveMQScheduledComponent.java:352) ~[artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.lambda$start$0(ActiveMQScheduledComponent.java:179) ~[artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) [?:?]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120) [artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
Caused by: java.io.IOException: Input/output error
    at java.base/sun.nio.ch.UnixFileDispatcherImpl.pread0(Native Method) ~[?:?]
    at java.base/sun.nio.ch.UnixFileDispatcherImpl.pread(UnixFileDispatcherImpl.java:57) ~[?:?]
    at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:339) ~[?:?]
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:295) ~[?:?]
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:284) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:1188) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:1152) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:1134) ~[?:?]
    at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.getState(FileLockNodeManager.java:360) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    ... 11 more
2025-07-23 13:21:11,921 WARN  [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] Lost the lock according to the monitor, notifying listeners
2025-07-23 13:21:13,686 ERROR [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=IO Error while calculating disk usage, message=NULL
java.nio.file.NoSuchFileException: /Volumes/NAS-01/data/paging
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
    at java.base/sun.nio.fs.UnixFileStore.readAttributes(UnixFileStore.java:115) ~[?:?]
    at java.base/sun.nio.fs.UnixFileStore.getUsableSpace(UnixFileStore.java:132) ~[?:?]
    at org.apache.activemq.artemis.core.server.files.FileStoreMonitor.tick(FileStoreMonitor.java:120) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.files.FileStoreMonitor.run(FileStoreMonitor.java:109) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForExecutor(ActiveMQScheduledComponent.java:326) ~[artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.lambda$bookedRunForScheduler$3(ActiveMQScheduledComponent.java:333) ~[artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:59) ~[artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) ~[artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) ~[artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120) [artemis-commons-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
2025-07-23 13:21:13,695 INFO  [org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl] Sending disconnect on client connections
2025-07-23 13:21:13,700 WARN  [org.apache.activemq.artemis.journal] AMQ144011: Critical IO Exception happened: Bad file descriptor on activemq-bindings-1.bindings
org.apache.activemq.artemis.api.core.ActiveMQIOErrorException: Bad file descriptor
    at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.close(NIOSequentialFile.java:231) ~[artemis-journal-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2884) ~[artemis-journal-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.internalStop(JournalStorageManager.java:325) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:275) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1385) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1227) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1220) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$stopTheServer$6(ActiveMQServerImpl.java:1027) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at java.base/java.lang.Thread.run(Thread.java:1575) [?:?]
Caused by: java.io.IOException: Bad file descriptor
    at java.base/sun.nio.ch.FileDispatcherImpl.force0(Native Method) ~[?:?]
    at java.base/sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:37) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.implForce(FileChannelImpl.java:602) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:617) ~[?:?]
    at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.close(NIOSequentialFile.java:220) ~[artemis-journal-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    ... 8 more
2025-07-23 13:21:13,700 ERROR [org.apache.activemq.artemis.core.server] AMQ224068: Unable to stop component: org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager
java.io.IOException: Bad file descriptor
    at java.base/sun.nio.ch.FileDispatcherImpl.force0(Native Method) ~[?:?]
    at java.base/sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:37) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.implForce(FileChannelImpl.java:602) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:617) ~[?:?]
    at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.close(NIOSequentialFile.java:220) ~[artemis-journal-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2884) ~[artemis-journal-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.internalStop(JournalStorageManager.java:325) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:275) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1385) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1227) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1220) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$stopTheServer$6(ActiveMQServerImpl.java:1027) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at java.base/java.lang.Thread.run(Thread.java:1575) [?:?]
2025-07-23 13:21:13,703 INFO  [org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl] Sending disconnect on client connections
2025-07-23 13:21:13,816 ERROR [org.apache.activemq.artemis.core.server] AMQ224068: Unable to stop component: org.apache.activemq.artemis.core.server.impl.PrimaryOnlyActivation
org.apache.activemq.artemis.core.server.NodeManager$NodeManagerException: java.io.IOException: Input/output error
    at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.writeFileLockStatus(FileLockNodeManager.java:347) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.setPaused(FileLockNodeManager.java:318) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.pausePrimaryServer(FileLockNodeManager.java:278) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.PrimaryOnlyActivation.close(PrimaryOnlyActivation.java:106) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1468) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1227) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1220) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$stopTheServer$6(ActiveMQServerImpl.java:1027) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    at java.base/java.lang.Thread.run(Thread.java:1575) [?:?]
Caused by: java.io.IOException: Input/output error
    at java.base/sun.nio.ch.UnixFileDispatcherImpl.pwrite0(Native Method) ~[?:?]
    at java.base/sun.nio.ch.UnixFileDispatcherImpl.pwrite(UnixFileDispatcherImpl.java:71) ~[?:?]
    at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:135) ~[?:?]
    at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:81) ~[?:?]
    at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:72) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:1256) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.implWrite(FileChannelImpl.java:1224) ~[?:?]
    at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:1206) ~[?:?]
    at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.writeFileLockStatus(FileLockNodeManager.java:337) ~[artemis-server-2.40.0.redhat-00004.jar:2.40.0.redhat-00004]
    ... 8 more
2025-07-23 13:21:13,816 INFO  [org.apache.activemq.artemis] AMQ241005: Stopping embedded web server
2025-07-23 13:21:13,822 INFO  [io.hawt.HawtioContextListener] Destroying Hawtio services
2025-07-23 13:21:13,824 INFO  [org.apache.activemq.artemis] AMQ241006: Stopped embedded web server
2025-07-23 13:21:13,824 INFO  [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.40.0.redhat-00004 [2003d26b-673e-11f0-b988-1e1f7ec2ce71] stopped, uptime 5 minutes

      At this point the JVM process is still running and never exits.  Thread dumps captured over a 20 min period have been attached.

        1. brokerDumps.zip
          8 kB

              csuconic@redhat.com Clebert Suconic
              rhn-support-jsherman Jason Sherman
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: