-
Bug
-
Resolution: Done
-
Major
-
None
-
AMQ 7.11.4.GA
Out of 3 clustered broker, one of broker ran out of memory due to large number of ConcurrentLinkedQueue objects.
Retained Set
Class Name | Objects | Shallow Heap | Retained Heap -------------------------------------------------------------------------------------------------------------------------------------- java.util.concurrent.ConcurrentLinkedQueue | 4,452,254 | 106,854,096 | >= 14,225,796,312 java.util.concurrent.ConcurrentLinkedQueue$Node | 65,425,283 | 1,570,206,792 | >= 14,043,195,760 org.apache.activemq.artemis.utils.actors.Actor | 4,204,420 | 168,176,800 | >= 11,907,240,184 byte[] | 45,426,124 | 10,342,156,760 | >= 10,342,156,760 org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionSendMessage_V3| 5,694,092 | 318,869,152 | >= 6,382,698,128 org.apache.activemq.artemis.core.message.impl.CoreMessage | 5,714,889 | 640,067,568 | >= 6,098,445,400 --------------------------------------------------------------------------------------------------------------------------------------
Dominator Tree
Class Name | Shallow Heap | Retained Heap | Percentage ----------------------------------------------------------------------------------------------------------------------------------------------------------- org.apache.activemq.artemis.utils.actors.Actor @ 0x101ec5cf0 | 40 | 5,155,236,176 | 17.20% |- tasks java.util.concurrent.ConcurrentLinkedQueue @ 0x101ec5d18 | 24 | 5,155,236,120 | 17.20% | |- head java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x14154f918 | 24 | 5,155,234,728 | 17.20% | | '- next java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x14154f930 | 24 | 5,155,234,704 | 17.20% | |- tail java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x7fecfb320 | 24 | 1,368 | 0.00% | '- Total: 2 entries | | | |- listener org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$335 @ 0x101ec5d90| 16 | 16 | 0.00% '- Total: 2 entries | | | org.apache.activemq.artemis.utils.actors.Actor @ 0x128ec7a70 | 40 | 4,182,791,208 | 13.96% '- tasks java.util.concurrent.ConcurrentLinkedQueue @ 0x128ec7a98 | 24 | 4,182,791,152 | 13.96% |- head java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x1416ba520 | 24 | 4,182,789,560 | 13.96% |- tail java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x7ff06bae0 | 24 | 1,568 | 0.00% '- Total: 2 entries | | | -----------------------------------------------------------------------------------------------------------------------------------------------------------
Thread dump extracted from heap shows may number of threads with a similar stack
at org.apache.activemq.artemis.core.protocol.mqtt.MQTTConnectionManager.disconnect(Z)V (MQTTConnectionManager.java: at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.disconnectExistingSession(Lorg/apache/activemq/artemis/core/protocol/mqtt/MQTTConnection;)V (MQTTProtocolHandler.java:463)| | | | | | | | | at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handleConnect(Lio/netty/handler/codec/mqtt/MqttConnectMessage;)V (MQTTProtocolHandler.java:258) | | | | at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(Lio/netty/handler/codec/mqtt/MqttMessage;)V (MQTTProtocolHandler.java: at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler$$Lambda$5 at org.apache.activemq.artemis.utils.actors.Actor.doTask(Ljava/lang/Object;)V (Actor.java:32) | | | | | | | | | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks()V (ProcessorBase.java:68) | | | | at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$ at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java: at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:628) | | | | 48 | | | | | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
