[jms-pool] Memory leak when to use camel-jms consumer with CACHE_SESSSION and pooled connection(messaginghub/pooled-jms)

[Problem]

• Memory leak when to use camel-jms consumer with CACHE_SESSSION and pooled connection(messaginghub/pooled-jms)
  • A simple camel route like the following will reproduce the problem:
    • When to use cacheLevelName=CACHE_SESSSION : <to uri="jms:queue:requestqueue?cacheLevelName=CACHE_SESSSION"/>
    • Or When to use replyToType=shared : <to uri="jms:queue:requestqueue?replyTo=replyqueue&replyToType=shared"/>
      • When to use replyToType=shared, the cacheLevel will be “CACHE_SESSSION” as default .
  • org.ops4j.pax.jms.service.PooledConnectionFactoryFactory use messaginghub/pooled-jms internally.
• With leaks by this problem, the number of consumers(org.messaginghub.pooled.jms.JmsPoolMessageConsumer and its raw javax.jms.MessageConsumer objects) continues to gradually increase, and the memory continues to grow.
  • If lucky, Fuse will freeze for several tens of seconds when heap size reaches its maximum. Then, due to various timeout errors[1], the connection is closed, the resource is released. And the leaks begin repeatedly.
  • If unlucky, Fuse will crash with out of memory error.
  • When to use replyToType=shared:
    • If the heap is about 7GB, 50000 Consumers will be created to be out of memory error.
    • When to sends 1000 messages per second, the errors(several kind of timeout errors etc,.) by insufficient memory are reproduced in about three minute.
  • When to use cacheLevelName=CACHE_SESSSION:
    • It takes more time to reproduce because the consumer uses less memory than when replyToType=shared.
• FYI, I've tested several patterns:
  • [PROBLEM       ] camel-jms request/reply route ===> messaginghub/pooled-jms connection pool  ===> Artemis connection factory ===> Artemis server
  • [PROBLEM       ] camel-jms request/reply route ===> messaginghub/pooled-jms connection pool  ===> Classic connection factory ===>  Classic server
  • [NO PROBLEM] camel-jms request/reply route ===> Classic connection pool  ===> Classic connection factory ===>  Classic server
  • [NO PROBLEM] camel-jms request/reply route ===> Artemis connection factory ===> Artemis server
  • [NO PROBLEM] camel-jms request/reply route ===> Classic connection factory  ===> Classic server
  • ※ explanatory notes
    • messaginghub/pooled-jms connection pool = org.messaginghub.pooled.jms.JmsPoolConnectionFactory
    • Classic connection pool = org.apache.activemq.pool.PooledConnectionFactory
    • Artemis connection factory = org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory
    • Classic connection factory = org.apache.activemq.ActiveMQConnectionFactory

[Memory leak location]

• The session(org.messaginghub.pooled.jms.JmsPoolSession) in the camel-jms consumer holds a large number of the closed consumers(org.messaginghub.pooled.jms.JmsPoolMessageConsumer and its raw javax.jms.MessageConsumer objects) in the ”consumers” list property.
  • messaginghub/pooled-jms creates proxies(JmsPoolSession and JmsPoolMessageConsumer) to intercept the calls for sessions and consumers internally.
    • org.messaginghub.pooled.jms.JmsPoolSession is a proxy to intercept the calls for a raw javax.jms.Session.
    • org.messaginghub.pooled.jms.JmsPoolMessageConsumer is a proxy to intercept the calls for a raw javax.jms.MessageConsumer.
  • https://github.com/messaginghub/pooled-jms/blob/67d24ede922adcd0fd7dc2959bcd242ee7e5e7b8/pooled-jms/src/main/java/org/messaginghub/pooled/jms/JmsPoolSession.java#L64

[Problem cause]

• There seems to be a bug in the messaginghub/pooled-jms connection pool.
  • In the case of “CACHE_SESSSION”(including cases using replyToType=shared), the session is kept(cached) by camel-jms, and its consumers are created and closed repeatedly(from the Spring JmsTemplate as expected).
  • However, even after closing a consumer(org.messaginghub.pooled.jms.JmsPoolMessageConsumer and its raw javax.jms.MessageConsumer objects), the session proxy(org.messaginghub.pooled.jms.JmsPoolSession) keeps to hold the reference to the consumer. So the references to the consumers keep growing.
  • At first glance, it looks like when the consumer is closed, an event(onConsumerClose) is sent to the session proxy to remove the reference to the consumer.
    • https://github.com/messaginghub/pooled-jms/blob/67d24ede922adcd0fd7dc2959bcd242ee7e5e7b8/pooled-jms/src/main/java/org/messaginghub/pooled/jms/JmsPoolMessageConsumer.java#L53
  • However, there is a bug in this part, the session proxy removes the consumer reference using the raw javax.jms.MessageConsumer, but it should be the consumer proxy(org.messaginghub.pooled.jms.JmsPoolMessageConsumer)
    • https://github.com/messaginghub/pooled-jms/blob/67d24ede922adcd0fd7dc2959bcd242ee7e5e7b8/pooled-jms/src/main/java/org/messaginghub/pooled/jms/JmsPoolSession.java#L491

[Environment]

• Red Hat Fuse 7.8 on Karaf
  • camel-jms
  • org.ops4j.pax.jms.service.PooledConnectionFactoryFactory(messaginghub/pooled-jms) is used
• Red Hat AMQ Broker 7.8 as a messaging broker