Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-9004

Artemis leaks JGroups channels on reload

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • 11.0.0.Alpha1
    • 11.0.0.Alpha1
    • Clustering
    • None
    • Hide

      Steps to reproduce - note that test does not fail and is necessary to use visualvm or another tool to track number of threads:

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout master

groovy -DEAP_VERSION=7.1.0.ER1 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/
mvn clean test -Dtest=MdbWithCdiTestCase#testSendReceiveReloadMdbWithCdi -DfailIfNoTests=false -Deap=7x  | tee log
      Show
      Steps to reproduce - note that test does not fail and is necessary to use visualvm or another tool to track number of threads : git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout master groovy -DEAP_VERSION=7.1.0.ER1 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=MdbWithCdiTestCase#testSendReceiveReloadMdbWithCdi -DfailIfNoTests= false -Deap=7x | tee log

    Description

      Customer Impact: If server is reloaded multiple times then there is growing number of threads. This can crash server on OutOfMemoryError. Also threads continuously increases memory footprint.

      If server is reloaded multiple times then there seems to be infinite grow in numbers of following 2 threads:

      "thread-2,shared=<CLUSTER>" #158395 daemon prio=5 os_prio=0 tid=0x00007fb5e351c000 nid=0x2672 waiting on condition [0x00007fb5cabaf000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000fc9e6d48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
	at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory$$Lambda$868/1712469388.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:744)

      "Timer runner-1,shared=<CLUSTER>" #158327 daemon prio=5 os_prio=0 tid=0x00007fb6bdd50800 nid=0x262b runnable [0x00007fb5d5d5e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000fc9e76f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.DelayQueue.take(DelayQueue.java:223)
	at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
	at org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:166)
	at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
	at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory$$Lambda$868/1712469388.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:744)

      There seems to be thread pool executor which is creating new and new thread after each reload waiting for a task to be executed. However number of those threads is not going down and most likely there is not set maximum limit.

      Attaching 2 thread dumps where diff between threads can be made. I've used script https://github.com/dudaerich/scripts/blob/master/src/thread-dump-analyzer.groovy to get number of threads from each thread pool and then made a diff.

      Attachments

        Issue Links

          causes

          Bug - A problem that impairs or prevents the functions of the product. WFLY-9246 discovery-group does not work for pooled-connection-factory because of NPE

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          clones

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-11799 Artemis leaks JGroups channels on reload

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Verified

          Activity

            People

              rhn-engineering-rhusar Radoslav Husar
              rhn-engineering-rhusar Radoslav Husar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: