Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-4926

JGroups ForkChannel can hold references to Hibernate SessionFactoryImpl which cause memory leak

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • 7.1.0.DR6
    • 7.0.1.CR1
    • Clustering
    • None
    • Hide

      1. deploy an application using JPA with Hibernate second level cache enabled
      2. use the second level cache (I'm note sure if this step is really needed)
      3. undeploy the application
      4. take a heap dump
      5. search for incoming references to SessionFactoryImpl

      Show
      1. deploy an application using JPA with Hibernate second level cache enabled 2. use the second level cache (I'm note sure if this step is really needed) 3. undeploy the application 4. take a heap dump 5. search for incoming references to SessionFactoryImpl

    Description

      We are using Hibernate second level cache through JPA configured as such:

          <properties>
              <property name="hibernate.cache.use_query_cache" value="true"/>
              <property name="hibernate.cache.use_second_level_cache" value="true"/>
              <property name="hibernate.cache.region.factory_class" value="org.jboss.as.jpa.hibernate5.infinispan.SharedInfinispanRegionFactory"/>
          </properties>
      

      After heap dump inspection, it seems that the JGroups ForkChannel identified by "hibernate" can hold a listener that hold the SessionFactoryImpl which then hold the whole application classloader.

      When undeploying the application, this can lead to classloader leak.
      I took an heap dump of such scenario and analysed it using eclipse memory analyzer (MAT) and here is the result:

      Class Name                                                                                                  | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      channel org.jgroups.JChannel @ 0xc0eac6a0                                                                   |            1 |          112 |               136 |           448
      '- channel_listeners java.util.concurrent.CopyOnWriteArraySet @ 0xc0ecd260                                  |            1 |           16 |               136 |           112
         '- al java.util.concurrent.CopyOnWriteArrayList @ 0xc0ecd270                                             |            1 |           24 |               136 |            96
            '- array java.lang.Object[2] @ 0xc0ecd2b8                                                             |            1 |           24 |               136 |            24
               '- [0] org.jgroups.fork.ForkChannel @ 0xc103d250                                                   |            1 |          120 |               136 |         1 328
                  '- channel_listeners java.util.concurrent.CopyOnWriteArraySet @ 0xc103d350                      |            1 |           16 |               136 |           968
                     '- al java.util.concurrent.CopyOnWriteArrayList @ 0xc103d360                                 |            1 |           24 |               136 |           952
                        '- array java.lang.Object[1] @ 0xc103d3a8                                                 |            1 |           24 |               136 |           880
                           '- [0] org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher @ 0xc103d1f0|            1 |           96 |               136 |           856
      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      

      To remove that leak I ugily patched JGroups ForkChannel close method as such:

          /** Closes the fork-channel, essentially setting its state to CLOSED. Note that - contrary to a regular channel -
           * a closed fork-channel can be connected again: this means re-attaching the fork-channel to the main-channel*/
          @Override
          public void close() {
              ((ForkProtocolStack)prot_stack).remove(fork_channel_id);
              if(state == State.CLOSED)
                  return;
              disconnect();                     // leave group if connected
              prot_stack.destroy();
              state=State.CLOSED;
              notifyChannelClosed(this);
      	this.clearChannelListeners(); // <-- this is the line I added
          }
      

      With that change in place, the memory leak is gone. I highly doubt though this is an acceptable fix. Though it does confirm my theory.

      I doubt that JGroups is really the culprit – I'm more in the thinking that the "thing managing" JGroups is the culprit.

      Since I'm not an expert around that field I've opened the issue against the Wildfly project. Feel free to move it to the proper project.

      If you need any other informations let me know.

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              pferraro@redhat.com Paul Ferraro
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: