Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-16256

Tracer Instance Leak in WildFlyClientTracingRegistrarProvider

XMLWordPrintable

    • Hide

      A reproducer is available here: https://github.com/Spindl/WFLY-16256_reproducer

      How to run the reproducer

      1. Build the WildFly bootable jar by running mvn clean package
      2. Run the server with java -Xmx64m -jar target/app-local-SNAPSHOT-bootable.jar
        1. It's not strictly necessary to limit the heap space, but this way it's easier to spot the leaked instances in a dump
        2. Also the client will be done faster with filling the heap
      3. Run the client with java src/main/java/main/Main.java

      The client will fill the server heap up to a configured threshold value.

      Show
      A reproducer is available here: https://github.com/Spindl/WFLY-16256_reproducer How to run the reproducer Build the WildFly bootable jar by running  mvn clean package Run the server with  java -Xmx64m -jar target/app-local-SNAPSHOT-bootable.jar It's not strictly necessary to limit the heap space, but this way it's easier to spot the leaked instances in a dump Also the client will be done faster with filling the heap Run the client with  java src/main/java/main/Main.java The client will fill the server heap up to a configured threshold value.
    • ---
    • ---

      Overview

      When analyzing a heap dump for a potential memory leak, we noticed a huge amount of contextual instances of a Tracer hanging on the  creational context of the TracingCDIExtension from org.wildfly.microprofile.opentracing-smallrye.

      We are using WildFly 24.0.0.Final.

      Further tests showed that the instances are never cleared and thus eventually will consume all available heap.

      Here is a Screenshot of the heap dump. Almost all of the 166779 elements are contextual instances containing the same JaegerTracer.

       

      What we found out so far

      During more detailed analysis we found that the leak occurs in the WildFlyClientTracingRegistrarProvider, and is kind of similar to the leak that was reported here https://issues.redhat.com/browse/WFLY-14094

      The new instance is created whenever a ClientBuilder is configured with tracing via the ClientTracingRegistrar, and never deleted.

      The reason for that seems to be that the instance is created via CDI.current().select(Tracer.class).get(), which creates a new dependent instance because the JaegerTracer does not have a scope. 
      The bean to which the instance added in the CreationalContext is the TracingCDIExtension, since it is registered as Bean with the type Tracer on the AfterBeanDiscovery event. As CDI Extension, it is kind of ApplicationScoped, meaning that it will never be destroyed during the execution of the application. Therefore the instances will never vanish either.

      Since the Tracer instance is not accessible in the ClientTracingFeature (where it's used later on) there is nothing we can do to fix this problem on our side. If you have any suggestions we are glad to hear them.

            jaslee@redhat.com Jason Lee
            roland.spindelbalker@ntsretail.com Roland Spindelbalker-Davila
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: