Uploaded image for project: 'RESTEasy'
  1. RESTEasy
  2. RESTEASY-2646

Cache implementation in class RootNode is vulnerable to DoS attack


    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • 4.5.6.Final
    • 4.2.0.Final
    • jaxrs
    • None

      Commit f948c45f4 added on 15 July 2019, introduced a cache for more quicker matching of incoming request URL to the controller. This cache has certain problems:

      • it is not limited in size, so can grow infinitely
      • there is no possibility to switch off the cache, as the flag CACHE is private static.

      In below scenario we are creating requests with the following content type:

      Content-type: multipart/form-data;boundary=bTHq6EqLI8B4NCLpr_stfilQniLLQOGKlJ

      were boundary is randomly generated and changes with each request. Due to this fact the call to Objects.equals(contentType, key.contentType) never returns true, so there is always a cache miss situation. As a result:

      • The cache grows infinitely which we can indirectly observe on heap dump where 4194320 cache entries have been created:
      • Time to handle the match doubles (half is spent to get the entry, half to put an entry):

      Such cache size wouldn't be a problem if it wouldn't be java.util.concurrent.ConcurrentHashMap that locks the map before adding a new entry into it. As the result under high load (3000+ threads) all threads (except one which tries to match request against thousands of baskets) get stuck with the following stack trace:

      "ajp-/" #7077674 daemon prio=5 os_prio=0 tid=0x00007f3bd922d800 nid=0x148d3 waiting for monitor entry [0x00007f3acaeeb000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1027)
      	- waiting to lock <0x00000006e09f8cb8> (a java.util.concurrent.ConcurrentHashMap$TreeBin)
      	at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
      	at org.jboss.resteasy.core.registry.RootNode.match(RootNode.java:62)
      	at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:47)
      	at org.jboss.resteasy.core.ResourceMethodRegistry.getResourceInvoker(ResourceMethodRegistry.java:482)
      	at org.jboss.resteasy.core.SynchronousDispatcher.getInvoker(SynchronousDispatcher.java:323)
      	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:244)
      	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$212/1356597929.run(Unknown Source)
      	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153)
      	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$215/989281117.get(Unknown Source)
      	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)

      Slow service performance degradation over the time:


      • Make cache on/off switching configurable.
      • Limit the size of the cache.
      • Implement correct matching of multipart/form-data content type.

      All RestEasy versions from v4.2.0 till v4.5.5 are affected.

        1. grafana.png
          164 kB
        2. grafana.png
          187 kB
        3. heap.png
          91 kB
        4. profile.png
          278 kB

            rhn-engineering-ema Jim Ma
            dma_k Dmitry Katsubo (Inactive)
            James Perkins, Jim Ma, Ronald Sigal, r searls, Weinan Li
            0 Vote for this issue
            3 Start watching this issue