Uploaded image for project: 'RESTEasy'
  1. RESTEasy
  2. RESTEASY-2646

Cache implementation in class RootNode is vulnerable to DoS attack

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: 4.2.0.Final
    • Fix Version/s: 4.5.6.Final
    • Component/s: jaxrs
    • Labels:
      None

      Description

      Commit f948c45f4 added on 15 July 2019, introduced a cache for more quicker matching of incoming request URL to the controller. This cache has certain problems:

      • it is not limited in size, so can grow infinitely
      • there is no possibility to switch off the cache, as the flag CACHE is private static.

      In below scenario we are creating requests with the following content type:

      Content-type: multipart/form-data;boundary=bTHq6EqLI8B4NCLpr_stfilQniLLQOGKlJ
      

      were boundary is randomly generated and changes with each request. Due to this fact the call to Objects.equals(contentType, key.contentType) never returns true, so there is always a cache miss situation. As a result:

      • The cache grows infinitely which we can indirectly observe on heap dump where 4194320 cache entries have been created:
      • Time to handle the match doubles (half is spent to get the entry, half to put an entry):

      Such cache size wouldn't be a problem if it wouldn't be java.util.concurrent.ConcurrentHashMap that locks the map before adding a new entry into it. As the result under high load (3000+ threads) all threads (except one which tries to match request against thousands of baskets) get stuck with the following stack trace:

      "ajp-/10.1.1.2:8009-3067" #7077674 daemon prio=5 os_prio=0 tid=0x00007f3bd922d800 nid=0x148d3 waiting for monitor entry [0x00007f3acaeeb000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1027)
      	- waiting to lock <0x00000006e09f8cb8> (a java.util.concurrent.ConcurrentHashMap$TreeBin)
      	at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
      	at org.jboss.resteasy.core.registry.RootNode.match(RootNode.java:62)
      	at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:47)
      	at org.jboss.resteasy.core.ResourceMethodRegistry.getResourceInvoker(ResourceMethodRegistry.java:482)
      	at org.jboss.resteasy.core.SynchronousDispatcher.getInvoker(SynchronousDispatcher.java:323)
      	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:244)
      	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$212/1356597929.run(Unknown Source)
      	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153)
      	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$215/989281117.get(Unknown Source)
      	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
      

      Slow service performance degradation over the time:

      Solutions:

      • Make cache on/off switching configurable.
      • Limit the size of the cache.
      • Implement correct matching of multipart/form-data content type.

      All RestEasy versions from v4.2.0 till v4.5.5 are affected.

        Attachments

        1. grafana.png
          grafana.png
          164 kB
        2. grafana.png
          grafana.png
          187 kB
        3. heap.png
          heap.png
          91 kB
        4. profile.png
          profile.png
          278 kB

          Issue Links

            Activity

              People

              Assignee:
              jim.ma Jim Ma
              Reporter:
              dma_k Dmitry Katsubo (Inactive)
              Involved:
              James Perkins, Jim Ma, Ronald Sigal, r searls, Weinan Li
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: