Uploaded image for project: 'RESTEasy'
  1. RESTEasy
  2. RESTEASY-2646

Cache implementation in class RootNode is vulnerable to DoS attack

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: 4.2.0.Final
    • Fix Version/s: 4.5.6.Final
    • Component/s: jaxrs
    • Labels:
      None

      Description

      Commit f948c45f4 added on 15 July 2019, introduced a cache for more quicker matching of incoming request URL to the controller. This cache has certain problems:

      • it is not limited in size, so can grow infinitely
      • there is no possibility to switch off the cache, as the flag CACHE is private static.

      In below scenario we are creating requests with the following content type:

      Content-type: multipart/form-data;boundary=bTHq6EqLI8B4NCLpr_stfilQniLLQOGKlJ
      

      were boundary is randomly generated and changes with each request. Due to this fact the call to Objects.equals(contentType, key.contentType) never returns true, so there is always a cache miss situation. As a result:

      • The cache grows infinitely which we can indirectly observe on heap dump where 4194320 cache entries have been created:
      • Time to handle the match doubles (half is spent to get the entry, half to put an entry):

      Such cache size wouldn't be a problem if it wouldn't be java.util.concurrent.ConcurrentHashMap that locks the map before adding a new entry into it. As the result under high load (3000+ threads) all threads (except one which tries to match request against thousands of baskets) get stuck with the following stack trace:

      "ajp-/10.1.1.2:8009-3067" #7077674 daemon prio=5 os_prio=0 tid=0x00007f3bd922d800 nid=0x148d3 waiting for monitor entry [0x00007f3acaeeb000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1027)
      	- waiting to lock <0x00000006e09f8cb8> (a java.util.concurrent.ConcurrentHashMap$TreeBin)
      	at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
      	at org.jboss.resteasy.core.registry.RootNode.match(RootNode.java:62)
      	at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:47)
      	at org.jboss.resteasy.core.ResourceMethodRegistry.getResourceInvoker(ResourceMethodRegistry.java:482)
      	at org.jboss.resteasy.core.SynchronousDispatcher.getInvoker(SynchronousDispatcher.java:323)
      	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:244)
      	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$212/1356597929.run(Unknown Source)
      	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153)
      	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$215/989281117.get(Unknown Source)
      	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
      

      Slow service performance degradation over the time:

      Solutions:

      • Make cache on/off switching configurable.
      • Limit the size of the cache.
      • Implement correct matching of multipart/form-data content type.

      All RestEasy versions from v4.2.0 till v4.5.5 are affected.

        Gliffy Diagrams

          Attachments

          1. grafana.png
            grafana.png
            164 kB
          2. grafana.png
            grafana.png
            187 kB
          3. heap.png
            heap.png
            91 kB
          4. profile.png
            profile.png
            278 kB

            Issue Links

              Activity

                People

                • Assignee:
                  jim.ma Jim Ma
                  Reporter:
                  dma_k Dmitry Katsubo
                  Involved:
                  James Perkins, Jim Ma, Ronald Sigal, r searls, Weinan Li
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: