Uploaded image for project: 'RESTEasy'
  1. RESTEasy
  2. RESTEASY-2643

Caching routes in RootNode may result in DoS

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate Issue
    • Affects Version/s: 4.5.5.Final
    • Fix Version/s: None
    • Component/s: jaxrs
    • Labels:
      None
    • Security Sensitive Issue:
      This issue is security relevant

      Description

      In PR-2104 by Bill Burke introduced caching of route matchers. The MatchCache.Key's equality compares various route characteristics, such as the contentType. Unfortunately this cache is unbounded and cannot be disabled externally.

      A multipart request has a MediaType that includes the boundary's generated, unique identifier. The equality of two request's MediaType will not be equal due to this parameter, for example these instances are unequal: The hashCode uses only a subset of fields, excluding the MediaType.

      multipart/form-data; boundary=----WebKitFormBoundaryvXJi08Xkffqp7VvG
      multipart/form-data; boundary=----WebKitFormBoundaryjR9gcYfNh3jTG14Q
      

      This results in hash flooding as the key's hash collisions requires long traversals of the bin entries, which cannot be mitigated unless the key implements Comparable. The broken equality results in a cache miss that computes the matcher and inserts it, causing lock contention on the hash bin. Due to being an unbounded map, the operations get slower, burn cpu time, and leaks memory. This can eventually result in a denial of service (aka HashDoS).

      The mitigation is to use a @Path expression that includes groups, which is not yet supported by this caching logic and bypasses inserting the invalid entry. This is how I discovered the problem, caused by removing a legacy route pattern to its simplified version. The fix was to restore it, e.g.

      // Switched from example.com/api/ to api.example.com/1.0/, with proxy to forward
      // @Path("/{a:1.0/entities|api/1.0/entities}")
      // Removed extra group as transition had occurred long ago
      @Path("/1.0/entities")
      public final class CrudService {
        @PUT @Path("/records")
        @Consumes(MediaType.MULTIPART_FORM_DATA)
        public Entity create(@Context HttpHeaders httpHeaders, MultipartFormDataInput multipart) 
      

      This removal of tech debt resulted in high CPU usage across a fleet of machines, exacerbated due to ETL imports making api calls. The high cpu machines showed a jstack showed high lock contention on the hashbin.

      "PUT /1.0/entities/records" #95518 prio=5 os_prio=0 cpu=1511.52ms elapsed=222.62s tid=0x00007fcf00ac7000 nid=0x4c00 waiting for monitor entry  [0x00007fcec06c4000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at java.util.concurrent.ConcurrentHashMap.putVal(java.base@14/ConcurrentHashMap.java:1031)
      	- waiting to lock <0x000000008542da18> (a java.util.concurrent.ConcurrentHashMap$TreeBin)
      	at java.util.concurrent.ConcurrentHashMap.putIfAbsent(java.base@14/ConcurrentHashMap.java:1541)
      	at org.jboss.resteasy.core.registry.RootNode.match(RootNode.java:62)
      	at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:47)
      	at org.jboss.resteasy.core.ResourceMethodRegistry.getResourceInvoker(ResourceMethodRegistry.java:481)
      	at org.jboss.resteasy.core.SynchronousDispatcher.getInvoker(SynchronousDispatcher.java:330)
      	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:251)
      	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$1772/0x0000000801befc40.run(Unknown Source)
      	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:160)
      	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$1773/0x0000000801bef040.get(Unknown Source)
      	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:362)
      	- locked <0x00000000bc6003c8> (a org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext)
      	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:163)
      	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:245)
      	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:249)
      	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:60)
      	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      

      Recommendations:
      1. Fix the MatchCache.Key equality method
      2. Allow for disabling or configuring this cache - do not leave unbounded and hidden

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  ben-manes Benjamin Manes
                  Involved:
                  Brad Maxwell, Flavia Rainone
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: