-
Bug
-
Resolution: Done
-
Blocker
-
4.2.0.Final
-
None
Commit f948c45f4 added on 15 July 2019, introduced a cache for more quicker matching of incoming request URL to the controller. This cache has certain problems:
- it is not limited in size, so can grow infinitely
- there is no possibility to switch off the cache, as the flag CACHE is private static.
In below scenario we are creating requests with the following content type:
Content-type: multipart/form-data;boundary=bTHq6EqLI8B4NCLpr_stfilQniLLQOGKlJ
were boundary is randomly generated and changes with each request. Due to this fact the call to Objects.equals(contentType, key.contentType) never returns true, so there is always a cache miss situation. As a result:
- The cache grows infinitely which we can indirectly observe on heap dump where 4194320 cache entries have been created:
- Time to handle the match doubles (half is spent to get the entry, half to put an entry):
Such cache size wouldn't be a problem if it wouldn't be java.util.concurrent.ConcurrentHashMap that locks the map before adding a new entry into it. As the result under high load (3000+ threads) all threads (except one which tries to match request against thousands of baskets) get stuck with the following stack trace:
"ajp-/10.1.1.2:8009-3067" #7077674 daemon prio=5 os_prio=0 tid=0x00007f3bd922d800 nid=0x148d3 waiting for monitor entry [0x00007f3acaeeb000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1027) - waiting to lock <0x00000006e09f8cb8> (a java.util.concurrent.ConcurrentHashMap$TreeBin) at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535) at org.jboss.resteasy.core.registry.RootNode.match(RootNode.java:62) at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:47) at org.jboss.resteasy.core.ResourceMethodRegistry.getResourceInvoker(ResourceMethodRegistry.java:482) at org.jboss.resteasy.core.SynchronousDispatcher.getInvoker(SynchronousDispatcher.java:323) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:244) at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$212/1356597929.run(Unknown Source) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153) at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$215/989281117.get(Unknown Source) at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
Slow service performance degradation over the time:
Solutions:
- Make cache on/off switching configurable.
- Limit the size of the cache.
- Implement correct matching of multipart/form-data content type.
All RestEasy versions from v4.2.0 till v4.5.5 are affected.
- is documented by
-
RESTEASY-3035 Add documentation for match_cache parameters
-
- Resolved
-
- relates to
-
WFLY-14559 Add six new RESTEasy context parameters
-
- Open
-
