After day or two Wildfly hangs and consumes 100% CPU

First disclaimer, this looks pointing to undertow/xnio but I am not sure for the root cause so I'm filing this for Wildfly.

After a day or two Wildfly just hangs and consumes the cpu. We have noticed that the site traffic itself has no impact on this: it has happened during peak traffic and also almost in idle situations.

We are handling SSL in Wildfly and example this https://issues.jboss.org/browse/UNDERTOW-282 looks similar to this but it has been resolved over year ago so it should be in Wildfly 9.01.

Our SSL conf in standalone.xml:
<https-listener name="https" socket-binding="https" security-realm="UndertowRealm" enabled-cipher-suites="TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" max-parameters="5000"/>

I have narrowed down the thread that hangs. Or at least tried, my first time to study thread dumps. PID for thread that is jammed is 22599 => 0x5847 as hex and it points to the thread end of this description. I attach the actual thread dumps to this ticket. Any workaround for this is preciated, we currently need to reboot the servers when the Wildfly hangs (almost every nigth).

"default task-20" #285 prio=5 os_prio=0 tid=0x00007f862c328800 nid=0x5847 runnable [0x00007f85f5ad6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)

• locked <0x00000006f6065210> (a java.lang.Object)
  at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:282)
  at org.xnio.ssl.JsseSslConduitEngine.handleUnwrapResult(JsseSslConduitEngine.java:737)
• locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer)
  at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:620)
• locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer)
  at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:574)
  at org.xnio.ssl.JsseSslStreamSourceConduit.read(JsseSslStreamSourceConduit.java:89)
  at org.xnio.conduits.AbstractStreamSourceConduit.read(AbstractStreamSourceConduit.java:51)
  at io.undertow.conduits.ReadDataStreamSourceConduit.read(ReadDataStreamSourceConduit.java:67)
  at io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:249)
  at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
  at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209)
  at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2136)
  at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.doParse(FormEncodedDataDefinition.java:133)
  at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:251)
  at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:753)
  at io.undertow.servlet.spec.HttpServletRequestImpl.getParameter(HttpServletRequestImpl.java:627)
  at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:45)
  at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:33)
  at org.apache.myfaces.util.AbstractAttributeMap.containsKey(AbstractAttributeMap.java:62)
  at org.apache.myfaces.renderkit.html.HtmlResponseStateManager.isPostback(HtmlResponseStateManager.java:303)
  at org.apache.myfaces.lifecycle.DefaultRestoreViewSupport.isPostback(DefaultRestoreViewSupport.java:242)
  at org.apache.myfaces.lifecycle.RestoreViewExecutor.execute(RestoreViewExecutor.java:120)
  at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:170)
  at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117)
  at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)
  at fi.elbit.skills.loginview.DefaultsFilter.doFilter(DefaultsFilter.java:117)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:357)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
  at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
  at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
  at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56)
  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
  at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
  at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:

• <0x00000006f20bb6c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)