Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-14656

Particular HTTP/2 Ajax requests to JSF resources yield "default task" threads stuck within AbstractFramedStreamSinkChannel.awaitWritable()

    XMLWordPrintable

Details

    • Hide

      1. Deploy the example project to a vanilla Wildfly server.
      2. Visit https://localhost:8443/wftest/index.xhtml using Firefox. Must be https so that HTTP/2 is used.
      3. Click "Auto inc".
      4. Watch the number of stuck threads increase: watch 'jstack <pid> | grep 'AbstractFramedStreamSinkChannel.awaitWritable' | wc -l'
      5. Repeat 2.-4. with Chrome, but number of stuck threads won't increase.

      JSF page

      index.xhtml
      <html xmlns="http://www.w3.org/1999/xhtml"
  xmlns:f="http://xmlns.jcp.org/jsf/core"
  xmlns:h="http://xmlns.jcp.org/jsf/html"
  xmlns:p="http://primefaces.org/ui">

<h:head>
</h:head>

<h:body>
 <h:form>
  <p:commandButton value="Inc">
   <p:ajax listener="#{foo.inc}" update="@all"/>
  </p:commandButton>

  <p:commandButton value="Auto inc" onclick="inc()" style="margin-left: 0.5rem"/>
  <p:remoteCommand name="inc" actionListener="#{foo.inc}" update="@all" oncomplete="setTimeout(inc, 0);"/>

  <p>
   <h:outputText id="label" value="#{foo.value}"/>
  </p>
 </h:form>
</h:body>
</html>

      Backing bean

      Foo.java
      package com.example;

import javax.faces.view.ViewScoped;
import javax.inject.Named;
import java.io.Serializable;


@Named
@ViewScoped
public class Foo implements Serializable {
 private int value;


 public int getValue() {
  return value;
 }


 public void setValue(int value) {
  this.value=value;
 }


 public void inc() {
  this.value++;
 }
}
      Show
      1. Deploy the example project to a vanilla Wildfly server. 2. Visit https://localhost:8443/wftest/index.xhtml using Firefox. Must be https so that HTTP/2 is used. 3. Click "Auto inc". 4. Watch the number of stuck threads increase: watch 'jstack <pid> | grep 'AbstractFramedStreamSinkChannel.awaitWritable' | wc -l' 5. Repeat 2.-4. with Chrome, but number of stuck threads won't increase. JSF page index.xhtml <html xmlns= "http://www.w3.org/1999/xhtml" xmlns:f = "http://xmlns.jcp.org/jsf/core" xmlns:h = "http://xmlns.jcp.org/jsf/html" xmlns:p = "http://primefaces.org/ui" > <h:head> </h:head> <h:body> <h:form> <p:commandButton value= "Inc" > <p:ajax listener= "#{foo.inc}" update= "@all" /> </p:commandButton> <p:commandButton value= "Auto inc" onclick= "inc()" style= "margin-left: 0.5rem" /> <p:remoteCommand name= "inc" actionListener= "#{foo.inc}" update= "@all" oncomplete= "setTimeout(inc, 0);" /> <p> <h:outputText id= "label" value= "#{foo.value}" /> </p> </h:form> </h:body> </html> Backing bean Foo.java package com.example; import javax.faces.view.ViewScoped; import javax.inject.Named; import java.io.Serializable; @Named @ViewScoped public class Foo implements Serializable { private int value; public int getValue() { return value; } public void setValue( int value) { this .value=value; } public void inc() { this .value++; } }
    • Undefined

    Description

      Particular HTTP/2 Ajax requests to JSF resources yield "default task" threads stuck within AbstractFramedStreamSinkChannel.awaitWritable().

      The requests are initiated by PrimeFaces, typically when updating @all components or opening global dialogs. They cause some .js or .css resources to be (re)loaded.

      It only happens when using Firefox and HTTP/2.

      In production, too many of those threads sometimes caused all "default I/O" threads to consume 100% CPU and Wildfly to become unresponsive. I tried setting various timeouts for the listener, but it didn't help. The threads disappear after a while, though. I noticed this first with Wildfly 20.0.1.Final and can still reproduce it with 23.0.0.Final.

      I attached a minimal example project to reproduce the issue (see "Steps to Reproduce").

      Stuck thread

      "default task-207" #1724 prio=5 os_prio=0 cpu=33,17ms elapsed=705,57s tid=0x000055d9e5b89000 nid=0x1f898 in Object.wait()  [0x00007f59bd2e0000]
   java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(java.base@11.0.10/Native Method)
 - waiting on <no object reference available>
 at java.lang.Object.wait(java.base@11.0.10/Object.java:328)
 at io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.awaitWritable(AbstractFramedStreamSinkChannel.java:289)
 - waiting to re-lock in wait() <0x000000043add1860> (a java.lang.Object)
 at org.xnio.conduits.StreamSinkChannelWrappingConduit.awaitWritable(StreamSinkChannelWrappingConduit.java:99)
 at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
 at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)
 at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:2115)
 at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:578)
 at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:151)
 at java.nio.channels.Channels$WritableByteChannelImpl.write(java.base@11.0.10/Channels.java:464)
 - locked <0x000000043adda930> (a java.lang.Object)
 at com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:318)
 at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:171)
 at org.primefaces.application.resource.PrimeResourceHandler.handleResourceRequest(PrimeResourceHandler.java:87)
 at javax.faces.webapp.FacesServlet.executeLifecyle(FacesServlet.java:704)
 at javax.faces.webapp.FacesServlet.service(FacesServlet.java:451)
 at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
 at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
 at io.opentracing.contrib.jaxrs2.server.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:52)
 at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
 at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
 at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
 at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
 at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
 at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
 at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
 at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
 at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
 at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
 at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
 at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
 at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
 at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
 at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
 at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
 at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
 at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
 at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
 at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
 at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
 at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
 at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
 at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
 at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
 at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
 at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
 at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
 at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
 at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
 at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
 at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction$$Lambda$1336/0x0000000841087040.call(Unknown Source)
 at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
 at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1337/0x0000000841087440.call(Unknown Source)
 at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
 at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1337/0x0000000841087440.call(Unknown Source)
 at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
 at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1337/0x0000000841087440.call(Unknown Source)
 at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
 at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1337/0x0000000841087440.call(Unknown Source)
 at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
 at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
 at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
 at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
 at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
 at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
 at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
 at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
 at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
 at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
 at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)

   Locked ownable synchronizers:
 - None

      Attachments

        Issue Links

          is caused by

          Bug - A problem that impairs or prevents the functions of the product. UNDERTOW-1910 CVE-2021-3629 AbstractFramedStreamSinkChannel.awaitWritable() never returns if peer fails to update HTTP2 window

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Resolved
          is incorporated by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. WFCORE-5598 Upgrade Undertow to 2.2.12.Final

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. WFLY-15363 Upgrade WildFly Core to 17.0.0.Final

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              flaviarnn Flavia Rainone
              foobartender S E (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: