Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-31319

[GSS](8.1.z) WFCORE-7388 - org.wildfly.io.max-threads capability is subject to race conditions

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • 8.1 Update 3
    • 8.1.0.GA, 8.1 Update 0.1
    • IO
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      You can use byteman to force the problem timing to occur with a sleep delay injected to WorkerAdd.performRuntime:

      1. Launch JBoss using byteman with WFCORE-7388.btm rule file and WFCORE-7388.xml for its standalone configuration
      2. Observer the slsb pool size.  It should match the configured io worker task-max-threads of 42, but will instead show a different value derived from the CPU core count:
      15:22:22,178 INFO  [org.jboss.as.ejb3] (MSC service thread 1-6) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 64 (per class), which is derived from thread worker pool sizing.
      
      Show
      You can use byteman to force the problem timing to occur with a sleep delay injected to WorkerAdd.performRuntime: Launch JBoss using byteman with WFCORE-7388 .btm rule file and WFCORE-7388 .xml for its standalone configuration Observer the slsb pool size.  It should match the configured io worker task-max-threads of 42, but will instead show a different value derived from the CPU core count: 15:22:22,178 INFO  [org.jboss.as.ejb3] (MSC service thread 1-6) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 64 (per class), which is derived from thread worker pool sizing.

      org.wildfly.io.max-threads capability is subject to race conditions.  The capability is installed and available before an io worker is actually added.  A worker might typically be added more shortly after to prevent issues.  But depending upon timing, it's possible that something may check the max thread count provided from this capability before any actual worker has been added.   Thus, something in start up could read a 0 value inconsistently instead of the actual io worker max threads (or possibly some value in between if there are multiple io workers to contribute to that total value).

       

      An impacted example would be the default EJB slsb that is deriving size from-worker-pools:

                          <strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/> 

      So sometimes it may see the actual io worker thread max and size to that value or it may see 0 and fall through to derive size from cpu counts instead here.  This particular bad result can occur if org.wildfly.extension.io.WorkerAdd.performRuntime happens to occur after org.jboss.as.ejb3.component.pool.StrictMaxPoolConfigService.<init> for any reason.

      It looks like this has been slightly race susceptible for a while but it became notably worse after https://github.com/wildfly/wildfly-core/commit/c16f103cb2057fb337bfac0b9be2debca8ec8802 and https://github.com/wildfly/wildfly-core/commit/083c7848daecdac98e0c4394213053cd99e0a242.  Prior to this, it looks like IOSubsystemAdd helped this race condition by starting the capability service in mode NEVER.

        1. WFCORE-7388.btm
          2 kB
          Aaron Ogburn
        2. WFCORE-7388.xml
          34 kB
          Aaron Ogburn

              flaviarnn Flavia Rainone
              rhn-support-aogburn Aaron Ogburn
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: