Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-3565

RequestController ignores queued tasks if max requests is configured

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 4.0.0.Alpha10
    • 3.0.8.Final, 4.0.0.Alpha7
    • None
    • None
    • Hide

      I created a minimal example project which demonstrates the EJB timer issue. The project can found on https://github.com/cboehme/wildfly-timer-issue/.

      Steps to reproduce

      1. Clone and build the wildfly-timer-issue project: 
            git clone https://github.com/cboehme/wildfly-timer-issue.git
    cd wildfly-timer-issue
    mvn clean package
      2. Start a clean Wildfly and configure the request controller to allow at max two requests. Additionally, debug logging for org.jboss.as.ejb3.timer should be enabled. The configure wildfly.cli Wildfly CLI script can be used to make this configuration automatically on a locally installed Wildfly.
      3. Deploy the request-maker application on the new wildfly. It is a simple web application with single endpoint /make-request that takes thirty seconds to complete a HTTP GET request. It enables us to clog up the request controller.
      4. Make two requests to http://localhost:8080/make-request in parallel. These requests will run for 30 seconds and take up all available requests on the Wildfly instance.
      5. While the two requests are running deploy the timer application on the same wildfly. This application defines an EJB timer which fires every five seconds. Its callback take one second to complete.
      6. The Wildfly console output shows that the timer application is deployed and that it registered an ejb timer. However, the callback method of the timer is never called. Not even after the two web requests have finished.
      7. Once the two web requests finished, log into the Wildfly admin interface and suspend and resume the server. Now the timer invocation will be processed and the timer starts working as expected.
      8. Now, try to make two more parallel requests to http://localhost:8080/make-request while the timer is running. If the timer is being activated just when the second request arrives, this request will be rejected (as expected) with a "503 - Service Unavailable". If the second request goes through, though, the timer stalls again until the requests finished and the server is suspended and resumed.
      Show
      I created a minimal example project which demonstrates the EJB timer issue. The project can found on https://github.com/cboehme/wildfly-timer-issue/ . Steps to reproduce Clone and build the wildfly-timer-issue project: git clone https: //github.com/cboehme/wildfly-timer-issue.git cd wildfly-timer-issue mvn clean package Start a clean Wildfly and configure the request controller to allow at max two requests. Additionally, debug logging for org.jboss.as.ejb3.timer should be enabled. The configure wildfly.cli Wildfly CLI script can be used to make this configuration automatically on a locally installed Wildfly. Deploy the request-maker application on the new wildfly. It is a simple web application with single endpoint /make-request that takes thirty seconds to complete a HTTP GET request. It enables us to clog up the request controller. Make two requests to http://localhost:8080/make-request in parallel. These requests will run for 30 seconds and take up all available requests on the Wildfly instance. While the two requests are running deploy the timer application on the same wildfly. This application defines an EJB timer which fires every five seconds. Its callback take one second to complete. The Wildfly console output shows that the timer application is deployed and that it registered an ejb timer. However, the callback method of the timer is never called. Not even after the two web requests have finished. Once the two web requests finished, log into the Wildfly admin interface and suspend and resume the server. Now the timer invocation will be processed and the timer starts working as expected. Now, try to make two more parallel requests to http://localhost:8080/make-request while the timer is running. If the timer is being activated just when the second request arrives, this request will be rejected (as expected) with a "503 - Service Unavailable". If the second request goes through, though, the timer stalls again until the requests finished and the server is suspended and resumed.

    Description

      We encountered an issue with EJB timers stopping randomly if the container is under load. Sometimes the timers get stalled right after deployment without firing at all. Sometimes the timers work fine for a couple of days before they suddenly stop working. The timers are defined using the @Scheduled annotation.

      We found out that the reason for the randomly stopping timers is a bug in the RequestController class which only appears if a maximum number of requests is configured in the request controller subsystem. If a timer is being triggered while the container has reached the request limit then the timer is put into a task queue to be processed later. However, the task queue is never checked for queued tasks if the request number goes down again. This leaves the timers stuck in the queue. Since every timer needs to reregister after it has been triggered, the timers appear to have stopped.

      In old versions of the RequestController class from before release wildfly-core-1.0.0.Alpha14 it looks like the idea was to check for queued requests whenever a request finished (see here).

      Attachments

        Activity

          People

            cboehme-1 Christoph Böhme (Inactive)
            cboehme-1 Christoph Böhme (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: