Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-27774

[GSS](7.4.z) WFLY-19681 - DatabaseTimerPersistence$RefreshTask can delay other threads' timer additions or removals when detecting many Timer removals from the database

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 7.4.20.GA, 7.4.20.CR1
    • None
    • EJB
    • None
    • False
    • None
    • False
    • ?
    • Hide

      1. Set up a jboss config like standalone-ds.xml with a datasource used for the EJB timer service default-data-store
      2. Increase the configured heap in standalone.conf to allow for millions of timers (i used -Xmx10g)
      3. set up a local database to use for your standalone configuration (I used a local postgres)
      4. Launch jboss with the below properties to launch the test app with persistent timers and a high amount of timers created in start up:

      ./standalone.sh -c standalone-ds.xml -Dexample.seedCount=1000000 -Dexample.persistent=true

      5. Run light load with a single recurring client request:

      ab -c 1 -n 99999 localhost:8080/ejb-timer/HelloWorld

      6. Clear the jboss_ejb_timer table from another external source. An easy way if using postgres:

      psql -U $username timers
truncate "jboss_ejb_timer";
SELECT * FROM "jboss_ejb_timer";

      7. Wait for the next refresh task to start in the next 30 seconds and note that request processing halts.

      Show
      1. Set up a jboss config like standalone-ds.xml with a datasource used for the EJB timer service default-data-store 2. Increase the configured heap in standalone.conf to allow for millions of timers (i used -Xmx10g) 3. set up a local database to use for your standalone configuration (I used a local postgres) 4. Launch jboss with the below properties to launch the test app with persistent timers and a high amount of timers created in start up: ./standalone.sh -c standalone-ds.xml -Dexample.seedCount=1000000 -Dexample.persistent= true 5. Run light load with a single recurring client request: ab -c 1 -n 99999 localhost:8080/ejb-timer/HelloWorld 6. Clear the jboss_ejb_timer table from another external source. An easy way if using postgres: psql -U $username timers truncate "jboss_ejb_timer" ; SELECT * FROM "jboss_ejb_timer" ; 7. Wait for the next refresh task to start in the next 30 seconds and note that request processing halts.

      If many EJB timers were recently removed from the database, then the DatabaseTimerPersistence$RefreshTask can have many existing timers to still process sequentially through this loop here while persistently holding the DatabaseTimerPersistence lock. If there are hundreds of thousands or even millions of existing entries remaining through this synchronized loop, then the lock can be held for 10-20 seconds consecutively or perhaps longer. Any other threads that attempt an EJB timer addition or cancel/removal during that refresh task will be notably delayed till the lock is released after the synchronized loop finishes combing through the existing entries.

        1. standalone-ds.xml
          31 kB
        2. ejb-timer.war
          9 kB

            rhn-support-aogburn Aaron Ogburn
            rhn-support-aogburn Aaron Ogburn
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: