Uploaded image for project: 'Teiid'
  1. Teiid
  2. TEIID-3099

RejectedExecutionException for Teiid during high query load

    XMLWordPrintable

Details

    • Hide

      Have a standalone Teiid server be bombarded with concurrent queries numbering larger than the size of the server's threadpool. The error is not consistent, and I believe it to be more common with more threads allocated on the server due to the core issue being a race condition.

      Show
      Have a standalone Teiid server be bombarded with concurrent queries numbering larger than the size of the server's threadpool. The error is not consistent, and I believe it to be more common with more threads allocated on the server due to the core issue being a race condition.

    Description

      Occasionally, when a standalone Teiid server is under high load from concurrent queries, a RejectedExecutionException is thrown. The relevant part of the stacktrace is below. I investigated the ThreadReuseExecutor source, and I believe the issue's cause is based around the sycnhronized(poolLock) code - see below the stack trace for info, and psuedo-code snippet with my analysis below the stacktrace.

      Caused by: java.util.concurrent.RejectedExecutionException: Task org.teiid.dqp.internal.process.ThreadReuseExecutor$3@c75a8114 rejected from org.teiid.dqp.internal.process.ThreadReuseExecutor$2@8b1166ec[Running, pool size = 128, active threads = 128, queued tasks = 0, completed tasks = 692637]
      at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) [rt.jar:1.7.0]
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) [rt.jar:1.7.0]
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) [rt.jar:1.7.0]
      at org.teiid.dqp.internal.process.ThreadReuseExecutor.executeDirect(ThreadReuseExecutor.java:200) [teiid-engine-8.7.0.FinalCAFix-SNAPSHOT.jar:8.7.0.FinalCAFix-SNAPSHOT]
      at org.teiid.dqp.internal.process.ThreadReuseExecutor.execute(ThreadReuseExecutor.java:177) [teiid-engine-8.7.0.FinalCAFix-SNAPSHOT.jar:8.7.0.FinalCAFix-SNAPSHOT]

      In the below executeDirect, the race condition that appears plausible to me is: given "activeCount" is at the thread pool size limit and a single thread is finishing execution...Let activeCount be reduced by one in the tpe.execute runnable's synchronized block. The synchronized block exits, and goes to the warnWait/logging code - at this point a thread context switch occurs, and a new PrioritizedRunnable (one which isn't waiting on the poolLock at the beginning of the executeDirect method) comes through executeDirect and proceeds forward because "poolLock" has been released, "activeCount" is now sizeLimit-1. The tpe tries to execute a new Runnable wrapping this PrioritizedRunnable, but the previous thread which hasn't completed it's logging code yet is still present in the the threadpool, and as such we get a RejectedExecutionException due to too many threads trying to be executed in a fixed size thread pool.

      private void executeDirect(final PrioritizedRunnable command) {
      boolean atMaxThreads = false;
      synchronized (poolLock)

      { .... if activeCount!=max_limit; activeCount++ }

      .......
      tpe.execute(.....
      finally {
      synchronized (poolLock)

      { ....... activeCont--; }

      if (success)

      { ......some log code }

      }

      Attachments

        Activity

          People

            rhn-engineering-shawkins Steven Hawkins
            markackert Mark Ackert (Inactive)
            Mark Ackert (Inactive), Mark Addleman (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: