The RESTful search "org.kie.server.remote.rest.jbpm.RuntimeDataResource.getTasksAssignedAsPotentialOwner" effectively runs single threaded due to a synchronized "execute" method in the TaskTransactionInterceptor.
When running a performance test on that RESTful resource (100 threads, looping), we can see in the database stats (PostgreSQL on EAP 7) that only a single database connection is used. At the same time, a single PostgreSQL process is maxing out on one core. As PostgreSQL uses a OS process per connection, but the system only uses one connection, the database starts struggling. This dramatically decrease the throughput and performance of the system.
We've created some thread dumps, and we can clearly see that all threads are blocking behind this one:
"default task-62" #310 prio=5 os_prio=0 tid=0x00007feeb8010800 nid=0x3fd4 runnable [0x00007fee36ebe000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71) at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1700) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) - locked <0x0000000643860da8> (a org.postgresql.core.v3.QueryExecutorImpl) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302) at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:504) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) at org.hibernate.loader.Loader.getResultSet(Loader.java:2090) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1866) at org.hibernate.loader.Loader.doQuery(Loader.java:905) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:347) at org.hibernate.loader.Loader.doList(Loader.java:2578) at org.hibernate.loader.Loader.doList(Loader.java:2564) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2394) at org.hibernate.loader.Loader.list(Loader.java:2389) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:198) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1230) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268) at org.jbpm.services.task.persistence.JPATaskPersistenceContext.queryStringWithParameters(JPATaskPersistenceContext.java:575) at org.jbpm.services.task.persistence.JPATaskPersistenceContext.queryWithParametersInTransaction(JPATaskPersistenceContext.java:398) at org.jbpm.services.task.impl.TaskQueryServiceImpl.getTasksAssignedAsPotentialOwner(TaskQueryServiceImpl.java:368) at org.jbpm.services.task.commands.GetTaskAssignedAsPotentialOwnerCommand.execute(GetTaskAssignedAsPotentialOwnerCommand.java:85) at org.jbpm.services.task.commands.GetTaskAssignedAsPotentialOwnerCommand.execute(GetTaskAssignedAsPotentialOwnerCommand.java:31) at org.jbpm.services.task.commands.TaskCommandExecutorImpl$SelfExecutionCommandService.execute(TaskCommandExecutorImpl.java:65) at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41) at org.jbpm.services.task.persistence.TaskTransactionInterceptor.execute(TaskTransactionInterceptor.java:69) - locked <0x000000065b1cbc90> (a org.jbpm.services.task.persistence.TaskTransactionInterceptor) at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41) at org.drools.persistence.jta.TransactionLockInterceptor.execute(TransactionLockInterceptor.java:73) at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41) at org.drools.persistence.jpa.OptimisticLockRetryInterceptor.execute(OptimisticLockRetryInterceptor.java:82) at org.jbpm.services.task.commands.TaskCommandExecutorImpl.execute(TaskCommandExecutorImpl.java:40) at org.jbpm.services.task.impl.command.CommandBasedTaskService.getTasksAssignedAsPotentialOwner(CommandBasedTaskService.java:229) at org.jbpm.kie.services.impl.RuntimeDataServiceImpl.getTasksAssignedAsPotentialOwner(RuntimeDataServiceImpl.java:752) at org.kie.server.services.jbpm.RuntimeDataServiceBase.getTasksAssignedAsPotentialOwner(RuntimeDataServiceBase.java:392) at org.kie.server.remote.rest.jbpm.RuntimeDataResource.getTasksAssignedAsPotentialOwner(RuntimeDataResource.java:381) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:175) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:207) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:802) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000007716d8b88> (a java.util.concurrent.ThreadPoolExecutor$Worker) - <0x00000006438cfec0> (a java.util.concurrent.locks.ReentrantLock$FairSync)
at the "TaskTransactionInterceptor.execute" method which is synchronized: https://github.com/droolsjbpm/jbpm/blob/master/jbpm-human-task/jbpm-human-task-jpa/src/main/java/org/jbpm/services/task/persistence/TaskTransactionInterceptor.java#L61
So effectively, this query runs single threaded and does not scale.
- causes
-
RHBPMS-4655 Cannot get tasks in the same transaction on WAS and WLS
-
- Closed
-
- relates to
-
-
- New
-
