Uploaded image for project: 'Red Hat Process Automation Manager'
  1. Red Hat Process Automation Manager
  2. RHPAM-4426

Performance downgrade possibly related to RHPAM-4069

    XMLWordPrintable

Details

    • 2022 Week 41-43 (from Oct 10), 2022 Week 44-46 (from Oct 31)

    Description

      rhpam performance downgrades significantly after migrating to 7.12.1.

      The following is the thread dump analysis:
      ----------------------
      "default task-817" #295624 prio=5 os_prio=0 cpu=5281.12ms elapsed=587.71s tid=0x0000564f06af8000 nid=0x4b66e waiting for monitor entry [0x00007fa4818c9000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.kie.server.services.jbpm.cluster.InfinispanClusterAwareService.addData(InfinispanClusterAwareService.java:179)

      • waiting to lock <0x000000040cfab2e8> (a org.kie.server.services.jbpm.cluster.InfinispanClusterAwareService)
        at org.kie.server.services.jbpm.cluster.ClusteredJobFailOverListener.afterJobScheduled(ClusteredJobFailOverListener.java:71)
        at org.jbpm.executor.impl.event.ExecutorEventSupportImpl.lambda$fireAfterJobScheduled$3(ExecutorEventSupportImpl.java:50)
        at org.jbpm.executor.impl.event.ExecutorEventSupportImpl$$Lambda$3333/0x00000008026fc440.accept(Unknown Source)
        at org.drools.core.event.AbstractEventSupport.notifyAllListeners(AbstractEventSupport.java:60)
        at org.jbpm.executor.impl.event.ExecutorEventSupportImpl.fireAfterJobScheduled(ExecutorEventSupportImpl.java:50)
        at org.jbpm.executor.impl.ExecutorImpl.scheduleRequest(ExecutorImpl.java:419)
        at org.jbpm.executor.impl.ExecutorImpl.scheduleRequest(ExecutorImpl.java:319)
        at org.jbpm.executor.impl.ExecutorServiceImpl.scheduleRequest(ExecutorServiceImpl.java:139)
        at org.jbpm.workflow.instance.node.AsyncEventNodeInstance.internalTrigger(AsyncEventNodeInstance.java:63)
        ....
        ----------------------

      The thread owner of the lock:

      ----------------------
      "Thread-81 (ActiveMQ-client-global-threads)" #244950 daemon prio=5 os_prio=0 cpu=81160.20ms elapsed=7642.90s tid=0x0000564f0a484000 nid=0x3ece5 waiting on condition [0x00007fa47aa7d000]
      java.lang.Thread.State: TIMED_WAITING (parking)
      at jdk.internal.misc.Unsafe.park(java.base@11.0.14.1/Native Method)

      • parking to wait for <0x00000005e58606e8> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.14.1/LockSupport.java:234)
        at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.14.1/CompletableFuture.java:1798)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.14.1/ForkJoinPool.java:3128)
        at java.util.concurrent.CompletableFuture.timedGet(java.base@11.0.14.1/CompletableFuture.java:1868)
        at java.util.concurrent.CompletableFuture.get(java.base@11.0.14.1/CompletableFuture.java:2021)
        at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:125)
        at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:36)
        at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246)
        at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:298)
        at org.infinispan.cache.impl.InvocationHelper.executeCommandWithInjectedTx(InvocationHelper.java:203)
        at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:101)
        at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:84)
        at org.infinispan.cache.impl.CacheImpl.computeIfAbsentInternal(CacheImpl.java:367)
        at org.infinispan.cache.impl.CacheImpl.computeIfAbsent(CacheImpl.java:357)
        at org.infinispan.cache.impl.CacheImpl.computeIfAbsent(CacheImpl.java:337)
        at org.infinispan.cache.impl.AbstractDelegatingCache.computeIfAbsent(AbstractDelegatingCache.java:424)
        at org.infinispan.cache.impl.EncoderCache.computeIfAbsent(EncoderCache.java:672)
        at org.infinispan.cache.impl.AbstractDelegatingCache.computeIfAbsent(AbstractDelegatingCache.java:424)
        at org.infinispan.Cache.computeIfAbsent(Cache.java:388)
        at org.kie.server.services.jbpm.cluster.InfinispanClusterAwareService.addData(InfinispanClusterAwareService.java:180)
      • locked <0x000000040cfab2e8> (a org.kie.server.services.jbpm.cluster.InfinispanClusterAwareService)
        at org.kie.server.services.jbpm.cluster.ClusteredJobFailOverListener.afterJobScheduled(ClusteredJobFailOverListener.java:71)
        at org.jbpm.executor.impl.event.ExecutorEventSupportImpl.lambda$fireAfterJobScheduled$3(ExecutorEventSupportImpl.java:50)
        at org.jbpm.executor.impl.event.ExecutorEventSupportImpl$$Lambda$3333/0x00000008026fc440.accept(Unknown Source)
        at org.drools.core.event.AbstractEventSupport.notifyAllListeners(AbstractEventSupport.java:60)
        at org.jbpm.executor.impl.event.ExecutorEventSupportImpl.fireAfterJobScheduled(ExecutorEventSupportImpl.java:50)
        ....
        ----------------------

      We suspect that this is due to the changes made in 7.12.1 [1]

      We notice that a couple of synchronized blocks are added in these changes.

      [1] https://issues.redhat.com/browse/RHPAM-4069

      Attachments

        Issue Links

          Activity

            People

              rhn-support-egonzale Enrique Gonzalez Martinez (Inactive)
              rhn-support-ghu Gary Hu
              Gonzalo Muñoz Fernández Gonzalo Muñoz Fernández
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: