Task deadline timers are not removed in multi-node setup and fire repeatedly

In a multi-node setup, the following sequence of events can lead to stale timer entries in the JBOSS_EJB_TIMER table:

1. Start a process on node1
2. The process reaches a human task node with notifications/reassignments and creates timers
3. Complete the task on node2 before the timers are refreshed

Result:

• Timer entries fail to get removed:

  2021-04-01 08:27:44,849 DEBUG [org.jbpm.services.ejb.timer.EJBTimerScheduler] (default task-2) Job handle EjbGlobalJobHandle [uuid=20857_23_START] does not match any timer on org.jbpm.services.ejb.timer.EJBTimerScheduler@5eb65329 scheduler service
  

• Once the notification/reassignment timer expires, it will be called repeatedly:

  2021-04-01 08:29:13,854 WARN  [org.jbpm.services.ejb.timer.EJBTimerScheduler] (EJB default - 1) Execution of time failed Interval Trigger failed. Skipping GlobalJpaTimerJobInstance [timerServiceId=project_reproducer_1.0.0-SNAPSHOT-timerServiceId, getJobHandle()=EjbGlobalJobHandle [uuid=20857_23_START]]
  2021-04-01 08:29:13,865 WARN  [org.jbpm.services.ejb.timer.EJBTimerScheduler] (EJB default - 4) Execution of time failed Interval Trigger failed. Skipping GlobalJpaTimerJobInstance [timerServiceId=project_reproducer_1.0.0-SNAPSHOT-timerServiceId, getJobHandle()=EjbGlobalJobHandle [uuid=20857_23_START]]
  2021-04-01 08:29:13,882 WARN  [org.jbpm.services.ejb.timer.EJBTimerScheduler] (EJB default - 3) Execution of time failed Interval Trigger failed. Skipping GlobalJpaTimerJobInstance [timerServiceId=project_reproducer_1.0.0-SNAPSHOT-timerServiceId, getJobHandle()=EjbGlobalJobHandle [uuid=20857_23_START]]
  2021-04-01 08:29:13,893 WARN  [org.jbpm.services.ejb.timer.EJBTimerScheduler] (EJB default - 2) Execution of time failed Interval Trigger failed. Skipping GlobalJpaTimerJobInstance [timerServiceId=project_reproducer_1.0.0-SNAPSHOT-timerServiceId, getJobHandle()=EjbGlobalJobHandle [uuid=20857_23_START]]
  

This can only be ended by stopping the node and manually deleting the timer from the timer table.

Notes

When a timer is triggered because it is still running and other node cancels those timers (Complete task) before the other node the timer is triggered can cause a race condition in the reschedule. verify that the timer does not exist anymore should be considererd that there is no need for rescheduling.