Provide runtime manager strategies and lock diagnose upon system lock broken mechanism

Platform issue for JBPM-9521 to provide different implementations and config options for runtime manager locking strategies.

solution

When systems go production sometimes the lock mechanism in the RuntimeManager can be difficult to troubleshoot.

Three different flags have been created:

• org.kie.jbpm.runtime.manager.lock.strategy: this allows to select which strategy should be use for all runtime managers created by any runtime manager
• org.kie.jbpm.runtime.manager.lock.factory: this allows to select witch type of lock to be used (normal, debug -diagnostics or self release)
• org.kie.jbpm.runtime.manager.lock.timeout: this allow to specify how long before timeout a timeout lock (for TimeoutRuntimeManagerLockStrategy)

This jira is to provide different lock strategies using the org.kie.jbpm.runtime.manager.lock.strategy system property:

• org.jbpm.runtime.manager.impl.lock.FreeRuntimeManagerLockStrategy
  no locks at all. This will lead to optimistic lock failures in db and CPU.
• org.jbpm.runtime.manager.impl.lock.SerializableRuntimeManagerLockStrategy
  perfect serializable locking strategy. It disallows for a runtime manager to access the same process instance id at the same time. lock waits forever till is unlocked. cannot be interrupted or is not affect by Thread.interrupted i.g. tx ThreadReaper
• org.jbpm.runtime.manager.impl.lock.TimeoutRuntimeManagerLockStrategy
  it allows to specify how long a lock should wait before failing using system property org.kie.jbpm.runtime.manager.lock.timeout (default: 5000ms). It is recomended: lock timeout < tx timeout.
  It can be affected by Thread.interrupt as well as tx ThreadReaper
• org.jbpm.runtime.manager.impl.lock.InterruptibleRuntimeManagerLockStrategy
  It locks forevever either because it gets the lock or recieves an interrupt signal. It is sensitive to Thread.interrupt and tx ThreadReaper
• org.jbpm.runtime.manager.impl.lock.LegacyRuntimeManagerLockStrategy
  It works with the exact same code as usual

This jira provides also different lock factories using the org.kie.jbpm.runtime.manager.lock.factory system property:

• org.jbpm.runtime.manager.impl.lock.DefaultRuntimeManagerLockFactory
  this provides a normal lock
• org.jbpm.runtime.manager.impl.lock.DebugRuntimeManagerLockFactory
  this locks retains information of the current lock and stack trace of all ways the lock reentered keeping such information until is unlocked. This is useful to retain the stack trace and program flow how this lock was acquired by the current owner.
  This is only useful with those locks that are sensitive with interrupt signal or timeout. This information is printed if the current thread is not capable of getting the lock or is being interrupted.
• org.jbpm.runtime.manager.impl.lock.SelfReleaseRuntimeManagerLockFactory Added new lock implementation that allows to unlock from another thread. Required for watcher reaper thread to force unlocked threads.

Also I created a thread watcher attached to every runtime Manager that spawns a thread to watch every lock for a runtime manager.
Two environment variables has been created

org.kie.jbpm.lock.polling

if equals to zero then it does not activate polling. greater to zero will check every interval if those locks have exceeded the maximum processing time.

The maximum processing time is the max time a lock is considered that is not unlocked.

org.kie.jbpm.lock.maxProcessingTime

IMPORTANT NOTE: Only the lock strategy and factory options above marked in GREEN are recommended configuration options. The other implementations are provided mainly for debugging and troubleshooting purposes and should only be used under supervision in the context of a support ticket!