verifyError can arise because CFG propagates wrong monitoropen count

try-catch blocks identify potential exception control flow from the try catch region to a handler block. if the try catch region overlaps one or more regions identified by a monitorenter and accompanying (non-exceptional control flow) monitorexit then the handler block will contain one or more monitorexits to ensure locks are correctly released.

Byteman's CFG analysis propagates details of open monitorenters along exception flow paths to their handler block, recording the open locations in a list attached to the handler. The list is ordered by code position of the enter. When analysing control flow for the handler block Byteman pairs off enters with matching exit locations in the block, on the assumption that any residual enters at any given code location in the block are still open. This allows it to propagate those residual open enters along direct and exceptional control flows out of the handler block.

The sort algorithm is flaky and this manifests when two distinct try catch regions for the same exception type employ the same handler block and are both within the scope of the same unclosed monitorenter (in the forum thread example this was two catch-all handlers associated with a finally block). The sorting error led to the enter being added to the list twice and, hence, only removed once. This caused an invalid enter to be propagated to an injected Byteman trigger block which Byteman protected with a wrapper handler that invoked monitorexit. The verifier rightly detected that the Byteman generated monitorexit was trying ot unlock an object that had already been unlocked.

The solution is to change the sort code so that it correctly detects a duplicated entry.