OpenShift - Distributed Timers, timer not found by TimerService::getTimers sometimes

We have an OpenShift test that fails in about 50% cases when executed on EAP QE Jenkins pipelines.

The mentioned test deploys:

• a PostgreSql service to store timer expirations metadata
• an EAP application service that exposes endpoints to handle such persistence operations
• an EAP application service that exposes two EJB timer beans, one which is transactionally calling the persistence APIs to record its expirations and another one that just logs messages, without storing any expiration metadata. This application service also exposes endpoints for the test class to be able to create and delete timers, and to retrieve information about them. The EJB timer beans are called remotely by such endpoints.

The timer persistence is delegated to the Infinispan subsystem, as per EAP7-1417.

After deploying the scenario, several tests are run, e.g. to verify a timer can be created/deleted successfully or fail-over cases.

Most of the tests eventually try to cancel the timers they create as the last thing by calling a REST API endpoint which remotely calls the EJB bean method that finally gets a list of all the timers to cancel the requested one. Sometimes such search ends up with a null value being returned, i.e. the timer, which is supposed to exist, is not found when calling TimerService::getTimers.

Links to internal resources documenting the test behavior are reported as part of the first comments.