If using Infinispan with local caches as second level cache provider with read-write concurrency strategy the cache becomes stale if a writing transaction fails.
If a transaction changes an entity and that transaction is rolled back, the changes are nonetheless put into the second level cache. This incorrect value is then read and used by consecutive transactions.
As soon as the concurrency strategy is set to transactional or the cache configuration uses a replicated configuration the issue seems to disappear. Although we still observed odd behavior with transactional local caches which I was not able to reliably reproduce (entries remaining inexplicably locked).
We wanted to switch to the read-write strategy as we encountered the performance issues described in https://hibernate.atlassian.net/browse/HHH-12036 and https://issues.jboss.org/browse/ISPN-8411