Current locking algorithm in non-transactional caches needs a remote thread on the primary owner to block while replicating the update to the backup owner. The thread is also holding the lock for the key, so it's blocking other threads that want to write to the same key. When there is a lot of contention, this can exhaust the remote executor thread pool and cause lock timeouts.
TO was designed with high contention in mind, and it doesn't block threads to acquire locks. So it should handle this much better.
An alternative solution would be the locking rework in