Motivation

In the context of asynchronous cross-site replication, applications may be unable to stick the requests to the same site. Using session externalization as an example, it is possible a session to be updated in site "A" but the next request is sent to site B.

In this scenario, the application may want to wait for the replication from site A to reach site B.

Idea -> StampedCache

The idea is on a write operation to return a "stamp" to the application. A write on site A, can return stamp "S", and the application can use this stamp "S" in site B to read to make sure it reads the previous write or something more recent. There will be a timeout where Infinispan gives up and returns the most up to date value locally.

The API could be this simple:

interface StampedCache<K, V> {
   CompletionStage<StampedValue<V>> put(K key, V value, Metadata metadata);

   CompletionStage<StampedValue<V>> remove(K key);

   CompletionStage<StampedValue<V>> get(K key, Stamp s, long timeout, TimeUnit unit)
}

The StampedValue contains the value, the stamp, and a method to check if the stamp is valid or not.

The same interface will be used for embedded and remote caches.

Implementation details

Stamp: use the vector clock created by the asynchronous cross-site. Investigate if the full vector clock is required or just the origin site position. Investigate if it is better to use an byte[] internally or if it should be a String. I'm guessing a String would be better, specially to be used as a cookie in the REST endpoint.

Writes: no change is expected to write operations. Just need to encode the vector clock in a Stamp.

Reads: reads will fetch the value from the local memory/disk. If it does not find any key/version or the version stored locally is older, wait until the timeout is reached.

Problematic patterns

Put and Remove

client-1 in site A does put(k,v) -> S
client-2 in site B does remove(k)
client-1 in site B does get(k, S)

In the read, the client-1 may wait forever (until timeout) if it happens after the tombstone is no longer available.

Others?