Currently, when using compatibility mode with distributed caches, it is possible for users to store keys using embedded API which hash differently to when the same key is access via remote API. This is due to the differences in source data. For embedded, the source object is the POJO, whereas for remote, the source object is the byte[] format of the pojo.

When using such keys, remote APIs might go to a node which is not owner, leading to a remote get. When two gets are executed, the first of which results in L1 entry being added to the container, the second get result in exceptions such as this on the client:

java.lang.ClassCastException: [B cannot be cast to xxx....

This is due to over eager unboxing by the BaseTypeConverterInterceptor. I discovered this while investigating ISPN-5449.

It would be ideal to not have scenarios where there are diverging keys but for that, clients would need to know when compatibility is enabled, which currently is not known by the clients. Maybe this could be negotiated with the ping command?