I am trying to setup remote cache by externalising the Infinispan server from Keycloak using the Cross-Datacenter Replication Mode. I am following the Keycloak official documentation here
https://www.keycloak.org/docs/latest/server_installation/#crossdc-mode

As per the documentation, the Keycloak and Infinispan version being used as as follows (I am hoping these versions are compatible to each other since this information is coming from Keycloak’s official. site) -

Keycloak Version - 11.0.2
Infinispan Version - 9.4.18

To my surprise, I have found a bug into Infinispan V9.4.18. Below exception is being constantly reported from infinispan server.

I downloaded Infinispan server (version infinispan-server-9.4.18) from their official site https://downloads.jboss.org/infinispan/9.4.18.Final/infinispan-server-9.4.18.Final.zip
Without making any changes into any of the infinispan config file, when I boot up the Infinispan server, it starts without any issue but when I try to bring up the Hotrod console at http://127.0.0.1:11222
(11222 is the port where hotrod listens on my localhost), it does not load. I get the issue as shown below :

¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 71¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 69¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 84¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 32¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 47¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 32¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 72¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 84¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 84¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 80¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 47¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 49¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 46¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 49¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 13¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 10¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 72¡�P�aorg.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 111¡�P�aorg.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 115¡�P�aorg.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 116¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 58¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 32¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 49¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 50¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 55¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 46¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 48¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 46¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 48¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 46¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 49¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 58¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 49¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 49¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 50¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id:

While on the server’s console, the exception is as follows :
18:41:28,551 ERROR [org.infinispan.server.hotrod.BaseRequestProcessor] (HotRod-ServerIO-4-1) ISPN005003: Exception reported: org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 71
at org.infinispan.server.hotrod.HotRodDecoder.switch0(HotRodDecoder.java:208)
at org.infinispan.server.hotrod.HotRodDecoder.switch1_0(HotRodDecoder.java:153)
at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.java:143)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

18:41:28,593 ERROR [org.infinispan.server.hotrod.BaseRequestProcessor] (HotRod-ServerIO-4-1) ISPN005003: Exception reported: org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 69
at org.infinispan.server.hotrod.HotRodDecoder.switch0(HotRodDecoder.java:208)
at org.infinispan.server.hotrod.HotRodDecoder.switch1_0(HotRodDecoder.java:153)

I tried with multiple versions of Infinispan servers (e.g. 9.4.18, 9.4.20, 10.0.1, 11.0.3) but the Infinispan versions 9.4.x gives me the issue while 10.x.x and 11.x.x) does not and I can load the Hotrod console . without any problem (with versions 10.x.x and 11.x.x). 
 
Keycloak documentation I am following ( https://www.keycloak.org/docs/latest/server_installation/#crossdc-mode ) for integrating Keycloak with external remote Infinispan ,  is based on Infinispan version v9.4.18 which differs quite a bit on architectural front from versions v10.0.1  or v11.0.3

I see one solution suggested by Redhat here https://access.redhat.com/solutions/3920921  around this issue which sounds like a hotrod client's protocol version issue (probably not being compatible with hotrod server etc) but in my case I am trying to load the hotrod server console thru browser. I don't have any hotrod client application per se), so the solution isn't of much use. 

Has any one ever come across or have any idea/pointers around the below infinispan issue i am facing and how to overcome this??

Appreciate pointers/suggestions here.
  
Thanks much
Jameel