Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-25790

(8.0.z) HotRod calls to remote caches use outdated topology information

XMLWordPrintable

      One interoperability test between EAP 8 and RHDG on OpenShift is failing.

      EAP 8 cluster instances are configured to use the remote RHDG cluster for a distributed cache. Specifically, the test persists ditributed timers status on the cache, as per EAP7-1417, but using a remote Infinispan.

      During the test one RHSG pod is stopped, in order to verify the timer "persistence" service fail-over mechanism, which actually takes place.

      Despite this, we noticed an issue: the logs show that one - or more - EAP instances perform a call to Infinispan via HotRod by using an IP which is no longer listed as part of the cluster topology, and after having received the up to date topology.

      • Working EAP 8 pod, using up to date information and showing no issues in Infinispan calls:
      ...
[0m[0m14:45:12,146 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004006: Server sent new topology view (id=-1899142735, age=0) containing 2 addresses: [10.130.2.116:11222, 10.129.2.215:11222]
[0m[0m14:45:12,148 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004014: New server added(10.129.2.215:11222), adding to the pool.
[0m[0m14:45:12,149 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004014: New server added(10.130.2.116:11222), adding to the pool.
...
[0m[0m14:45:35,466 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004006: Server sent new topology view (id=-198812969, age=0) containing 1 addresses: [10.130.2.116:11222]
[0m[0m14:45:35,467 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004016: Server not in cluster anymore(10.129.2.215:11222), removing from the pool.
...
[0m[0m14:45:45,496 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004006: Server sent new topology view (id=-251298871, age=0) containing 2 addresses: [10.130.2.116:11222, 10.129.2.216:11222]
[0m[0m14:45:45,496 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004014: New server added(10.129.2.216:11222), adding to the pool.
...
      • EAP pod that is using a removed Infinispan pod address, and failing the call:
      ...
[0m[0m14:44:07,452 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004006: Server sent new topology view (id=-1899142735, age=0) containing 2 addresses: [10.130.2.116:11222, 10.129.2.215:11222]
[0m[0m14:44:07,453 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004014: New server added(10.129.2.215:11222), adding to the pool.
[0m[0m14:44:07,454 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004014: New server added(10.130.2.116:11222), adding to the pool.
...
[0m[0m14:46:31,449 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004006: Server sent new topology view (id=-251298871, age=0) containing 2 addresses: [10.130.2.116:11222, 10.129.2.216:11222]
[0m[0m14:46:31,449 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004014: New server added(10.129.2.216:11222), adding to the pool.
[0m[0m14:46:31,450 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004016: Server not in cluster anymore(10.129.2.215:11222), removing from the pool.
[0m[31m14:47:05,274 ERROR [org.infinispan.HOTROD] (HotRod-client-async-pool-16) ISPN004007: Exception encountered. Retry 10 out of 10: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: /10.129.2.215:11222
...
    at io.netty.netty-common@4.1.94.Final-redhat-00001//io.netty.util.internal.ThreadExecutorMap.run(ThreadExecutorMap.java:74)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.wildfly.clustering.context@8.0.0.GA-redhat-00009//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
...

      This affects EAP 8 CR 1, and is filed as a blocker targeting EAP 8 GA for two reasons:
      1. it could affect the Infinispan subsystem outside of the scope of the RFE which has been tested
      2. to be considered for fix before EAP 8 GA.

        1. hotrod-504-topology-full.log
          657 kB
        2. hotrod-504-topology-74.log
          823 kB
        3. hotrod-504-topology.log
          189 kB

              pferraro@redhat.com Paul Ferraro
              fburzigo Fabio Burzigotti
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: