-
Bug
-
Resolution: Done
-
Critical
-
8.0.0.GA-CR1, 8.0.0.GA-CR2
One interoperability test between EAP 8 and RHDG on OpenShift is failing.
EAP 8 cluster instances are configured to use the remote RHDG cluster for a distributed cache. Specifically, the test persists ditributed timers status on the cache, as per EAP7-1417, but using a remote Infinispan.
During the test one RHSG pod is stopped, in order to verify the timer "persistence" service fail-over mechanism, which actually takes place.
Despite this, we noticed an issue: the logs show that one - or more - EAP instances perform a call to Infinispan via HotRod by using an IP which is no longer listed as part of the cluster topology, and after having received the up to date topology.
- Working EAP 8 pod, using up to date information and showing no issues in Infinispan calls:
... [0m[0m14:45:12,146 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004006: Server sent new topology view (id=-1899142735, age=0) containing 2 addresses: [10.130.2.116:11222, 10.129.2.215:11222] [0m[0m14:45:12,148 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004014: New server added(10.129.2.215:11222), adding to the pool. [0m[0m14:45:12,149 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004014: New server added(10.130.2.116:11222), adding to the pool. ... [0m[0m14:45:35,466 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004006: Server sent new topology view (id=-198812969, age=0) containing 1 addresses: [10.130.2.116:11222] [0m[0m14:45:35,467 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004016: Server not in cluster anymore(10.129.2.215:11222), removing from the pool. ... [0m[0m14:45:45,496 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004006: Server sent new topology view (id=-251298871, age=0) containing 2 addresses: [10.130.2.116:11222, 10.129.2.216:11222] [0m[0m14:45:45,496 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004014: New server added(10.129.2.216:11222), adding to the pool. ...
- EAP pod that is using a removed Infinispan pod address, and failing the call:
... [0m[0m14:44:07,452 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004006: Server sent new topology view (id=-1899142735, age=0) containing 2 addresses: [10.130.2.116:11222, 10.129.2.215:11222] [0m[0m14:44:07,453 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004014: New server added(10.129.2.215:11222), adding to the pool. [0m[0m14:44:07,454 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-0) ISPN004014: New server added(10.130.2.116:11222), adding to the pool. ... [0m[0m14:46:31,449 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004006: Server sent new topology view (id=-251298871, age=0) containing 2 addresses: [10.130.2.116:11222, 10.129.2.216:11222] [0m[0m14:46:31,449 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004014: New server added(10.129.2.216:11222), adding to the pool. [0m[0m14:46:31,450 INFO [org.infinispan.HOTROD] (HotRod-client-async-pool-2) ISPN004016: Server not in cluster anymore(10.129.2.215:11222), removing from the pool. [0m[31m14:47:05,274 ERROR [org.infinispan.HOTROD] (HotRod-client-async-pool-16) ISPN004007: Exception encountered. Retry 10 out of 10: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: /10.129.2.215:11222 ... at io.netty.netty-common@4.1.94.Final-redhat-00001//io.netty.util.internal.ThreadExecutorMap.run(ThreadExecutorMap.java:74) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.wildfly.clustering.context@8.0.0.GA-redhat-00009//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49) ...
This affects EAP 8 CR 1, and is filed as a blocker targeting EAP 8 GA for two reasons:
1. it could affect the Infinispan subsystem outside of the scope of the RFE which has been tested
2. to be considered for fix before EAP 8 GA.
- is caused by
-
ISPN-15239 HotRod client repeatedly retries iteration operation against failed server after removal from topology
- Resolved
- is cloned by
-
JBEAP-26515 EAP 8 on OpenShift - slow topology update on ARM and s390x due to jgroups socket connection errors
- New
-
JBEAP-26118 (7.4.z) ISPN-15239 - HotRod calls to remote caches use outdated topology information
- Reopened
- is incorporated by
-
JBEAP-26016 (8.0.z) Upgrade Infinispan to 14.0.20.Final
- Closed
-
JBEAP-26770 (8.0.z) Upgrade EAP codebase to 8.0.3.GA-redhat-SNAPSHOT in EAP 8.0 Update 2
- Closed
- relates to
-
JBEAP-26224 (8.0.z) Upgrade Infinispan to 14.0.24.Final-redhat-00001
- Closed