Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-1826

Hot Rod protocol fails to retrieve correct topology after node join

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • 5.1.1.FINAL
    • 5.1.0.FINAL
    • None
    • None

      Another resilience test: 4 nodes node2 goes down and up again:
      http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-failover-dist-basic/16/artifact/report/stats-throughput.png

      Test timeline goes like this:

      2012-02-01 04:39:21,993: 20 hotrod clients started
      2012-02-01 04:42:21,995: node02 killed
      2012-02-01 04:45:27,950: node02 up again

      Endpoints:
      172.18.1.1:11222 - node01
      172.18.1.3:11222 - node02
      172.18.1.5:11222 - node03
      172.18.1.7:11222 - node04

      in the client side log
      http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-failover-dist-basic/16/console-edg-perf06/consoleText
      http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-failover-dist-basic/16/console-edg-perf07/consoleText

      some warnings about connection refused and then node02 being removed from the pool by Timer thread:

      2012-02-01 04:43:15,608 254109 INFO  [org.infinispan.client.hotrod.impl.protocol.Codec11] (Timer-1:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.7:11222, /172.18.1.5:11222]
2012-02-01 04:43:15,609 254110 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool.
2012-02-01 04:43:15,610 254111 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool.
2012-02-01 04:43:15,611 254112 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool.
2012-02-01 04:43:15,612 254113 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004016: Server not in cluster anymore(/172.18.1.3:11222), removing from the pool.

      Log of Client-19

      2012-02-01 04:39:20,962 19437 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Statically configured servers: [/172.18.1.7:11222, /172.18.1.5:11222, /172.18.1.3:11222, /172.18.1.1:11222]
2012-02-01 04:39:20,962 19437 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Load balancer class: org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy
2012-02-01 04:39:20,963 19438 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Tcp no delay = true; client socket timeout = 120000 ms; connect timeout = 120000 ms
2012-02-01 04:39:20,996 19471 INFO  [org.infinispan.client.hotrod.impl.protocol.Codec11] (Client-19:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.3:11222, /172.18.1.7:11222, /172.18.1.5:11222]
2012-02-01 04:39:20,997 19472 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool.
2012-02-01 04:39:20,998 19473 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.3:11222), adding to the pool.
2012-02-01 04:39:20,999 19474 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool.
2012-02-01 04:39:21,000 19475 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool.
2012-02-01 04:45:26,202 384677 INFO  [org.infinispan.client.hotrod.impl.protocol.Codec11] (Client-19:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.7:11222, /172.18.1.5:11222]
2012-02-01 04:45:26,206 384681 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool.
2012-02-01 04:45:26,207 384682 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool.
2012-02-01 04:45:26,208 384683 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool.

      it seems like they get new topology update but without newly started node02

              rh-ee-galder Galder Zamarreño
              mlinhard Michal Linhard (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: