Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-1826

Hot Rod protocol fails to retrieve correct topology after node join

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • 5.1.1.FINAL
    • 5.1.0.FINAL
    • None
    • None

      Another resilience test: 4 nodes node2 goes down and up again:
      http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-failover-dist-basic/16/artifact/report/stats-throughput.png

      Test timeline goes like this:

      2012-02-01 04:39:21,993: 20 hotrod clients started
      2012-02-01 04:42:21,995: node02 killed
      2012-02-01 04:45:27,950: node02 up again

      Endpoints:
      172.18.1.1:11222 - node01
      172.18.1.3:11222 - node02
      172.18.1.5:11222 - node03
      172.18.1.7:11222 - node04

      in the client side log
      http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-failover-dist-basic/16/console-edg-perf06/consoleText
      http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-failover-dist-basic/16/console-edg-perf07/consoleText

      some warnings about connection refused and then node02 being removed from the pool by Timer thread:

      2012-02-01 04:43:15,608 254109 INFO  [org.infinispan.client.hotrod.impl.protocol.Codec11] (Timer-1:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.7:11222, /172.18.1.5:11222]
2012-02-01 04:43:15,609 254110 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool.
2012-02-01 04:43:15,610 254111 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool.
2012-02-01 04:43:15,611 254112 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool.
2012-02-01 04:43:15,612 254113 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004016: Server not in cluster anymore(/172.18.1.3:11222), removing from the pool.

      Log of Client-19

      2012-02-01 04:39:20,962 19437 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Statically configured servers: [/172.18.1.7:11222, /172.18.1.5:11222, /172.18.1.3:11222, /172.18.1.1:11222]
2012-02-01 04:39:20,962 19437 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Load balancer class: org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy
2012-02-01 04:39:20,963 19438 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Tcp no delay = true; client socket timeout = 120000 ms; connect timeout = 120000 ms
2012-02-01 04:39:20,996 19471 INFO  [org.infinispan.client.hotrod.impl.protocol.Codec11] (Client-19:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.3:11222, /172.18.1.7:11222, /172.18.1.5:11222]
2012-02-01 04:39:20,997 19472 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool.
2012-02-01 04:39:20,998 19473 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.3:11222), adding to the pool.
2012-02-01 04:39:20,999 19474 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool.
2012-02-01 04:39:21,000 19475 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool.
2012-02-01 04:45:26,202 384677 INFO  [org.infinispan.client.hotrod.impl.protocol.Codec11] (Client-19:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.7:11222, /172.18.1.5:11222]
2012-02-01 04:45:26,206 384681 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool.
2012-02-01 04:45:26,207 384682 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool.
2012-02-01 04:45:26,208 384683 INFO  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool.

      it seems like they get new topology update but without newly started node02

              rh-ee-galder Galder Zamarreño
              mlinhard Michal Linhard (Inactive)
              Archiver:
              rhn-support-adongare Amol Dongare

                Created:
                Updated:
                Resolved:
                Archived: