Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-9405

Cluster failover not always working when first node not available

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Out of Date
    • Affects Version/s: 9.1.7.Final
    • Fix Version/s: None
    • Component/s: Remote Protocols
    • Labels:
      None

      Description

      If a client configures a main cluster with the first node unavailable and a backup cluster, ping-pong failover ends up failing with the first unavailable node failing. E.g.

      19:16:29,491 ERROR (testng-NonExistentNodeSiteDownFailoverTest) [RetryOnFailureOperation] ISPN004007: Exception encountered. Retry 3 out of 3
      org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
      	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:426)
      	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:258)
      	at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:40)
      	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:55)
      	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.get(RemoteCacheImpl.java:366)
      	at org.infinispan.client.hotrod.xsite.NonExistentNodeSiteDownFailoverTest.testRotateSiteDownFailover(NonExistentNodeSiteDownFailoverTest.java:51)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
      
      	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:426)
      	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:258)
      	at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:40)
      	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:55)
      	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.get(RemoteCacheImpl.java:366)
      	at org.infinispan.client.hotrod.xsite.NonExistentNodeSiteDownFailoverTest.testRotateSiteDownFailover(NonExistentNodeSiteDownFailoverTest.java:51)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
      	at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
      	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
      	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
      	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
      	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
      	at org.testng.TestRunner.privateRun(TestRunner.java:767)
      	at org.testng.TestRunner.run(TestRunner.java:617)
      	at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
      	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
      	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
      	at org.testng.SuiteRunner.run(SuiteRunner.java:254)
      	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
      	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
      	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
      	at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
      	at org.testng.TestNG.run(TestNG.java:1057)
      	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
      	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
      Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: localhost:1234
      	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:95)
      	at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:38)
      	at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:17)
      	at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
      	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:420)
      	... 28 more
      Caused by: java.net.ConnectException: Connection refused
      	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
      	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
      	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.connectSocket(TcpTransport.java:107)
      	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:84)
      	... 32 more
      

      This issue only affects 9.1.x, master works fine.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  galder.zamarreno Galder Zamarreño
                  Reporter:
                  galder.zamarreno Galder Zamarreño
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: