Uploaded image for project: 'Infinispan Spark'
  1. Infinispan Spark
  2. ISPRK-22

InfinispanRDD is not fault tolerant

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 0.3
    • Fix Version/s: 0.4
    • Component/s: RDD
    • Labels:
      None

      Description

      When primary ISPN server fails during processing InfinispanRDD, Spark is not able to overcome this failure.
      This is caused by re-creating RemoteCachManager with pre-configured ISPN server address (for read here, for writes here), so when this server fails during RDD processing and Spark calls some function, which under the hood creates RemoteCacheManager, it will fail with connection refused exception.

      Here are some basic tests and example of exception thrown by HR client:

      org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:395)
              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:241)
              at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:26)
              at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)
              at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:490)
              at org.infinispan.client.hotrod.impl.RemoteCacheImpl.resolveCompatibility(RemoteCacheImpl.java:551)
              at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:341)
              at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:222)
              at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:217)
              at org.infinispan.spark.rdd.InfinispanRDD$$anonfun$1.apply(InfinispanRDD.scala:52)
              at org.infinispan.spark.rdd.InfinispanRDD$$anonfun$1.apply(InfinispanRDD.scala:52)
              at scala.Option.map(Option.scala:146)
              at org.infinispan.spark.rdd.InfinispanRDD.compute(InfinispanRDD.scala:52)
              at org.infinispan.spark.rdd.InfinispanRDD.compute(InfinispanRDD.scala:66)
              at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
              at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
              at org.apache.spark.scheduler.Task.run(Task.scala:89)
              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /127.0.0.1:11222
              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)
              at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:35)
              at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)
              at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:390)
              ... 21 more
      Caused by: java.net.ConnectException: Connection refused
              at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
              at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
              at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)
              ... 25 more
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              gfernand@redhat.com Gustavo Fernandes
              Reporter:
              vjuranek@redhat.com Vojtech Juranek
              Archiver:
              samahaja@redhat.com Sagar Mahajan

                Dates

                Created:
                Updated:
                Resolved:
                Archived: