Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-11033

Cluster fails while inserting data for a while

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Blocker
    • None
    • 10.0.1.Final
    • Server
    • None

    Description

      Inserting data into an Infinispan cluster works for a while, and then the cluster fails. Showing the following log messages in one pod:

      14:20:34,432 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p4-t1) ISPN000136: Error executing command ReplaceCommand on Cache 'devices', writing keys [WrappedByteArray{bytes=8201\*\i\o\.\e\n\m\a\s\s\e\.\i\o\t\.\i\n\f\i\n\i\s\p\a\n\.\d\e\v\i\c\e\.\D\e\v\i\c\e\K\e\y8A01\<0A1F\j\b\t\e\s\t\.\i\o\t\/\2\0\1\9\-\1\2\-\0\4\T\0\8\:\2\5\:\3\4\Z1219\h\t\t\p\-\i\n\s\e\r\t\e\r\-\f\r\8\l\m\1\5\2\2\4\7, hashCode=-381217399}]: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray{bytes=8201\*\i\o\.\e\n\m\a\s\s\e\.\i\o\t\.\i\n\f\i\n\i\s\p\a\n\.\d\e\v\i\c\e\.\D\e\v\i\c\e\K\e\y8A01\<0A1F\j\b\t\e\s\t\.\i\o\t\/\2\0\1\9\-\1\2\-\0\4\T\0\8\:\2\5\:\3\4\Z1219\h\t\t\p\-\i\n\s\e\r\t\e\r\-\f\r\8\l\m\1\5\2\2\4\7, hashCode=-381217399} and requestor GlobalTx:infinispan-8-8720:1383960. Lock is held by GlobalTx:infinispan-8-8720:33804
      	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:292)
      	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:222)
      	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:440)
      	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.lambda$toInvocationStage$3(InfinispanLock.java:416)
      	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
      	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
      	at org.infinispan.commons.util.concurrent.CallerRunsRejectOnShutdownPolicy.rejectedExecution(CallerRunsRejectOnShutdownPolicy.java:19)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
      	at org.infinispan.executors.LazyInitializingExecutorService.execute(LazyInitializingExecutorService.java:138)
      	at java.base/java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:568)
      	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:638)
      	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
      	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
      	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.notifyListeners(InfinispanLock.java:527)
      	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.cancel(InfinispanLock.java:382)
      	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.call(DefaultLockManager.java:286)
      	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.call(DefaultLockManager.java:222)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      
      

      While showing the following message in the other nodes log:

      14:44:26,310 ERROR [org.jgroups.protocols.TCP] (jgroups-133,infinispan-3-50867) JGRP000034: infinispan-3-50867: failure sending message to infinispan-8-17029: java.net.SocketTimeoutException: connect timed out
      14:44:28,611 ERROR [org.jgroups.protocols.TCP] (jgroups-133,infinispan-3-50867) JGRP000034: infinispan-3-50867: failure sending message to infinispan-8-17029: java.net.SocketTimeoutException: connect timed out
      14:44:30,912 ERROR [org.jgroups.protocols.TCP] (jgroups-126,infinispan-3-50867) JGRP000034: infinispan-3-50867: failure sending message to infinispan-8-17029: java.net.SocketTimeoutException: connect timed out
      

      The node showing the exception gets killed after a while by Kubernetes:

      NAME            READY   STATUS                 RESTARTS   AGE
      infinispan-0    1/1     Running                0          83m
      infinispan-1    1/1     Running                0          83m
      infinispan-10   1/1     Running                0          83m
      infinispan-11   1/1     Running                0          83m
      infinispan-2    1/1     Running                0          83m
      infinispan-3    1/1     Running                0          83m
      infinispan-4    1/1     Running                0          83m
      infinispan-5    1/1     Running                0          83m
      infinispan-6    1/1     Running                0          83m
      infinispan-7    1/1     Running                0          83m
      infinispan-8    0/1     CreateContainerError   3          83m
      infinispan-9    1/1     Running                0          83m
      

      But it never becomes ready again.

      Attachments

        1. deviceManagement.proto
          1 kB
        2. infinispan.xml
          4 kB
        3. logs-2.tar.gz
          259 kB

        Activity

          People

            Unassigned Unassigned
            jreimann-2 Jens Reimann
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: