Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-18882

Clustering: org.infinispan.util.concurrent.TimeoutException in rolling update tests on OpenShift 4

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Major
    • None
    • 7.3.0.GA
    • Clustering
    • None

    Description

      The error happens during a rolling update on OpenShift 4;

      The first replica

      {eap-rolling-1}

      is scaled down from 5 to 0 while the second replica

      {eap-rolling-2}

      is scaled up from 0 to 5;

      The error happens on the second replica after roll out completes:

      eap-rolling-2-4nzf5:

      14:30:06,947 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-njjx6 joined the cluster
      14:30:06,949 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-njjx6 joined the cluster
      14:30:06,952 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-njjx6 joined the cluster
      14:30:06,954 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-njjx6 joined the cluster
      14:30:07,855 INFO  [org.infinispan.CLUSTER] (thread-17,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-vs6db joined the cluster
      14:30:07,856 INFO  [org.infinispan.CLUSTER] (thread-17,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-vs6db joined the cluster
      14:30:07,861 INFO  [org.infinispan.CLUSTER] (thread-17,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-vs6db joined the cluster
      14:30:07,862 INFO  [org.infinispan.CLUSTER] (thread-17,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-vs6db joined the cluster
      
      14:30:21,257 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-brgng left the cluster
      14:30:21,260 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-brgng left the cluster
      14:30:21,261 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-brgng left the cluster
      14:30:21,268 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-brgng left the cluster
      
      14:30:24,900 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-ltqqr left the cluster
      14:30:24,902 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-ltqqr left the cluster
      14:30:24,903 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-ltqqr left the cluster
      14:30:24,906 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-ltqqr left the cluster
      14:30:24,927 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-hpnns left the cluster
      14:30:24,928 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-hpnns left the cluster
      14:30:24,930 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-hpnns left the cluster
      14:30:24,932 INFO  [org.infinispan.CLUSTER] (thread-6,null,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-hpnns left the cluster
      
      14:30:44,863 INFO  [org.infinispan.CLUSTER] (thread-13,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-6fpvw joined the cluster
      14:30:44,866 INFO  [org.infinispan.CLUSTER] (thread-13,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-6fpvw joined the cluster
      14:30:44,869 INFO  [org.infinispan.CLUSTER] (thread-13,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-6fpvw joined the cluster
      14:30:44,869 INFO  [org.infinispan.CLUSTER] (thread-13,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-6fpvw joined the cluster
      
      14:30:46,382 INFO  [org.infinispan.CLUSTER] (thread-17,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-v7js2 joined the cluster
      14:30:46,383 INFO  [org.infinispan.CLUSTER] (thread-17,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-v7js2 joined the cluster
      14:30:46,384 INFO  [org.infinispan.CLUSTER] (thread-17,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-v7js2 joined the cluster
      14:30:46,385 INFO  [org.infinispan.CLUSTER] (thread-17,ee,eap-rolling-2-4nzf5) ISPN100000: Node eap-rolling-2-v7js2 joined the cluster
      
      14:31:04,791 INFO  [org.infinispan.CLUSTER] (thread-13,ee,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-nsdzs left the cluster
      14:31:04,794 INFO  [org.infinispan.CLUSTER] (thread-13,ee,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-nsdzs left the cluster
      14:31:04,795 INFO  [org.infinispan.CLUSTER] (thread-13,ee,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-nsdzs left the cluster
      14:31:04,800 INFO  [org.infinispan.CLUSTER] (thread-13,ee,eap-rolling-2-4nzf5) ISPN100001: Node eap-rolling-1-nsdzs left the cluster
      

      eap-rolling-2-v7js2:

      14:32:40,351 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p5-t1) ISPN000136: Error executing command PrepareCommand on Cache 'eap-s2i.ear/eap-s2i-ejb.jar/infinispan', writing keys [InfinispanBeanKey(UUIDSessionID [08e7fb0c-f257-4e3f-b509-d9972a7c1ad3]), InfinispanBeanGroupKey(UUIDSessionID [08e7fb0c-f257-4e3f-b509-d9972a7c1ad3])]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 32 from eap-rolling-2-njjx6,eap-rolling-2-vs6db
      
      ... other 23 errors follow ...
      

      eap-rolling-2-vs6db:

      14:31:49,739 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p17-t1) ISPN000136: Error executing command PrepareCommand on Cache 'eap-s2i.ear/eap-s2i-ejb.jar/infinispan', writing keys [InfinispanBeanKey(UUIDSessionID [5d83e5ba-85f3-40b7-8a6e-f6d77f8eef36]), InfinispanBeanGroupKey(UUIDSessionID [5d83e5ba-85f3-40b7-8a6e-f6d77f8eef36])]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 48 from eap-rolling-2-njjx6
      
      ... other 30 errors follow ...
      

      Attachments

        1. eap-rolling-2-4nzf5.log
          45 kB
        2. eap-rolling-2-6fpvw.log
          31 kB
        3. eap-rolling-2-njjx6.log
          39 kB
        4. eap-rolling-2-v7js2.log
          331 kB
        5. eap-rolling-2-vs6db.log
          441 kB

        Activity

          People

            pferraro@redhat.com Paul Ferraro
            tborgato@redhat.com Tommaso Borgato
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: