Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-6183

Initial state transfer fails with unexpected timeout

    Details

    • Type: Bug
    • Status: New (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 7.2.5.Final
    • Fix Version/s: None
    • Component/s: State Transfer
    • Labels:
      None

      Description

      Hi guys,

      I would like to report a somewhat odd issue with initial state transfer. It was observed in two instances - an Infinispan 7.2.5 cluster with 2 nodes and an Infinispan 7.2.5 cluster with 6 nodes. The two clusters had been running for 2 weeks, the smaller for dev purposes with very light load - about a dozen cached objects. Upon adding an extra node an initial state transfer exception was encountered with both clusters, after about 4 minutes which is the default timeout setting for such situations. Several attempts were made to add a new node, incl. one with increased timeout (10 mins), but state transfer would still not complete, and throw an exception:

      "message": "Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl",
            "name": "org.infinispan.commons.CacheException",
            "cause": {
              "commonElementCount": 25,
              "localizedMessage": "Initial state transfer timed out for cache authzStore.codeMap on ip-10-180-242-223-40643",
              "message": "Initial state transfer timed out for cache authzStore.codeMap on ip-10-180-242-223-40643",
              "name": "org.infinispan.commons.CacheException",
              "extendedStackTrace": [
                {
                  "class": "org.infinispan.statetransfer.StateTransferManagerImpl",
                  "method": "waitForInitialStateTransferToComplete",
                  "file": "StateTransferManagerImpl.java",
                  "line": 222,
                  "exact": false,
                  "location": "StateTransferManagerImpl.class",
                  "version": "?"
                },
      

      The JMX console reported "stateTransferInProgress=true" and "joinComplete=true".

      The original clusters where then shut down and started again together with the new node, after which the clusters were successfully formed.

      Attached is the exception stack trace and the JGroups config (based on the stock S3 ping).

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                pruivo Pedro Ruivo
                Reporter:
                vdzhuvinov Vladimir Dzhuvinov
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: