Uploaded image for project: 'JGroups'
  1. JGroups
  2. JGRP-2250

Stuck Threads - BLOCKED infinitely on - JGroups - org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Rejected
    • Affects Version/s: 3.4.1
    • Fix Version/s: None

      Description

      Ours is a J2EE application (clustered across 4 nodes) internally using Infinispan Cache (Ver 6.0.2) with JGroups (Ver 3.4.1) as transport layer of our cache.

      Application had been running perfectly fine, but suddenly on one node, one cache-put got stuck with below stack trace. (Cause being blocked on org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)).

      Now all further puts are getting stuck - blocked - on the same line on the same node. Other nodes are working fine in this respect.

      Currently, while raising the issue, the threads are still in stuck state. And the first stuck thread is stuck from last 27 hours.

      Stack Trace:
      "[STUCK] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" #107862 daemon prio=1 os_prio=0 tid=0x00007f5964373000 nid=0x75a9 in Object.wait() [0x00007f57d596b000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)

      • locked <0x00007f5e64893168> (a org.jgroups.protocols.FlowControl$Credit)
        at org.jgroups.protocols.UFC.handleDownMessage(UFC.java:121)
        at org.jgroups.protocols.FlowControl.down(FlowControl.java:330)
        at org.jgroups.protocols.FlowControl.down(FlowControl.java:340)
        at org.jgroups.protocols.FRAG2.down(FRAG2.java:136)
        at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1024)
        at org.jgroups.JChannel.down(JChannel.java:760)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:683)
        at org.jgroups.blocks.RequestCorrelator.sendUnicastRequest(RequestCorrelator.java:202)
        at org.jgroups.blocks.UnicastRequest.sendRequest(UnicastRequest.java:43)
        at org.jgroups.blocks.Request.execute(Request.java:83)
        at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:399)
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353)
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521)
        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)
        at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:233)
        at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:72)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
        at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321)
        at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)
        at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
        at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:219)
        at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:141)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
        at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
        at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
        at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306)
        at org.infinispan.CacheImpl.putInternal(CacheImpl.java:878)
        at org.infinispan.CacheImpl.put(CacheImpl.java:870)
        at org.infinispan.DecoratedCache.put(DecoratedCache.java:401)
        at com.nucleus.finnone.pro.cache.common.NeutrinoCache.put(NeutrinoCache.java:18)
        at com.nucleus.config.persisted.service.ConfigurationServiceImpl.getConfigurationGroupFor(ConfigurationServiceImpl.java:478)

      Note: Seems related to below-
      https://issues.jboss.org/browse/JGRP-1675
      https://issues.jboss.org/browse/ISPN-3645

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                belaban Bela Ban
                Reporter:
                rohit6600 Rohit Singh
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: