Uploaded image for project: 'JGroups'
  1. JGroups
  2. JGRP-1927

RELAY2: Delays during shutdown

    XMLWordPrintable

Details

    • Enhancement
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Done
    • 3.6.2
    • 3.6.4
    • None

    Description

      Say we have 2 clusters connected via RELAY2, AB and CD. A is the site master of AB, C is the site master of CD, and A is the coordinator of the bridge cluster.

      When node A is stopped, it first leaves the AB cluster, and then the bridge cluster. When B receives the new B view, it tries to join the bridge cluster. But because RELAY2 hasn't yet finished stopping the relay channel on A, B still sees A as the coordinator of the bridge channel, and sends the JOIN_REQ message to it.

      The bridge channel stops on A before receiving B's join response, so B won't receive any response and will eventually time out. If the user tries to stop node B, it will block waiting for the JChannel lock:

      "testng@1" prio=5 tid=0x1 nid=NA waiting for monitor entry
        java.lang.Thread.State: BLOCKED
      	 waiting for Incoming-1,NodeB@2894 to release lock on <0xd26> (a org.jgroups.JChannel)
      	  at org.jgroups.JChannel.close(JChannel.java:385)
      	  at org.jgroups.util.Util.close(Util.java:408)
      	  at org.jgroups.protocols.relay.Relayer$Bridge.stop(Relayer.java:256)
      	  at org.jgroups.protocols.relay.Relayer.stop(Relayer.java:109)
      	  at org.jgroups.protocols.relay.RELAY2.stop(RELAY2.java:280)
      	  at org.jgroups.stack.ProtocolStack.stopStack(ProtocolStack.java:1015)
      	  at org.jgroups.JChannel.stopStack(JChannel.java:1003)
      	  at org.jgroups.JChannel.disconnect(JChannel.java:373)
      	  - locked <0xd54> (a org.jgroups.JChannel)
      	  at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:262)
      	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
      	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	  at java.lang.reflect.Method.invoke(Method.java:497)
      
      "Incoming-1,B@2894" prio=5 tid=0x2f nid=NA runnable
        java.lang.Thread.State: RUNNABLE
      	 blocks testng@1
      	  at org.jgroups.protocols.pbcast.ClientGmsImpl.joinInternal(ClientGmsImpl.java:114)
      	  at org.jgroups.protocols.pbcast.ClientGmsImpl.join(ClientGmsImpl.java:44)
      	  at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1084)
      	  at org.jgroups.protocols.FlowControl.down(FlowControl.java:353)
      	  at org.jgroups.protocols.FRAG2.down(FRAG2.java:136)
      	  at org.jgroups.protocols.RSVP.down(RSVP.java:153)
      	  at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1038)
      	  at org.jgroups.JChannel.down(JChannel.java:791)
      	  at org.jgroups.JChannel._connect(JChannel.java:564)
      	  at org.jgroups.JChannel.connect(JChannel.java:294)
      	  - locked <0xd26> (a org.jgroups.JChannel)
      	  at org.jgroups.JChannel.connect(JChannel.java:279)
      	  at org.jgroups.protocols.relay.Relayer$Bridge.start(Relayer.java:250)
      	  at org.jgroups.protocols.relay.Relayer.start(Relayer.java:86)
      	  at org.jgroups.protocols.relay.RELAY2.startRelayer(RELAY2.java:681)
      	  at org.jgroups.protocols.relay.RELAY2.handleView(RELAY2.java:663)
      	  at org.jgroups.protocols.relay.RELAY2.up(RELAY2.java:416)
      	  at org.jgroups.protocols.RSVP.up(RSVP.java:201)
      	  at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
      	  at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
      	  at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
      	  at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
      	  - locked <0xd2e> (a org.jgroups.protocols.pbcast.GMS)
      	  at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:146)
      

      Ideally, I'd like B to realize that the coordinator of the bridge cluster has the same site id, and delay sending the join request until C becomes the coordinator. Failing that, maybe JChannel.stop could interrupt the thread doing the join?

      Attachments

        Issue Links

          Activity

            People

              rhn-engineering-bban Bela Ban
              dberinde@redhat.com Dan Berindei
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: