Uploaded image for project: 'JGroups'
  1. JGroups
  2. JGRP-2533

Internal messages should use DONT_LOOPBACK

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Minor
    • 4.2.12, 5.1.6
    • 4.0.21, 4.2.11
    • None
    • False
    • False
    • Undefined

    Description

      Looping back internal messages is expensive: TP submits a task to the thread pool for the loopback itself, and then UNICAST3 submits another task to handle other regular message that might need to be delivered:

      OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at org.jgroups.protocols.TP.runInNewThread(TP.java:1410)
	at org.jgroups.protocols.TP.submitToThreadPool(TP.java:1394)
	at org.jgroups.protocols.TP.submitToThreadPool(TP.java:1391)
	at org.jgroups.protocols.TP.submitToThreadPool(TP.java:1377)
	at org.jgroups.protocols.UNICAST3.processInternalMessage(UNICAST3.java:776)
	at org.jgroups.protocols.UNICAST3.handleDataReceivedFromSelf(UNICAST3.java:766)
	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:387)
	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:590)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:131)
	at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:203)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:253)
	at org.jgroups.protocols.MERGE3.up(MERGE3.java:280)
	at org.jgroups.protocols.Discovery.up(Discovery.java:295)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1250)
	at org.jgroups.util.SubmitToThreadPool.lambda$loopback$0(SubmitToThreadPool.java:30)
	at java.lang.Thread.run(Thread.java:748)

      I noticed lots of protocols are sending messages with loopback enabled and fail to process those loopback messages when the JVM is unable to create more temporary threads:

      	at org.jgroups.protocols.FlowControl.sendCredit(FlowControl.java:480)

	at org.jgroups.protocols.FD_ALL$HeartbeatSender.run(FD_ALL.java:416)

	at org.jgroups.protocols.pbcast.STABLE.sendStableMessage(STABLE.java:668)

	at org.jgroups.protocols.pbcast.STABLE._sendStabilityMessage(STABLE.java:736)

	at org.jgroups.protocols.FD_SOCK.broadcastUnuspectMessage(FD_SOCK.java:710)

      I don't know if there are any consequences of those messages being dropped locally, but I don't think the possibility should exist at all, and that work should be done directly instead of sending a message to self.

      Note: line numbers are from 4.0.21.Final.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JGRP-2532 FlowControl should not send credits to self for DONT_LOOPBACK messages

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved

          Activity

            Public project attachment banner

              context keys: [headless, issue, helper, isAsynchronousRequest, project, action, user]
              current Project key: JGRP

              People

                rhn-engineering-bban Bela Ban
                dberinde@redhat.com Dan Berindei
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved: