Uploaded image for project: 'JGroups'
  1. JGroups
  2. JGRP-706

with dynamic keys in ENCRYPT, a node taking over coordinator can block itself from taking over the view

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 2.6.3, 2.7
    • 2.6.1
    • None

    Description

      This is with JGroups 2.6.1 on Linux 2.6.18, and a cluster of 10-20 nodes with several channels on each node.

      Everything is running smoothly, and then we kill the coordinator (of one or more channels), plus the next few nodes in the view, all at about the same time. SUSPECTs happen, and then the highest-ranking remaining node decides to become coordinator. It tries to mcast a new view, but never gets any view ACKs - so nobody else sees the updated view and the channel remains basically unusable. This only happens sometimes, and usually on just some of several channels - even though they share the same stack config.

      Enabling TRACE logging revealed that ENCRYPT.down() is queuing the new view - we use dynamically-generated keys and apparently the view changes have flipped queue_down to 'true'. We see this, over and over:

      2008-03-02 10:54:47,970 [TRACE] GMS mcasting view

      {[10.193.48.119:33274|693] [10.193.48.119:33274, 10.193.48.115:52962, 10.192.226.30:45130, 10.193.66.111:43645]}

      (4 mbrs)
      2008-03-02 10:54:47,970 [TRACE] ENCRYPT queueing down message as no session key established[dst: <null>, src: <null> (1 headers), size=0 bytes]
      2008-03-02 10:54:49,972 [WARN] GMS failed to collect all ACKs (4) for view [10.193.48.119:33274|693] [10.193.48.119:33274, 10.193.48.115:52962, 10.192.226.30:45130, 10.193.66.111:43645] after 2000ms, missing ACKs from [10.193.48.119:33274, 10.193.48.115:52962, 10.192.226.30:45130, 10.193.66.111:43645] (received=[]), local_addr=10.193.48.119:33274

      Attachments

        1. ENCRYPT.java.patch
          4 kB
        2. ENCRYPT.patch
          2 kB
          Gray Watson

        Activity

          People

            rhn-engineering-bban Bela Ban
            graywatson_jira Gray Watson (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: