Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-6473

Git Fails to Initialize After the Master Moves

XMLWordPrintable

    • % %
    • Hide

      (Reproducer in progress)

      I have had some success in reproducing the initialization issue (empty git configuration in cluster-list) by introducing network delays and packet loss in my local network (between containers), forcing intermittent zookeeper connection losses and restarting containers.

      Using netem and tc, I introduced the network conditions with:

      # tc qdisc add dev eth0 root netem delay 250ms 20ms distribution normal

      [EDIT]
      The above was enough to instigate connection loss errors. Packet loss can be added with:

       tc qdisc change dev eth0 root netem delay 250ms 20ms distribution normal loss 0.5% 25%

      [/EDIT]

      I ran this on each host for the interface used by fabric / zookeeper. A second interface was used for monitoring / tailing logs, etc without the settings applied. Some tweaking may be needed, but I was able to get a working 3-node ensemble with periodic zookeeper errors with the settings above.

      Show
      (Reproducer in progress) I have had some success in reproducing the initialization issue (empty git configuration in cluster-list) by introducing network delays and packet loss in my local network (between containers), forcing intermittent zookeeper connection losses and restarting containers. Using netem and tc, I introduced the network conditions with: # tc qdisc add dev eth0 root netem delay 250ms 20ms distribution normal [EDIT] The above was enough to instigate connection loss errors. Packet loss can be added with: tc qdisc change dev eth0 root netem delay 250ms 20ms distribution normal loss 0.5% 25% [/EDIT] I ran this on each host for the interface used by fabric / zookeeper. A second interface was used for monitoring / tailing logs, etc without the settings applied. Some tweaking may be needed, but I was able to get a working 3-node ensemble with periodic zookeeper errors with the settings above.

      We have seen several similar cases in recent weeks / months with git master initialization - the issue seems to impact JBoss Fuse 6.2.1 with rollups (R4, R5, possibly others). Symptoms noted include:

      Sometimes after a zookeeper event causing a git master move, the git configuration is empty or incomplete. 

      > cluster-list | grep git

      returns the word "git" with no container listing or url. In at least one case where this was noted, using the git CLI to query the origin 

      (git remote show origin)

      revealed that the master url had not been updated in data/git/local/fabric/config and was still pointing to the shutdown master.

      Another symptom encountered with at least three customers is that git fails to fully initialize when the master changes to a child container. Running the query above returns:

      "fatal: http://<host>:<port>/git/fabric//info/refs not found: did you run git update-server-info on the server?"

      The servlet directory appears to be in place and populated on the new master, but the servlet seems unresponsive. Forcing the git master onto a root container seems to resolve the issue and the servlet initializes normally.

      A third common symptom noted is that when these problems appear we see "orphan" data/git/local/servlet directories on multiple containers. Ordinarily, these directories don't exist on non-master containers and we normally see them cleaned up after the master switches to a new container, but in the cases where we saw the issues above, we saw the directories lingering on multiple containers.

              ggrzybek Grzegorz Grzybek
              rhn-support-dhawkins Duane Hawkins
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: