Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-5628

Master endpoint sometimes fails to clean up its ZooKeeper entry

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Minor Minor
    • fuse-7.0
    • jboss-fuse-6.3
    • Fabric8 v1
    • % %
    • Hide
      1. Run JBoss Fuse in Fabric mode. Only 1 node root is sufficient.
      2. Install and deploy the reproducer master-controlbus: 
        $ mvn clean install
$ mvn fabric8:deploy

        Next, in the karaf console run the following command:

        JBossFuse:karaf@root> fabric:container-add-profile root issue-master-controlbus
      3. Create test messages on the `TEST` queue: 
        JBossFuse:karaf@root> activemq:producer --user admin --password admin
      4. Observe ZK entries growing indefinitely: 
        JBossFuse:karaf@root> watch zk:list -r -d /fabric/registry/clusters/camel/master/issue-master-controlbus
      Show
      Run JBoss Fuse in Fabric mode. Only 1 node root is sufficient. Install and deploy the reproducer master-controlbus : $ mvn clean install $ mvn fabric8:deploy Next, in the karaf console run the following command: JBossFuse:karaf@root> fabric:container-add-profile root issue-master-controlbus Create test messages on the `TEST` queue: JBossFuse:karaf@root> activemq:producer --user admin --password admin Observe ZK entries growing indefinitely: JBossFuse:karaf@root> watch zk:list -r -d /fabric/registry/clusters/camel/master/issue-master-controlbus

      In Fabric, when a Master endpoint is shut down, it sometimes fails to clean up its ZK entry and the stale entries remain in the ZK registry even after restart of the route. Thus, in a use case like frequently stopping and restarting a route with a Master endpoint, the stale ZK entries grow indefinitely.

      It happens quite reliably and rapidly when Master component is used with a scheduled route policy (for starting), control bus (for stopping), and delayer DSL, like the route below:

      ShutdownStrategy shutdownStrategy = getContext().getShutdownStrategy(); 
shutdownStrategy.setTimeUnit(TimeUnit.MILLISECONDS);
shutdownStrategy.setTimeout(1000);

CronScheduledRoutePolicy startPolicy = new CronScheduledRoutePolicy();
startPolicy.setRouteStartTime("0/20 * * * * ?");

from("master:issue-master-controlbus:activemq:queue:TEST").routeId("MasterControlBusRoute")
    .routePolicy(startPolicy).noAutoStartup()
    .toF("controlbus:route?routeId=%s&action=stop&async=true", "MasterControlBusRoute")
    .delay(1000)
    .log("${body}");

      You can confirm the stale entries growing by running this command:

      BossFuse:karaf@root> zk:list -r -d /fabric/registry/clusters/camel/master/issue-master-controlbus
/fabric/registry/clusters/camel/master/issue-master-controlbus/00000000031 = {"id":"issue-master-controlbus","container":"root","uuid":"93bba515-be0e-41d1-aa38-5dcef9b44303","consumer":"activemq://queue:TEST","started":true}
/fabric/registry/clusters/camel/master/issue-master-controlbus/00000000152 = {"id":"issue-master-controlbus","container":"root","uuid":"6f416f09-9402-4898-b3e4-77e97dd5fc44","consumer":"activemq://queue:TEST","started":true}
/fabric/registry/clusters/camel/master/issue-master-controlbus/00000000032 = {"id":"issue-master-controlbus","container":"root","uuid":"84fae826-3937-4d4e-8ba9-f93ca95578b9","consumer":"activemq://queue:TEST","started":true}
...

      Note this issue has been there even in Fuse 6.2.x, but from 6.3 it's much accelerated and starts to happen quite frequently.

        1. master-controlbus.zip
          7 kB

              rhn-support-tasato Tadayoshi Sato
              rhn-support-tasato Tadayoshi Sato
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: