Authentication Error when Adding Containers to Ensemble

Note: These directions represent the steps followed to create an ensemble with an external git repo. I have seen this issue pop up without an external repo, as well, but the issue reproduces faithfully in this scenario. 1. Install 3 instances of Fuse 6.2.1 2. Configure etc/io.fabric8.datastore.cfg on each instance to hold the authentication info for external git (workaround for https: //issues.jboss.org/browse/ENTESB-4939 - original reproducer used a locally available gitlab instance) 3. Configure etc/users.properties to give the admin user the same authentication tokens as the git user (user/password) - may not be necessary, but this is how it was configured in my environment ( this may not be necessary to reproduce the issue, but was configured this way in the original reproducer e.g. change "admin=admin,admin... to <gituser>=<gitpasswd>,admin.. in users.properties before first container boot.) 4. Create a fabric from the first node with: fabric:create --wait- for -provisioning --external-git-url http: //gitlab/gituser/ensemble.git --external-git-user <gituser> --external-git-password <gitpassword> --zookeeper-password <gitpassword> 5. Install fabric-zookeeper-utils: fabric:profile-edit --feature fabric-zookeeper-commands/0.0.0 default 6. From the second two nodes, join the first: fabric:join --zookeeper-password <gitpassword> node0:2181 node1, fabric:join --zookeeper-password <gitpassword> node0:2181 node2. I had to restart these containers from once to several times to get them to finish provisioning. 7. From the original fabric container, execute: ensemble-add node1 node2 and answer yes to the warning prompt. 8. Operation fails with: "Error executing command: io.fabric8.api.FabricAuthenticationException: java.lang.SecurityException: Authentication failed" and a corresponding stacktrace in the log. 9. Verify that node /fabric/authentication/containers/root (where root is the nodename for the first ensemble container) is missing with zk:get /fabric/authentication/containers/root. You should see something like "Error executing command: KeeperErrorCode = NoNode for /fabric/authentication/containers/root" 10. Insert the missing nodes with: zk:create /fabric/authentication/containers zk:create /fabric/authentication/containers/root zk:set /fabric/authentication/containers "" zk:set /fabric/authentication/containers/root "" 11. Retry the ensemble-add command. 12. Ensemble expansion now succeeds