if there is possibility to detect docbooks xmls -> try to detect or use some heuristics to detect docbook file type from file content.
other way -> do not put "JBoss Tools Docbook Editor" in Open With menu - or put just in case when you can state what this is docbook file.