When a merge operation on a versionable node fails, the jcr:mergeFailed property should be set with the ID of the offending node(s), leaving the rest unchanged. This should work even if the node is checked in.