Identify the reason why the failing tests that needed https://github.com/wildfly/wildfly/commit/90643bc435a9ba439cf2988e5b7407d1d48fb040 to get working did not fail the build. Those tests were failing for 2.5 months. Fortunately it was just a test problem, not a broken feature.
They involved separate surefire executions. We want to avoid those, but sometimes they are needed to use different config files. We have separate executions in testsuite/integration/basic as well, covering a much broader scope, so if this kind of problem happened there as well we could have a big problem.