Fix Flaky AbstractValidationUnitTest

Fix Flaky AbstractValidationUnitTest

Using the flaky test discovery tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] there are ~137 flaky tests in Wildfly.

This JIRA issue aims to fix a subset of ~47 flaky tests in the dist, ee-dist, and preview-dist modules, since there is a common source of flakiness.

To replicate the flaky tests errors:

# 1. Setup NonDex
git clone https://github.com/TestingResearchIllinois/NonDex.git
cd NonDex
mvn clean install -DskipTests

# 2. Run NonDex in Wildfly on the target maven modules
mvn clean -am -pl dist edu.illinois:nondex-maven-plugin:2.1.7-SNAPSHOT:nondex -Dtest=org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase -DnondexRuns=10 -DfailIfNoTests=false
mvn clean -am -pl ee-dist edu.illinois:nondex-maven-plugin:2.1.7-SNAPSHOT:nondex -Dtest=org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase -DnondexRuns=10 -DfailIfNoTests=false
mvn clean -am -pl preview/dist edu.illinois:nondex-maven-plugin:2.1.7-SNAPSHOT:nondex -Dtest=org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase -DnondexRuns=10 -DfailIfNoTests=false

NonDex will run each test case multiple times. Some runs NonDex shuffles the the output of non-deterministic Java APIs (i.e. HashMap keySet()), while in other runs NonDex will not shuffle non-deterministic APIs. NonDex will provide a summary of all tests, showing tests that failed due to shuffling (flaky) and how those test pass without shuffling. For example, in the ee-dist you will see output similar to this:

INFO: Surefire failed when running tests for hBG69zCLvYHCaIKs0AasjTFCdGbX6WTKb05H97ZNIis=
[INFO] NonDex SUMMARY:
[INFO] *********
[INFO] mvn nondex:nondex  -DnondexFilter='.*' -DnondexMode=FULL -DnondexSeed=933178 -DnondexStart=0 -DnondexEnd=9223372036854775807 -DnondexPrintstack=false -DnondexDir="/root/wildfly/ee-dist/.nondex" -DnondexJarDir="/root/wildfly/ee-dist/.nondex" -DnondexExecid=5RBCJBZsiLqnSBfjJ9jn2YW0lSnqn0ab43cDSkQNTg= -DnondexLogging=CONFIG
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2FullHA
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipHA
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2HA
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipFullHA
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
[WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS

Flakiness Cause

The group of 47 tests targeted in this Jira (see list of tests at the bottom) flaky tests in (dist, ee-dist, and preview-dist modules) rely on a shared `parseXml` function in `AbstractValidationUnitTest`:

protected void parseXml(String xmlName) throws SAXException, IOException {
  final File xmlFile = getXmlFile(xmlName);     
  SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);     
  schemaFactory.setErrorHandler(new ErrorHandlerImpl(xmlFile.toPath()));     
  schemaFactory.setResourceResolver(DEFAULT_RESOURCE_RESOLVER);     
  Schema schema = schemaFactory.newSchema(SCHEMA_SOURCES);     
  Validator validator = schema.newValidator();     
  validator.setErrorHandler(new ErrorHandlerImpl(xmlFile.toPath()));
  validator.setFeature("http://apache.org/xml/features/validation/schema", true);
  validator.setResourceResolver(DEFAULT_RESOURCE_RESOLVER);
  validator.validate(new StreamSource(xmlFile));  //noinspection ResultOfMethodCallIgnored     
  xmlFile.delete(); 
}

During NonDex runs, the single point of failure is: `Schema schema = schemaFactory.newSchema(SCHEMA_SOURCES);`. The `SCHEMA_SOURCES` is an array of `StreamSource` objects whose order depends on the a nondeterministic listing of files in a directory:

final File schemaDir = new File(JBOSS_DIST_DIR, SCHEMAS_LOCATION);
final File[] xsds = schemaDir.listFiles(new SchemaFilter(EXCLUDED_SCHEMA_FILES.toArray(new String[0])));

The files are passed through a series of HashMaps which are read using nondeterministic `keySet()` and `values()` apis to eventually populate `SCHEMA_SOURCES`. However, the order of the `SCHEMA_SOURCES` matters since some schemas may be dependent on each other, (i.e. one schema importing another)

How To Fix

Enforce a deterministic non-conflicting order of `SCHEMA_SOURCES`. 

For example, sort the files returned by `schemaDir.listFiles(new SchemaFilter(EXCLUDED_SCHEMA_FILES.toArray(new String[0])));`, convert `JBOSS_SCHEMAS_MAP` to a LinkedHashMap, and convert `CURRENT_JBOSS_SCHEMAS_MAP` to a LinkedHashMap.

Group of 47 Flaky Test Cases

wildfly-preview-dist

1. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
2. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
3. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
4. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
5. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
6. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
7. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
8. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
9. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneActiveMQEmbedded
10. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
11. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
12. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
13. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS

wildfly-dist

1. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2FullHA
2. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
3. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
4. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
5. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
6. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
7. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
8. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
9. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipHA
10. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMicroProfileHA
11. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
12. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2HA
13. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMicroProfile
14. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipFullHA
15. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
16. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
17. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
18. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS

wildfly-ee-dist

1. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2FullHA
2. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
3. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
4. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
5. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
6. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
7. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
8. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
9. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipHA
10. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
11. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2HA
12. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipFullHA
13. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
14. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
15. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
16. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS