-
Quality Risk
-
Resolution: Done
-
Minor
-
30.0.0.Final
-
None
-
Low
-
Other
-
Other
Fix Flaky AbstractValidationUnitTest
Using the flaky test discovery tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] there are ~137 flaky tests in Wildfly.
This JIRA issue aims to fix a subset of ~47 flaky tests in the dist, ee-dist, and preview-dist modules, since there is a common source of flakiness.
To replicate the flaky tests errors:
# 1. Setup NonDex git clone https://github.com/TestingResearchIllinois/NonDex.git cd NonDex mvn clean install -DskipTests # 2. Run NonDex in Wildfly on the target maven modules mvn clean -am -pl dist edu.illinois:nondex-maven-plugin:2.1.7-SNAPSHOT:nondex -Dtest=org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase -DnondexRuns=10 -DfailIfNoTests=false mvn clean -am -pl ee-dist edu.illinois:nondex-maven-plugin:2.1.7-SNAPSHOT:nondex -Dtest=org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase -DnondexRuns=10 -DfailIfNoTests=false mvn clean -am -pl preview/dist edu.illinois:nondex-maven-plugin:2.1.7-SNAPSHOT:nondex -Dtest=org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase -DnondexRuns=10 -DfailIfNoTests=false
NonDex will run each test case multiple times. Some runs NonDex shuffles the the output of non-deterministic Java APIs (i.e. HashMap keySet()), while in other runs NonDex will not shuffle non-deterministic APIs. NonDex will provide a summary of all tests, showing tests that failed due to shuffling (flaky) and how those test pass without shuffling. For example, in the ee-dist you will see output similar to this:
INFO: Surefire failed when running tests for hBG69zCLvYHCaIKs0AasjTFCdGbX6WTKb05H97ZNIis= [INFO] NonDex SUMMARY: [INFO] ********* [INFO] mvn nondex:nondex -DnondexFilter='.*' -DnondexMode=FULL -DnondexSeed=933178 -DnondexStart=0 -DnondexEnd=9223372036854775807 -DnondexPrintstack=false -DnondexDir="/root/wildfly/ee-dist/.nondex" -DnondexJarDir="/root/wildfly/ee-dist/.nondex" -DnondexExecid=5RBCJBZsiLqnSBfjJ9jn2YW0lSnqn0ab43cDSkQNTg= -DnondexLogging=CONFIG [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2FullHA [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipHA [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2HA [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipFullHA [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS
Flakiness Cause
The group of 47 tests targeted in this Jira (see list of tests at the bottom) flaky tests in (dist, ee-dist, and preview-dist modules) rely on a shared `parseXml` function in `AbstractValidationUnitTest`:
protected void parseXml(String xmlName) throws SAXException, IOException { final File xmlFile = getXmlFile(xmlName); SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI); schemaFactory.setErrorHandler(new ErrorHandlerImpl(xmlFile.toPath())); schemaFactory.setResourceResolver(DEFAULT_RESOURCE_RESOLVER); Schema schema = schemaFactory.newSchema(SCHEMA_SOURCES); Validator validator = schema.newValidator(); validator.setErrorHandler(new ErrorHandlerImpl(xmlFile.toPath())); validator.setFeature("http://apache.org/xml/features/validation/schema", true); validator.setResourceResolver(DEFAULT_RESOURCE_RESOLVER); validator.validate(new StreamSource(xmlFile)); //noinspection ResultOfMethodCallIgnored xmlFile.delete(); }
During NonDex runs, the single point of failure is: `Schema schema = schemaFactory.newSchema(SCHEMA_SOURCES);`. The `SCHEMA_SOURCES` is an array of `StreamSource` objects whose order depends on the a nondeterministic listing of files in a directory:
final File schemaDir = new File(JBOSS_DIST_DIR, SCHEMAS_LOCATION); final File[] xsds = schemaDir.listFiles(new SchemaFilter(EXCLUDED_SCHEMA_FILES.toArray(new String[0])));
The files are passed through a series of HashMaps which are read using nondeterministic `keySet()` and `values()` apis to eventually populate `SCHEMA_SOURCES`. However, the order of the `SCHEMA_SOURCES` matters since some schemas may be dependent on each other, (i.e. one schema importing another)
How To Fix
Enforce a deterministic non-conflicting order of `SCHEMA_SOURCES`.
For example, sort the files returned by `schemaDir.listFiles(new SchemaFilter(EXCLUDED_SCHEMA_FILES.toArray(new String[0])));`, convert `JBOSS_SCHEMAS_MAP` to a LinkedHashMap, and convert `CURRENT_JBOSS_SCHEMAS_MAP` to a LinkedHashMap.
Group of 47 Flaky Test Cases
wildfly-preview-dist
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneActiveMQEmbedded
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS
wildfly-dist
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2FullHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMicroProfileHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2HA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMicroProfile
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipFullHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS
wildfly-ee-dist
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2FullHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2HA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipFullHA
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
- org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS