Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-18823

Fix Flaky AbstractValidationUnitTest

XMLWordPrintable

      Fix Flaky AbstractValidationUnitTest

      Using the flaky test discovery tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] there are ~137 flaky tests in Wildfly.

      This JIRA issue aims to fix a subset of ~47 flaky tests in the dist, ee-dist, and preview-dist modules, since there is a common source of flakiness.

      To replicate the flaky tests errors:

      # 1. Setup NonDex
      git clone https://github.com/TestingResearchIllinois/NonDex.git
      cd NonDex
      mvn clean install -DskipTests
      
      # 2. Run NonDex in Wildfly on the target maven modules
      mvn clean -am -pl dist edu.illinois:nondex-maven-plugin:2.1.7-SNAPSHOT:nondex -Dtest=org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase -DnondexRuns=10 -DfailIfNoTests=false
      mvn clean -am -pl ee-dist edu.illinois:nondex-maven-plugin:2.1.7-SNAPSHOT:nondex -Dtest=org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase -DnondexRuns=10 -DfailIfNoTests=false
      mvn clean -am -pl preview/dist edu.illinois:nondex-maven-plugin:2.1.7-SNAPSHOT:nondex -Dtest=org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase -DnondexRuns=10 -DfailIfNoTests=false
      

      NonDex will run each test case multiple times. Some runs NonDex shuffles the the output of non-deterministic Java APIs (i.e. HashMap keySet()), while in other runs NonDex will not shuffle non-deterministic APIs. NonDex will provide a summary of all tests, showing tests that failed due to shuffling (flaky) and how those test pass without shuffling. For example, in the ee-dist you will see output similar to this:

      INFO: Surefire failed when running tests for hBG69zCLvYHCaIKs0AasjTFCdGbX6WTKb05H97ZNIis=
      [INFO] NonDex SUMMARY:
      [INFO] *********
      [INFO] mvn nondex:nondex  -DnondexFilter='.*' -DnondexMode=FULL -DnondexSeed=933178 -DnondexStart=0 -DnondexEnd=9223372036854775807 -DnondexPrintstack=false -DnondexDir="/root/wildfly/ee-dist/.nondex" -DnondexJarDir="/root/wildfly/ee-dist/.nondex" -DnondexExecid=5RBCJBZsiLqnSBfjJ9jn2YW0lSnqn0ab43cDSkQNTg= -DnondexLogging=CONFIG
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2FullHA
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipHA
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2HA
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipFullHA
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
      [WARNING] org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS

      Flakiness Cause

      The group of 47 tests targeted in this Jira (see list of tests at the bottom) flaky tests in (dist, ee-dist, and preview-dist modules) rely on a shared `parseXml` function in `AbstractValidationUnitTest`:

      protected void parseXml(String xmlName) throws SAXException, IOException {
        final File xmlFile = getXmlFile(xmlName);     
        SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);     
        schemaFactory.setErrorHandler(new ErrorHandlerImpl(xmlFile.toPath()));     
        schemaFactory.setResourceResolver(DEFAULT_RESOURCE_RESOLVER);     
        Schema schema = schemaFactory.newSchema(SCHEMA_SOURCES);     
        Validator validator = schema.newValidator();     
        validator.setErrorHandler(new ErrorHandlerImpl(xmlFile.toPath()));
        validator.setFeature("http://apache.org/xml/features/validation/schema", true);
        validator.setResourceResolver(DEFAULT_RESOURCE_RESOLVER);
        validator.validate(new StreamSource(xmlFile));  //noinspection ResultOfMethodCallIgnored     
        xmlFile.delete(); 
      }
      

      During NonDex runs, the single point of failure is: `Schema schema = schemaFactory.newSchema(SCHEMA_SOURCES);`. The `SCHEMA_SOURCES` is an array of `StreamSource` objects whose order depends on the a nondeterministic listing of files in a directory:

      final File schemaDir = new File(JBOSS_DIST_DIR, SCHEMAS_LOCATION);
      final File[] xsds = schemaDir.listFiles(new SchemaFilter(EXCLUDED_SCHEMA_FILES.toArray(new String[0])));

      The files are passed through a series of HashMaps which are read using nondeterministic `keySet()` and `values()` apis to eventually populate `SCHEMA_SOURCES`. However, the order of the `SCHEMA_SOURCES` matters since some schemas may be dependent on each other, (i.e. one schema importing another)

      How To Fix

      Enforce a deterministic non-conflicting order of `SCHEMA_SOURCES`. 

      For example, sort the files returned by `schemaDir.listFiles(new SchemaFilter(EXCLUDED_SCHEMA_FILES.toArray(new String[0])));`, convert `JBOSS_SCHEMAS_MAP` to a LinkedHashMap, and convert `CURRENT_JBOSS_SCHEMAS_MAP` to a LinkedHashMap.

      Group of 47 Flaky Test Cases

      wildfly-preview-dist

      1. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
      2. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
      3. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
      4. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
      5. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
      6. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
      7. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
      8. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
      9. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneActiveMQEmbedded
      10. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
      11. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
      12. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
      13. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS

      wildfly-dist

      1. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2FullHA
      2. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
      3. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
      4. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
      5. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
      6. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
      7. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
      8. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
      9. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipHA
      10. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMicroProfileHA
      11. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
      12. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2HA
      13. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMicroProfile
      14. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipFullHA
      15. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
      16. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
      17. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
      18. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS

      wildfly-ee-dist

      1. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2FullHA
      2. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostSecondary
      3. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandalone
      4. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHost
      5. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneHA
      6. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneJTS
      7. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneRTS
      8. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneXTS
      9. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipHA
      10. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneMinimalistic
      11. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneEC2HA
      12. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGossipFullHA
      13. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testHostPrimary
      14. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneFull
      15. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testDomain
      16. org.wildfly.dist.subsystem.xml.StandardConfigsXMLValidationUnitTestCase#testStandaloneGenericJMS

       

              abadia2@illinois.edu Alex Badia (Inactive)
              abadia2@illinois.edu Alex Badia (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: