Step 1 of 4: Choose Issues

Cancel

Key Summary Description
WFLY-648

WFLY-576 TS: Make Surefire test ordering configurable.

WFLY-647

WFLY-576 TS: API changes report - Animal sniffer maven plugin

Add support for such report.

See: http://mojo.codehaus.org/animal-sniffer-maven-plugin/

mvn animal-sniffer:build -DincludeJavaHome=false -DskipIfNoJavaHome=true
WFLY-646

WFLY-576 TS: Dynamic way to get ports of various services.

private int getRecoveryManagerPort() throws IOException {
    /*    /socket-binding-group=standard-sockets/socket-binding=txn-recovery-environment:read-attribute(name=port)  */
    final ModelNode address = new ModelNode();
    address.add("socket-binding-group", "standard-sockets");
    address.add("socket-binding", "txn-recovery-environment");
    final ModelNode operation = new ModelNode();
    operation.get(OP).set(READ_ATTRIBUTE_OPERATION);
    operation.get(OP_ADDR).set(address);
    operation.get("name").set("port");
    try {
        return executeOperation(operation).asInt();
    } catch (MgmtOperationException ignored) { }
    return -1;
}
WFLY-645

WFLY-576 Implement fallback mechanism for AS7 properties in pom.xml

There are several places where is used fallback mechanism when some content from property is taken like:

static final String someIPproperty = System.getProperty("<some> key>", "localhost");

The final decision based on (linked) AS7 devel discussion is leaving this approach and set-up this things in pom.xml's only.

After this task will be done, old approach will be leaved at all and every new patch will be forced to not use/include a such fallback mechanism only.

WFLY-644

WFLY-576 TS: Allow XSLT to fit any namespace versions.

Namespaces versions change during development, which leads to upstream testsuite versus EAP releases mismatch, which is undesirable as the testsuite sometimes contains more up-to-date tests.

It would be good to have XLST matching namespaces but except version.

WFLY-643

WFLY-576 TS: Implement replacing of ...=localhost in .properties resources

There are 3 occurences of $SUBJ, see below.

Since they are in src/, it needs to be copied elsewhere and added to classpath from there.

$ git grep -A1 =localhost
testsuite/integration/basic/src/test/resources/jboss-ejb-client.properties:remote.connection.default.host=localhost
testsuite/integration/basic/src/test/resources/jboss-ejb-client.properties-remote.connection.default.port=4447
--
testsuite/integration/clust/src/test/resources/cluster/ejb3/stateful/failover/sfsb-failover-jboss-ejb-client.properties:remote.connection.default.host=localhost
testsuite/integration/clust/src/test/resources/cluster/ejb3/stateful/failover/sfsb-failover-jboss-ejb-client.properties-remote.connection.default.port=4447
--
testsuite/integration/manualmode/src/test/resources/jboss-ejb-client.properties:remote.connection.default.host=localhost
testsuite/integration/manualmode/src/test/resources/jboss-ejb-client.properties-remote.connection.default.port=4447
WFLY-642

WFLY-576 TS: -Dtest=*<something>* only matches one test per module.

It seems like some regression.

When I've tried run like:

mvn3 clean install -Dts.basic '-Dtest=*.jaxrs.*'

I got this result:

[pjanouse@pjanouse testsuite]$ mvn3 clean install -Dts.basic '-Dtest=*.jaxrs.*'
Running org.jboss.as.test.smoke.jaxrs.JaxrsTestCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.689 sec
...

------ basic module:
Running org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.099 sec
...

So you can see - Integration basic is invoked, but only TestCases org.jboss.as.test.smoke.jaxrs.JaxrsTestCase and org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCas were run although there are plenty of other TestCases fulfilled mask *.jaxrs.* - many of them are localted in testsuite/integration/basic/src/test/java/org/jboss/as/test/integration/jaxrs/ + sub=dirs.

WFLY-641

WFLY-576 TS: Allow debugging client with -DdebugClient (only AS is possible now)

WFLY-640

WFLY-576 TS: Some tests missing from <includes> and are not run.

WFLY-639

WFLY-576 TS: Script to check missed tests.

WFLY-638

WFLY-576 TS: Add $managementAddress and $managementPort to all arquillian.xml's.

<property name="managementAddress">${node0:127.0.0.1}</property>
<property name="managementPort">${as.managementPort:9999}</property>

See https://docs.jboss.org/author/display/ARQ/JBoss+AS+7.0+-+Managed

WFLY-637

WFLY-576 TS: Pass node0, node1 to <ant ...> since inheritAll="true" doesn't work.

WFLY-636

WFLY-576 TS: parametrize surefire test.redirectTestOutputToFile property

WFLY-635

WFLY-576 TS: Fix XSLT transformations (namespaces problem)

Applying the standard XSLT copy template,

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:ds="urn:jboss:domain:datasources:1.0"
                xmlns="urn:jboss:domain:1.1"
                version="1.0">

    <xsl:template match="@*|node()">
        <xsl:copy>
            <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
    </xsl:template>

to the JBoss AS 7 standalone.xml, leads to loss of parameters:

<?xml version="1.0" encoding="UTF-8"?><server xmlns="urn:jboss:domain:1.1">
    <extensions>
        <extension/>
        <extension/>

instead of

<?xml version="1.0" encoding="UTF-8"?><server xmlns="urn:jboss:domain:1.1">
    <extensions>
        <extension module="org.jboss.as.clustering.infinispan"/>
        <extension module="org.jboss.as.configadmin"/>

It's something namespace-related because ds:... attribs ARE copied:

        <subsystem>
            <datasources>
                <ds:datasource xmlns:ds="urn:jboss:domain:1.1" jndi-name="java:jboss/datasources/ExampleDS"
                           pool-name="MSQL"
                           enabled="true"
                           jta="true"
                           use-java-context="true">
WFLY-634

WFLY-576 TS: Pass the path to docs/examples to the tests.

In AS7, the path to the config file, passed in a system property, is only allowed to be a relative path.

In the testsuite, we would make use of absolute paths.
The reason is that we reuse the configs from docs/examples, and they change quite often, so having them in resources would be hardly maintainable.
However, docs/examples is not guaranteed to be always in the same place: E.g. for EAP, or for RPM-based installation of EAP it might end up in /var/docs or such.

I would like to pass some property from maven to the tests, but that results in absolute path.

WFLY-633

WFLY-576 TS: Make container JVM --enable-assertions configurable.

Arquillian's AS7 container has a property:

<property name="enableAssertions">false</property>

And it's true by default.
Let's make it configurable.

<jvm.args.enableAssertions></jvm.args.enableAssertions>
WFLY-632

WFLY-576 TS: Check how JDBC drivers are copied

I created some initial impl, but that copies drivers to deployments/.

The preferred way for TS harness is to create a module, since the other way is easy to do manually in case if interrest.

What's specially needed to check is that the driver and DB config ends up in all instances of multi-node modules (cluster, multinode, etc.).

https://community.jboss.org/wiki/DataSourceConfigurationInAS7

WFLY-631

WFLY-576 TS: Make commons-lang3, commons-io and commons-codec available in tests.

To relieve people from writing methods like toHex(), defaultIfNull() and escapeHtml(),
I'd like to add commons-lang3, commons-io and commons-codec.
These are de-facto standard, well tested, mostly thread-safe.

For tests running in server, these would still need to be packaged into the deployment.

WFLY-630

WFLY-576 TS: Check endorsed jboss-annotations-api.

(22.12.2011 04:09:13) ozizka: Speaking of hacks, how about jboss-annotations-api_1.1_spec being set as endorsed during testsuite compilation
(04:09:25) ozizka: The comment says "Big complex hack just to get @Resource(lookup="foo")."
(04:09:34) ozizka: Is that stil needed?
(04:37:14) bstansberry: I don't know. I never really knew what that was all about
(04:37:39) bstansberry: i doubt it's needed, just because it sounds so weird and dates so far back
(04:38:12) bstansberry: oh, i can guess what it's about
(04:38:50) bstansberry: wacky Sun/Oracle have the same class in both SE and EE but with extra members in EE
(04:39:20) bstansberry: "lookup" isn't in @Resource in SE

WFLY-629

WFLY-576 TS: Find a way to retain same value of property with ${basedir} in submodules.

John Casey:

Can you explain exactly how the ${basedir} property will be used? This matters, because expression resolution happens two different ways, once for POM interpolation and again for plugin parameter injection. For more information on the differences, you can play with an expression plugin I built: http://github.com/jdcasey/expression-maven-plugin (see the README for information on using it).

We need to refer to ${basedir} for two purposes:

1) To be able to pass important directories to the tests, for example:

  • AS project root dir
  • Testsuite root dir (to access global resources)
  • Integration testsuite root dir (to access int. ts. resources)
    In this case, we need to be able to define the value in parent pom and retain it for sub-modules.

2) For various operations (plugin executions) which are defined in parent pom, but inherited in sub-modules.
In this case, we really need to have it done using relative paths, e.g. ${basedir}/src/config/arquillian .

In both cases, ${basedir} is used for plugins configuration.

John Casey:

I've written a plugin called org.commonjava.maven.plugins:directory-maven-plugin that has goals:

  • execution-root = Resolves to the directory where maven was invoked
  • directory-of = uses a <project/> configuration with groupId and artifactId to find another project in the current session, and returns its basedir
  • highest-basedir = traverses the parent hierarchies of all current projects, up until it finds a parent that has been resolved instead of loaded locally. Then, it sorts these basedir paths and returns the one closest to the filesystem root. If the top two hits (highest results) are not nested within one another, it will fail.

I'd expect you to use something like these to pin down a particular project basedir reference, and inject it consistently into each project in the session. It may be that you actually need something like a combination of the last two, to reference a particular parent project that was loaded from disk (but may not be in the current session per se). If so, that's another easy one to write.

The plugin is here:

http://github.com/jdcasey/directory-maven-plugin

You can take a look at the README.md for more information.

WFLY-628

WFLY-576 TS: Way to build with a clean local repo.

This can be done either by using an empty local repo, setting it in settings.xml,

<settings>
   <localRepository>mvn-repo</localRepository>
   ...

or by using the dependency:purge-local-repository goal in a profile:

        <profile>
            <id>cleanRepo.profile</id>
            <activation><property><name>cleanRepo</name></property></activation>
            <build>
                <plugins>
                    <plugin>
                        <artifactId>maven-dependency-plugin</artifactId>
                        <goals><goal>purge-local-repository</goal></goals>
                    </plugin>

                    <plugin>
                        <groupId>org.codehaus.mojo</groupId>
                        <artifactId>build-helper-maven-plugin</artifactId>
                        <version>1.7</version>
                        <executions> <execution>
                           <id>remove-old-installers</id>
                           <goals><goal>remove-project-artifact</goal></goals>
                           <configuration> <removeAll>true</removeAll> </configuration>
                        </execution> </executions>
                    </plugin>

                </plugins>
            </build>
        </profile>

However first can only be done in settings.xml, and not in a profile, and the later can be done with `mvn dependency:purge-local-repository build-helper:remove-project-artifact -Dbuildhelper.removeAll`.
So it's not to be done in the testsuite itself. Resolving.

WFLY-627

WFLY-576 TS: -Djboss.dist=... has no effect, /build/target/jboss-as* is always used

Please check the usage of -Djboss.dist parameter provided from the command line. It seems the testsuite does not take it and still uses jboss.dist as defined in testsuite/pom.xml#<properties>. Looks like maven properties are a bit different than system properties, read for example the following:

http://maven.40175.n5.nabble.com/Overriding-properties-from-command-line-arguments-td3345432.html

Thanks.

WFLY-626

WFLY-576 TS: Database switch (in datasources)

This is how Hibernate did it:
https://github.com/hibernate/hibernate-core/blob/3.6/hibernate-parent/pom.xml#L575

WFLY-625

WFLY-576 TS: Multiple Surefire executions - FAILURE in one supresses successive.

Jason wrote that we can live with this for CR1; However having it fixed for EAP is important.

Work in progress at http://jira.codehaus.org/browse/SUREFIRE-803 .
John Casey keeps an eye on this.

WFLY-624

WFLY-576 TS: Option to fail if AS is running (~ a port is open)

http://ant.apache.org/manual/Tasks/conditions.html

<target name="check-port" description="Checks whether AS is running.">
    <condition property="jbossas.running">
        <socket server="${jbossas.host}" port="${jbossas.port}"/> 
    </condition>
</target>
WFLY-623

WFLY-576 TS: Configurable timeouts

Testsuite run can take different amount of time on various HW (or virtuals).
All timeouts must be configurable, in the sense of being possible to prolong.

We should also divide them into categories, depending on what is expected to take long (consider slow disks, slow networks, cloud environment)
Few ideas for categories:

  • Filesystem I/O
  • Processor
  • Network I/O
  • Memory I/O
  • Database operations
    For each category, there would be a ratio, 1.0 by default, which would multiply the timeout.

So the timeout would look like:

something.setTimeout( 1000 * Timeouts.getFSIORatio() );

Other possibility is to use @Annotations to mark tests belonging to certain group, like @LongRunning or such.

WFLY-622

WFLY-576 TS: Option to run under security manager.

-Dsecurity.manager

and

-Dsecurity.policy

There are some tests failing under globally set SM because they use their own.
These need to be exluded from running when security manager is in use.

WFLY-621

WFLY-576 TS: Implement self-test.

Implement a script which would verify the health of the testsuite - i.e.:

  • Run scripts with various params & check the results
  • Run mvn with various params & check the results

By results, we mean:

  • Same set of tests run
  • Same results as before
  • AS is configured the same way (if not intentionally changed)
WFLY-620

WFLY-576 TS: Check tests for hard-coded strings like URLs etc. - candidates for parametrization.

WFLY-619

WFLY-576 TS: Update windows scripts (build.bat and integration-tests.bat)

WFLY-618

WFLY-576 TS: (Re-)Create test group for web profile.

(20:46:10) Nihility: OndrejZizka: got a few seconds to talk testsuite
(20:46:10) smcgowan: baileyje: http://pastebin.test.redhat.com/67894
(20:46:20) OndrejZizka: Nihility: Sure
(20:46:23) jharting opustil místnost (quit: Quit: Leaving.).
(20:46:54) Nihility: OndrejZizka: ok so for the beta release i need to resurect the old default profile "the web profile" and make the existing one the "full" profile
(20:47:17) Nihility: OndrejZizka: i had some long conversations with various folks about how to do that without screwing up the progress that has already been made with the testsuite
(20:48:06) Nihility: OndrejZizka: it seems that the best solution was to have a property which tells the testsuite to run against the "web" profile, using the different config, and also disabling tests using a pattern match
(20:48:27) Nihility: OndrejZizka: so basically we would have to manually kick off a web run, or have a hudson run or something
(20:49:04) Nihility: OndrejZizka: did you have any thoughts on that? I had originally hoped we would use different configs for different tests, but that seems to be a nightmare in our current maven configuration
(20:49:28) Nihility: OndrejZizka: or any other ideas would be welcome
(20:49:32) OndrejZizka: Nihility: Have you seen my last post in jboss-as7-dev?
(20:49:59) OndrejZizka: Nihility: I have modified the testsuite a bit, so I believe this task is easier to do with it now too,
(20:50:07) OndrejZizka: I only need someone to fix OSGi tests
(20:50:41) Nihility: OndrejZizka: all i have is "printing project number"
(20:50:42) OndrejZizka: After that, web profile vs. full profile can be simply a profile activating different modules
(20:51:35) OndrejZizka: Nihility: Forwarded
(20:51:46) Nihility: OndrejZizka: i mean do you think the different config per test module is reasonable?
(20:51:59) Nihility: OndrejZizka: it seemed like we would have to do a BIG resturcturing for that
(20:52:23) OndrejZizka: Nihility: Depends on what are the differences, and for which modules,
(20:52:45) OndrejZizka: Inside the integration module, the differences are not significant,
(20:53:01) OndrejZizka: the modules are used to separate tests virtually
(20:53:07) Nihility: OndrejZizka: yeah thats the sisue the integration module mixes full and non-full
(20:53:59) OndrejZizka: Nihility: Could you send me a list of which modules are non-full? And also, the config for web AS profile
(20:54:15) OndrejZizka: or a list of differences
(20:54:40) Nihility: OndrejZizka: im working under the assumption we have to run twice (i figured there was no time to restructure to that level)
(20:55:23) Nihility: OndrejZizka: yeah sure, its basically standalone but without jacorb, hornetq, and jaxr
(20:55:34) Nihility: OndrejZizka: and webservices
(20:55:55) OndrejZizka: Twice? There are two executions of surefire, which is because some tests test some persistent stuff
(20:56:43) Nihility: OndrejZizka: no i dont mean automatically run twice, i mean the user has to say something like ./build.sh -Dweb-profile or something
(20:57:01) Nihility: OndrejZizka: to be honest i dont like that solution as much as doing a config per module
(20:57:28) OndrejZizka: Nihility: I think this could be solved too
(20:57:31) Nihility: OndrejZizka: but the config per module sounds like there is just not enough time, we need to get teh beta out this week
(20:58:20) smcgowan opustil místnost (quit: Quit: Leaving).
(20:59:02) OndrejZizka: Nihility: Check this
(20:59:03) OndrejZizka: https://github.com/OndraZizka/jboss-as/tree/TS-modules-tmp/testsuite/integration
(20:59:29) OndrejZizka: It's basically done, only OSGi tests fail because there's some assumption about relative paths
(20:59:38) Nihility: ah intersting
(20:59:47) Nihility: so you use a maven module to activate profiles?
(20:59:51) OndrejZizka: If someone fixes that, I think we could use that for the requirements you stated
nickarls Nihility
(21:00:13) OndrejZizka: Nihility: and vice versa - this way is the most flexible
(21:00:47) OndrejZizka: Using profiles, I activate sets of modules, which define -DallTests, or a specific group;
(21:01:17) OndrejZizka: then, in these modules, there are profiles which can be disabled, and also "global" profiles which change database config etc.
(21:02:35) Nihility: ah ok so in theory
(21:02:44) Nihility: we would have a group of "full" tests
(21:02:47) Nihility: or something
(21:03:48) OndrejZizka: Nihility: yes
(21:04:16) OndrejZizka: Either as a module, or as a profile.
(21:04:24) OndrejZizka: in the integration/basic module
(21:04:28) OndrejZizka: I'd prefer the second,
(21:04:43) OndrejZizka: but if it turns out there's some problem with maven, we can fall back to module-based selection
(21:04:52) OndrejZizka: and they can run both in one run
(21:05:35) OndrejZizka: I mean, running two executions of same tests using profiles is sometimes troublesome
(21:05:52) Nihility: yeah and its time consuming
nickarls Nihility
(21:08:52) OndrejZizka: Nihility: So, do we need to run the web tests twice?
(21:09:01) OndrejZizka: in this full/non-full scenario
(21:09:16) OndrejZizka: Or in full, only the complement would be run?
(21:09:42) OndrejZizka: * complementary tests
(21:11:12) ctomc ctomc@upc.si.94.140.92.246.dc.cable.static.telemach.net vstoupil do místnosti.
(21:11:19) Nihility: OndrejZizka: well IMO we only need to run the "full-only" tests against standalone-full.xml
(21:12:25) OndrejZizka: Nihility: Ok, that should be okay. So I'll prepare a PoC and ping you, okay?
(21:12:33) OndrejZizka: Which TZ is thomas.diesler@jboss.com in?
(21:13:03) Nihility: OndrejZizka: ok, hes in munich which i think is GMT+1
(21:13:33) Nihility: OndrejZizka: we should chat with stuartdouglas about this as well
(21:16:11) maxandersen Adium@redhat/jboss/maxandersen vstoupil do místnosti.
(21:16:11) režim (+v maxandersen) od ChanServ
(21:16:18) Nihility: OndrejZizka: so according to that email the issue he brings up is that we have duplicate sources?
(21:17:50) stuartdouglas: I really don't like the 1 big ball of sources thing
(21:18:14) stuartdouglas: I think it overcomplicates things, requires additional configuration, and may cause dependency and other problems down the road
stalep stuartdouglas
nickarls Nihility
(21:22:56) OndrejZizka: stuartdouglas, Nihility, when such problems arise, there's no problem in separating them
(21:23:31) OndrejZizka: cd testsuite/integration; mv src/test/java/foo/bar module/src/test/java/foo/bar
(21:23:38) OndrejZizka:
(21:23:57) OndrejZizka: Not counting resources
(21:24:19) OndrejZizka: But I hope that tests keep their resources well separated
(21:24:33) OndrejZizka: As per Jason's recent post to dev list
(21:26:40) darranl darranl@redhat/jboss/darranl vstoupil do místnosti.
(21:26:40) režim (+v darranl) od ChanServ
(21:27:47) OndrejZizka: stuartdouglas: Either way - pushing everything to a single module doesn't work, so the question is not whether to stay with current or split,
(21:28:08) OndrejZizka: but whether to split the modules and keeps tests together, or split them "physically"
(21:28:27) OndrejZizka: I have a) done, stuart likes b), can be done too
(21:28:31) stuartdouglas: I think we should go with splitting them up into different modules, with the relevant source in each module
(21:28:38) stuartdouglas: it just seems like the simplest approach
(21:28:49) stuartdouglas: and it is also the 'standard' maven way
(21:29:12) stuartdouglas: From your point of view, what are the downsides to doing it this way?
(21:31:48) CheffPJ pmcdonou@219.sub-75-248-111.myvzw.com vstoupil do místnosti.
(21:32:26) Nihility: stuartdouglas: OndrejZizka assuming we did the split into smaller finer grained modules, would the activation thing still work (e.g. we create dummy modules that activate the other ones)?
(21:33:02) Nihility: stuartdouglas: OndrejZizka or would we just use the shell scripts?
navssurtani1 nickarls Nihility
nickarls Nihility
(21:33:53) OndrejZizka: Nihility: The separation needs was the original reason why I started this change
nickarls Nihility
(21:34:25) OndrejZizka: Nihility: So, the activation would be done by activating different profiles, which would include
(21:34:42) OndrejZizka: <modules> <module>...</module> <module>...</module> <module>...</module> ... </modules>

WFLY-617

WFLY-576 TS: Hitting max file limit with reporting (`mvn site`)

WFLY-616

WFLY-576 TS: Clustering tests - figure out how they will be run

The current single test is not multinode - when set up as multinode, it fails. So it runs as single node, even though being in package .cluster.

WFLY-615

WFLY-576 TS: Move *-arquillian.xml from testsuite.integration/src/test/resources/** to some better place.

WFLY-614

WFLY-576 TS: Add properties "jbossas.ts.integ.dir", "jbossas.ts.dir", "jbossas.project.dir" and use them in test.

Instead of implying that the test runs in certain subdirectory of the testsuite, define use these System variables passed to Surefire:

"jbossas.ts.integ.dir", "jbossas.ts.dir", "jbossas.project.dir"

WFLY-613

WFLY-576 Propagate property node0 to system properties

Propagate property node0 (and other ones) to system properties for test being able to take IP address where the AS runs.

WFLY-612

WFLY-576 TS: Print help banner on `mvn install` without params.

Jaikiran:

{qoute}
Would it be possible to somehow print out all the available options and
params that need to used, when you run mvn clean install from the
testsuite modules? Something like a help?{qoute}

Not sure if it will be possible to check the condition "There are no params", but we can print it every time as well.

WFLY-611

WFLY-576 TS: Test output is not stored in ...-output.txt

./integration-tests.sh clean install -Dtest=org.jboss.as.test.integration.ejb.async.* -Dintegration.module -Dbasic.integration.tests

WFLY-610

WFLY-576 TS: Generate server config for Arq instances (standalone-*.xml) using XSLT instead of keeping them in resources.

WFLY-609

WFLY-576 TS: Move smoke and integration tests before the domain stuff.

{query}
(21:31:55) bstansberry: OndrejZizka1: I'd like to re-order the testsuite execution a bit but am a bit lost on what's driving the current order
(21:33:11) bstansberry: OndrejZizka1: it would be better if smoke and integration tests ran before the domain stuff, as the domain stuff is slow and just delays an overall failure if one of the more numerous smoke/integration tests fails{query}
WFLY-608

WFLY-576 TS: `mvn install` and `build.sh` must run smoke tests by default.

WFLY-607

WFLY-576 TS: Define testsuite acceptance test (a testsuite for the testsuite)

Define a procedure to perform before committing something to the testsuite.

Create scripts to help determining whether given testsuite is okay to be committed.
E.g.

IN='-DallTests -Dinteg-tests -Dcluster-tests -Dbasic-tests -Dbenchmark-tests -Dsmoke-tests -Dstress-tests -Ddomain-tests -Dcompat-tests';
for GROUP in $(echo $IN | tr ";" "\n"); do
  ## Create results dir.
  RES_DIR=~/work/AS7/res/$GROUP
  mkdir -p $RES_DIR
  ## Run the testsuite.
  ./integration-tests.sh clean -DallTests;
  ./integration-tests.sh install $GROUP;
  ## Archive the reports.
  for REPORT in `find testsuite -name 'TEST-*.xml' -or -name '*TestCase.txt' -or -name '*TestCase-output.txt'`; do
    SUB_DIR=`dirname $REPORT | sed 's#testsuite/##' | sed 's#/target/surefire-reports##'`
    mkdir -p $RES_DIR/$SUB_DIR
    cp $REPORT $RES_DIR/$SUB_DIR
  done
  ## Run the testsuite for 2nd time to ensure that unclean run works too.
  ./integration-tests.sh install $GROUP;
done
## Create a JUnitDiff report.

Also, check what's being passed to `mvn` on various params combinations:

./build.sh -DallTests
./build.sh -DallTests -Dfoo
./build.sh install -DallTests
./build.sh clean -DallTests
./build.sh clean install -DallTests -Dfoo
./integration-tests.sh -DallTests;
./integration-tests.sh install -DallTests;
./integration-tests.sh clean -DallTests;
./integration-tests.sh clean install -DallTests -Dfoo;
./integration-tests.sh test -DallTests -Dfoo;
...
WFLY-606

WFLY-576 TS: On -Dtest=..., activate appropriate modules automatically (using package prefix)

...to relieve people from writing `-Dintegration.module` all the time.

WFLY-605

WFLY-576 TS: Unify activation properties' names (e.g. -DstressTests instead of -Dstress-test)

WFLY-604

WFLY-576 TS: IPv6 testing issues (tracking)

This includes:

  • Check AS management support for IPv6
  • Check Arquillian's support
  • Config param to scripts / maven property passed to arquillian
  • Replacing "http://localhost:..." in tests with a configurable value.

QA contact: Pavel Janousek

https://docspace.corp.redhat.com/docs/DOC-71739
https://docspace.corp.redhat.com/docs/DOC-47796
https://docspace.corp.redhat.com/docs/DOC-70336 - See 4. Future work ... more than 50% missing

Java IPv6 guide: http://docs.oracle.com/javase/6/docs/technotes/guides/net/ipv6_guide/index.html#details

WFLY-603

WFLY-576 TS: Fix -Ddebug (-Djpda)

ozizka-as7/testsuite/integration $ mvn install -Dclustering.integration.tests -Djpda

doesn't work.

WFLY-602

WFLY-576 TS: Move Arquillian configs from src/test/resources/X to some special dir.

Currently it's these:

src/test/resources/arquillian.xml
src/test/resources/clustering/clustering-arquillian.xml
src/test/resources/iiop/iiop-arquillian.xml
src/test/resources/smoke/smoke-arquillian.xml

WFLY-601

WFLY-576 TS: Logging issues (tracking)

Some tests print to System.out, and also ex.printStackTrace(). Turn it to proper logging. Also check logging config.

WFLY-600

WFLY-576 TS: Check tests workdirs

Some tests (also unit tests) have work dirs in ${basedir}, which leaves garbage after running.
That should change either to target/work or /tmp/... (consider what's better).

WFLY-599

WFLY-576 TS: Get new harness to master.

  • pom.xml's
  • scripts
  • common variables script
  • integration
    + XSLT's
  • arquillian.xml's
    + Ant scripts (src/test/scripts)
    + basic... (removed stuff for removed test)
    + removed compat... and timerservice... (melded)
WFLY-598

WFLY-576 Alternative JDKs for building and running - OpenJDK 6, 7, Sun JDK 7, IcedTea

James Perkins:

At one point there was a JIRA for OpenJDK 6, in fact I think a couple. The issue with OpenJDK 6 is in the build we use some JavaScript and OpenJDK 6 doesn't come with a JavaScript engine. I tried a couple things to get Rhino working with it as that's what the Sun JDK uses, but I think it needs to implement an SPI to get it to work. I didn't look into it much beyond that.

JDK 7 is a different issue. It's a bug in the annotation processing API which the JBoss Logging Tooling uses. I just refreshed my OpenJDK 7 update source and it looks like the bug is fixed in there. It's fixed in IcedTea as well. There could be the JavaScript issue here as well I can't actually remember, but I thought I had some successful builds with custom JDK's I compiled.

I did just try with IcedTea 7 and got some other errors. I'm building the latest upstream of OpenJDK 7 now and we'll see if it works. I'll let you know either way.

Let me know if you have any more questions on this. I did dig into a while ago a little.

WFLY-597

WFLY-576 TS: Put all management tests into one dir (org.jboss.as.test.integration.mgmt)

integration/org.jboss.as.test.integration.management

and further separated by subsystems/interfaces:
org.jboss.as.test.integration.management.cli
org.jboss.as.test.integration.management.http
org.jboss.as.test.integration.management.api
org.jboss.as.test.integration.management.api.messaging

Thanks Dominik for pointing out.

WFLY-596

WFLY-576 TS: Pick the most acceptable test packages organization

1) Split classes to src/main and src/test ?
This seems to be decided before - integration tests suite has all classes in src/test.

(02:35:15) barreiro: When I start I done it that way so that it clear to me what was the test code, the test subject and the resources — I knew that probably I would have to change the location later.

2) Low-level Test packages micro

(02:37:51) barreiro: should I create one sub package for each test — like /web/xxxTestCase — or stick with the old schema of having web/tests and web/servlets, ... etc.
(02:38:01) ozizka: Not sure if democracy is the principle to follow here, but I'd like devs to agree upon these things. TBD next week
(02:38:41) ozizka: Depends if you reuse them.
(02:39:24) ozizka: But generally, I'd prefer the first approach -
(02:39:53) ozizka: with reused things being placed one level higher, or next to
(02:40:16) ozizka: The structure should reflect test's structure
(02:40:24) barreiro: IMO, democracy is not the way to go in a OSS project ... I'll do it the way you ask me to
(02:40:34) ozizka: Ok
(02:40:57) ozizka: But anyway - what's your preference?
(02:41:07) ozizka: Or what would suite better for your case
(02:42:42) barreiro: the ultimate goal is that someone from outside can easily understand what's going on. Like me, I just landed in this project.
(02:43:58) barreiro: surely that it should reflect the test's structure. but I would rather have a common structure in most modules ...
(02:45:30) barreiro: like, at some level there is a clear list of the tests in the package that is not mixed with other stuff — be it a list of packages or a test package with all the test classes
(02:45:43) ozizka: Well, 1st option is current practice, I'd stick with that.
(02:46:47) barreiro: and from there should be clear (as possible) what resources are used.
(02:47:34) ozizka: I was thinking about using a mechanism used in Wicket framework, if you know that,
(02:47:57) ozizka: which is, having a mechanism to load resources from places
(02:48:04) barreiro: After a while I start to understand how AS6 is structured ... the problem is that there is too much voodoo in the back that defeats it's purpose.
(02:48:18) ozizka: where the test class is loaded, or it's superclass, etc.
(02:48:45) ozizka: Eventually it could go by packages up. But perhaps too magical for a testsuite
(02:49:10) ozizka: Right. That's the voodoo.
(02:56:27) barreiro: yep. Shrinkwrap and arquilling are great for that. The magic is greatly reduced, without adding much complexity.
(02:58:10) ozizka: One other argument to have the package-per-test separation is that if we decided the other way, moving the files would be easier

WFLY-595

WFLY-576 TS: Review smoke tests

Currently, smoke tests are in package .smoke.
I'd rather see them in appropriate package,
and differentiated by some grouping mechanism. See AS7-2086.

Also:
(02:07:46) stuartdouglas: but really, the smoke tests need review
(02:08:03) stuartdouglas: to make sure they cover lots of different stuff in a general way
(02:08:12) stuartdouglas: and also I would like to remove their dependency on the demo's
(02:08:22) stuartdouglas: my thoughts where to just have them named SmokeTestCase

Also, let's remove the "embedded" from package name.

WFLY-594

WFLY-576 TS: Add a module for shared util classes.

To prevent people coding their own impl of what was coded twenty-times in other packages of TS.

There's already a package: `org.jboss.as.test.integration.common`

It could be a module to keep it's dependencies separated.

Perhaps also Apache Commons Util should be added as a dependency.

WFLY-593

WFLY-576 TS: Tests grouping

Decide which mechanism to use to assign the tests into one or multiple groups.

Posibilities:

1)

@Test
@Category(IntegrationTests.class)

http://weblogs.java.net/blog/johnsmart/archive/2010/04/25/grouping-tests-using-junit-categories-0
Also, Arquillian itself supports grouping: https://github.com/weld/core/blob/master/tests-arquillian/src/test/java/org/jboss/weld/tests/Categories.java

2)

@Ignore

3) Maven Surefire <includes> and <excludes>

WFLY-592

WFLY-576 TS: Consolidate documentation of the testsuite.

1) Update README.txt
2) Update https://docs.jboss.org/author/display/AS71/AS+7+Integration+Test+Suite+User+Guide#
3) Finish http://community.jboss.org/wiki/AS7TestsuiteLifecycle .
4) Maybe update http://community.jboss.org/wiki/JBossAS7UserGuide

WFLY-591

WFLY-576 TS: Discuss resources location - next to .java files?

Having resources in completely different dir tree makes sense for business apps where they are often maintained by different teams (css, images, ...).
For a testsuite, where all resources are maintained by devs, usually even the same as sources which use them, there's no real reason to keep them separated.

Many devs prefer having resources as close to the sources as possible, which sometime leads to putting text content to .java sources - especially with Arquillian. See ejb/descriptor/AS835TestCase.java at https://github.com/stuartwdouglas/jboss-as/compare/master...tests#diff-20 .

Maven allows to keep sources and resources in the same dir tree.
TODO: Discuss with devs how they'd like this.

WFLY-590

WFLY-576 TS: Discuss package names - org.jboss.as.testsuite.integration.xyz vs. org.jboss.as.test.xyz

org.jboss.as.testsuite.integration.xyz:
"integration" gives a hint of test's affiliation right from the package name.

org.jboss.as.test.xyz:
Shorter.
Less subdir levels.

Discuss (with QA?) whether they really care.

WFLY-589

WFLY-576 TS: Check clustering and compat and spec tests

Stuart Douglas:

1) Clustering tests do not run, and as they require a different arquillian adaptor they must not be in the same maven module as other integration tests.

2) Compat tests do not run correctly. Even though they have a different hibernate version specified in the profile, when attempting to run alongside other tests the hibernate version will be overriden. Also as this modifies a build time dependency test classes need to be re-compiled each run, to make sure they are built against the correct hibernate version.

I'll merge our changes into Stuart's branch since moving poms and scripts is easier than moving Clustering and Compat dirs back to their original location.

WFLY-588

WFLY-576 TS: Prepare for merging into master.

Current authoritative branch is at https://github.com/OndraZizka/jboss-as/tree/testsuite_reorg
I keep rebasing this daily to be ready to merge into master.
When all the issues under AS7-2007 are resolved, I'll create a pull request at https://github.com/jbossas/jboss-as .

WFLY-587

WFLY-576 TS: #!/bin/sh prevents using bash construct

$ ./integration-tests.sh clean integration-test -Dtest=*SessionFactoryTestCase*
./integration-tests.sh: 238: [[: not found
/home/ondra/work/AS7/jboss-as/tools/maven/bin/mvn -s ../tools/maven/conf/settings.xml  clean integration-test -Dtest=*SessionFactoryTestCase*
WFLY-586

WFLY-576 build.sh clean install -DallTests doesn't work after scripts were split

WFLY-585

WFLY-576 TS: Coverage reports

Cobertura coverage jobs in Hudson (unit-test only, see 1st comment):

We can also use Emma:

Or this? http://www.eclemma.org/jacoco/index.html

TCK coverage reports: http://www.qa.jboss.com/~smcgowan/CDI-TCK-Assertions/jsr299-tck-impl-coverage-cdi.html

WFLY-584

WFLY-576 Maven plugin to process properties

WFLY-583

WFLY-576 TS: Some tests are still in org.jboss.as.testsuite.integration package

org.jboss.as.testsuite.integration.weld.alternative.WeldAlternativeTestCase
org.jboss.as.testsuite.integration.websecurity.WebSecurityCERTTestCase
org.jboss.as.testsuite.integration.jaxrs.integration.ejb.JaxrsEjbInterceptorsTestCase
org.jboss.as.testsuite.integration.jpa.transaction.TransactionTestCase
org.jboss.as.testsuite.integration.osgi.webapp.WebAppNegativeTestCase
org.jboss.as.testsuite.integration.pojo.test.CollectionsBeansTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarJbossStructureDepedenciesTestCase
org.jboss.as.testsuite.integration.jaxrs.integration.ejb.JaxrsEjbInterceptorsEarTestCase
org.jboss.as.testsuite.integration.osgi.interceptor.LifecycleInterceptorTestCase
org.jboss.as.testsuite.integration.jaxrs.packaging.war.ApplicationPathIntegrationTestCase
org.jboss.as.testsuite.integration.injection.resource.persistenceunitref.PersistenceUnitRefTestCase
org.jboss.as.testsuite.integration.weldejb.removemethod.RemoveMethodExceptionTestCase
org.jboss.as.testsuite.integration.jpa.multipleinjections.MultiplePersistenceContextInjectionsTestCase
org.jboss.as.testsuite.integration.jaxrs.jsapi.JaxrsJSApiTestCase
org.jboss.as.testsuite.integration.weld.extensions.ExtensionsInEarDiscoveredTestCase
org.jboss.as.testsuite.integration.interceptors.exceptions.ExceptionsFromInterceptorsTestCase
org.jboss.as.testsuite.integration.jaxrs.packaging.war.NoApplicationIntegrationTestCase
org.jboss.as.testsuite.integration.ejb.packaging.war.namingcontext.WarEjbNamingContextTestCase
org.jboss.as.testsuite.integration.pojo.test.TypeBeansTestCase
org.jboss.as.testsuite.integration.jaxrs.packaging.ear.ApplicationPathOverrideIntegrationTestCase
org.jboss.as.testsuite.integration.jpa.defaultdatasource.DatasourceTestCase
org.jboss.as.testsuite.integration.ejb.injection.ejb.EjbInjectionSameEjbNameTestCase
org.jboss.as.testsuite.integration.ejb.remote.ejbnamespace.EjbNamespaceInvocationTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.WarCannotAccessOtherWarTestCase
org.jboss.as.testsuite.integration.deployment.classloading.war.WarNestedBundleTestCase
org.jboss.as.testsuite.integration.ejb.transaction.bmt.BeanManagedTransactionsTestCase
org.jboss.as.testsuite.integration.ejb.interceptor.defaultinterceptor.DefaultInterceptorsTestCase
org.jboss.as.testsuite.integration.osgi.xml.parser.DOMParserTestCase
org.jboss.as.testsuite.integration.jaxrs.atom.JaxrsAtomProviderTestCase
org.jboss.as.testsuite.integration.jpa.hibernate.SecondLevelCacheTestCase
org.jboss.as.testsuite.integration.ejb.stateless.systemexception.SystemExceptionTestCase
org.jboss.as.testsuite.integration.injection.resource.infinispan.InfinispanInjectionTestCase
org.jboss.as.testsuite.integration.osgi.jmx.BundleStateTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarClassPath4TestCase
org.jboss.as.testsuite.integration.deployment.classloading.transientdependencies.RarTransientDependenciesTestCase
org.jboss.as.testsuite.integration.jpa.hibernate.EntityTestCase
org.jboss.as.testsuite.integration.jpa.epcpropagation.contextduel.AddEPC2TxAfterPCTestCase
org.jboss.as.testsuite.integration.jaxb.unit.JAXBUsageTestCase
org.jboss.as.testsuite.integration.weldejb.inheritance.EjbInheritanceTestCase
org.jboss.as.testsuite.integration.osgi.xservice.ModuleAccessesBundleServiceTestCase
org.jboss.as.testsuite.integration.osgi.deployment.BundleDeploymentCaseTwoTestCase
org.jboss.as.testsuite.integration.datasourcedefinition.DataSourceDefinitionTestCase
org.jboss.as.testsuite.integration.deployment.classloading.transformer.WarJbossStructureClassFileTransformerTestCase
org.jboss.as.testsuite.integration.jaxrs.integration.cdi.CDIApplicationPathIntegrationTestCase
org.jboss.as.testsuite.integration.ejb.injection.compenvbindings.EjbJavaCompBindingTestCase
org.jboss.as.testsuite.integration.ejb.remote.classloading.CrossDeploymentRemoteInvocationTestCase
org.jboss.as.testsuite.integration.ejb.security.unit.EJBSecurityTestCase
org.jboss.as.testsuite.integration.weldejb.packaging.warlib.EjbInWarLibPackagingTestCase
org.jboss.as.testsuite.integration.ejb.singleton.concurrency.inheritance.SingletonConcurrencyInheritanceTestCase
org.jboss.as.testsuite.integration.osgi.deployment.BundleDeploymentCaseOneTestCase
org.jboss.as.testsuite.integration.osgi.ds.DeclarativeServicesTestCase
org.jboss.as.testsuite.integration.weldejb.multipleviews.NormalScopedEjbWithMultipleViewsTestCase
org.jboss.as.testsuite.integration.ejb.packaging.multimodule.MultiModuleLifecycleMethodTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarClassPathTransitiveClosureTestCase
org.jboss.as.testsuite.integration.osgi.ejb3.StatelessBeanIntegrationTestCase
org.jboss.as.testsuite.integration.injection.resource.superclass.BindingsOnInterceptorTestCase
org.jboss.as.testsuite.integration.weldejb.injectionTarget.EjbInjectionTargetWrapTestCase
org.jboss.as.testsuite.integration.jpa.hibernate.SessionFactoryTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarManifestDependencyPropagatedTestCase
org.jboss.as.testsuite.integration.ejb.stateful.exception.SystemExceptionTestCase
org.jboss.as.testsuite.integration.weld.jpa.scoping.WeldJpaInjectionScopeTestCase
org.jboss.as.testsuite.integration.ejb.stateful.serialization.StatefulSerializationTestCase
org.jboss.as.testsuite.integration.weldejb.injection.EjbInjectionIntoCdiBeanTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarJbossStructureExtendedVisibilityTestCase
org.jboss.as.testsuite.integration.pojo.test.SimpleBeansTestCase
org.jboss.as.testsuite.integration.as859.AS859TestCase
org.jboss.as.testsuite.integration.sar.unit.SarWithinEarTestCase
org.jboss.as.testsuite.integration.ejb.home.localhome.SimpleLocalHomeTestCase
org.jboss.as.testsuite.integration.osgi.http.HttpServiceTestCase
org.jboss.as.testsuite.integration.pojo.test.CallbackBeansTestCase
org.jboss.as.testsuite.integration.ejb.stateful.locking.ReentrantLockTestCase
org.jboss.as.testsuite.integration.deployment.classloading.war.WarChildFirstClassLoadingTestCase
org.jboss.as.testsuite.integration.jaxrs.multipart.JaxrsMultipartProviderTestCase
org.jboss.as.testsuite.integration.ejb.transaction.cmt.never.NeverTransactionTestCase
org.jboss.as.testsuite.integration.osgi.xservice.BundleAccessesModuleServiceTestCase
org.jboss.as.testsuite.integration.jaxrs.packaging.war.ApplicationPathOverrideIntegrationTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarClassLoadingTestCase
org.jboss.as.testsuite.integration.osgi.logging.LoggingTestCase
org.jboss.as.testsuite.integration.ejb.view.unit.BusinessViewAnnotationProcessorTestCase
org.jboss.as.testsuite.integration.ejb.remove.RemoveTestCase
org.jboss.as.testsuite.integration.jaxrs.packaging.ear.ApplicationIntegrationTestCase
org.jboss.as.testsuite.integration.weldejb.SessionObjectReferenceTestCase
org.jboss.as.testsuite.integration.jaxrs.jaxb.JaxbProviderTestCase
org.jboss.as.testsuite.integration.injection.resource.enventry.EnvEntryInjectionTestCase
org.jboss.as.testsuite.integration.injection.resource.multipleinterceptors.BindingsOnInterceptorTestCase
org.jboss.as.testsuite.integration.jpa.resourcelocal.ResourceLocalTestCase
org.jboss.as.testsuite.integration.ejb.stateful.timeout.StatefulTimeoutTestCase
org.jboss.as.testsuite.integration.jaxrs.jackson.JaxrsJacksonProviderTestCase
org.jboss.as.testsuite.integration.injection.resource.ejblocalref.EjbLocalRefInjectionTestCase
org.jboss.as.testsuite.integration.jpa.jarfile.JpaJarFileTestCase
org.jboss.as.testsuite.integration.jaxrs.packaging.ear.ApplicationPathIntegrationTestCase
org.jboss.as.testsuite.integration.jpa.epcpropagation.requiresnew.EPCPropagationNewTransactionTestCase
org.jboss.as.testsuite.integration.ejb.interceptor.lifecycle.order.PostConstructOrderTestCase
org.jboss.as.testsuite.integration.jpa.webnontxem.NonTransactionalEmTestCase
org.jboss.as.testsuite.integration.osgi.eventadmin.EventAdminTestCase
org.jboss.as.testsuite.integration.osgi.webapp.WebAppTestCase
org.jboss.as.testsuite.integration.ejb.descriptor.CustomDescriptorTestCase
org.jboss.as.testsuite.integration.osgi.xml.parser.SAXParserTestCase
org.jboss.as.testsuite.integration.jpa.epcpropagation.EPCPropagationTestCase
org.jboss.as.testsuite.integration.deployment.classloading.war.WarClassLoadingTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.WarCanAccessEjbJarTestCase
org.jboss.as.testsuite.integration.webejb.ServletInjectionTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarClassPath3TestCase
org.jboss.as.testsuite.integration.ejb.interceptor.lifecycle.chains.InterceptorLifecycleSFSBTestCase
org.jboss.as.testsuite.integration.jaxrs.packaging.war.ApplicationIntegrationTestCase
org.jboss.as.testsuite.integration.osgi.xservice.ModuleAccessesModuleServiceTestCase
org.jboss.as.testsuite.integration.jpa.hibernate.envers.BasicEnversTestCase
org.jboss.as.testsuite.integration.jaxrs.integration.cdi.CDIResourceInjectionTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarLibClassPathTransitiveClosureTestCase
org.jboss.as.testsuite.integration.pojo.test.CycleBeansTestCase
org.jboss.as.testsuite.integration.ejb.transaction.cmt.beforecompletion.BeforeCompletionExceptionDestroysBeanTestCase
org.jboss.as.testsuite.integration.injection.resource.enventry.DuplicateGlobalBindingInjectionTestCase
org.jboss.as.testsuite.integration.injection.resource.persistencecontextref.PersistenceContextRefTestCase
org.jboss.as.testsuite.integration.jaxrs.subresource.SubResourceTestCase
org.jboss.as.testsuite.integration.jpa.ormxml.OrmTestCase
org.jboss.as.testsuite.integration.jpa.packaging.PersistenceUnitPackagingTestCase
org.jboss.as.testsuite.integration.websecurity.WebSecurityFORMTestCase
org.jboss.as.testsuite.integration.beanvalidation.hibernate.scriptassert.ScriptAssertTestCase
org.jboss.as.testsuite.integration.osgi.jmx.MBeanServerTestCase
org.jboss.as.testsuite.integration.weldejb.constructor.EjbConstructorInjectionTestCase
org.jboss.as.testsuite.integration.xerces.unit.XercesUsageTestCase
org.jboss.as.testsuite.integration.websecurity.WebSecurityJBossWebXmlSecurityRolesTestCase
org.jboss.as.testsuite.integration.weldejb.interceptor.injection.CDIInjectionIntoEJBInterceptorTestCase
org.jboss.as.testsuite.integration.ejb.descriptor.MergedDescriptorTestCase
org.jboss.as.testsuite.integration.osgi.jta.TransactionTestCase
org.jboss.as.testsuite.integration.ejb.remote.byvalue.RemoteInvocationByValueTestCase
org.jboss.as.testsuite.integration.ejb.interceptor.method.EjbMethodInterceptorTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarClassPath2TestCase
org.jboss.as.testsuite.integration.deployment.classloading.rar.RarClassLoadingTestCase
org.jboss.as.testsuite.integration.jsf.managedbean.xml.XmlManagedBeanTestCase
org.jboss.as.testsuite.integration.ejb.transaction.cmt.mandatory.SFSBMandatoryTransactionTestCase
org.jboss.as.testsuite.integration.ejb.remote.async.RemoteAsyncInvocationTestCase
org.jboss.as.testsuite.integration.weldejb.interceptor.exceptions.CDIEJBInterceptorExceptionTestCase
org.jboss.as.testsuite.integration.osgi.webapp.ServletIntegrationTestCase
org.jboss.as.testsuite.integration.security.CustomLoginModuleTestCase
org.jboss.as.testsuite.integration.ejb.packaging.injection.CrossModuleInjectionTestCase
org.jboss.as.testsuite.integration.jaxrs.packaging.ear.NoApplicationIntegrationTestCase
org.jboss.as.testsuite.integration.websecurity.WebSecurityBASICTestCase
org.jboss.as.testsuite.integration.as835.AS835TestCase
org.jboss.as.testsuite.integration.osgi.configadmin.ConfigurationAdminTestCase
org.jboss.as.testsuite.integration.injection.resource.resourceref.ResourceRefTestCase
org.jboss.as.testsuite.integration.deployment.structure.JBossDeploymentStructureTestCase
org.jboss.as.testsuite.integration.jaxrs.integration.cdi.CDIResourceInjectionEarTestCase
org.jboss.as.testsuite.integration.ejb.localview.EjbLocalViewTestCase
org.jboss.as.testsuite.integration.jpa.epcpropagation.hierarchy.EPCPropagationHierarchyTestCase
org.jboss.as.testsuite.integration.weld.jpa.WeldJpaInjectionTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarJbossStructureAdditionalModuleTestCase
org.jboss.as.testsuite.integration.weldejb.interceptor.packaging.CdiInterceptorEarPackagingTestCase
org.jboss.as.testsuite.integration.jmx.ModelControllerMBeanTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EjbJarCanAccessOtherEjbJarTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarNestedBundleTestCase
org.jboss.as.testsuite.integration.injection.resource.noncomponent.NonComponentResourceInjectionTestCase
org.jboss.as.testsuite.integration.injection.resource.infinispan.InfinispanResourceRefTestCase
org.jboss.as.testsuite.integration.weldejb.interceptor.binding.WeldInterceptorBindingTestCase
org.jboss.as.testsuite.integration.ejb.transaction.annotation.NoExplicitTransactionManagementValueTestCase
org.jboss.as.testsuite.integration.weld.interceptor.packaging.InterceptorPackagingTestCase
org.jboss.as.testsuite.integration.ejb.descriptor.MetadataCompleteCustomDescriptorTestCase
org.jboss.as.testsuite.integration.deployment.classloading.war.WarInEarChildFirstClassLoadingTestCase
org.jboss.as.testsuite.integration.osgi.blueprint.BlueprintTestCase
org.jboss.as.testsuite.integration.osgi.jndi.JNDITestCase
org.jboss.as.testsuite.integration.ejb.remote.stateful.StatefulInVmRemoteInvocationTestCase
org.jboss.as.testsuite.integration.deployment.classloading.ear.EarClassPath1TestCase
org.jboss.as.testsuite.integration.ejb.injection.ejbs.EjbsInjectionTestCase
org.jboss.as.testsuite.integration.jsf.managedbean.annotation.AnnotatedManagedBeanTestCase

WFLY-582

WFLY-576 TS: Check -Dtest

Make sure that -Dtest can be used to run a single test (at the
moment, this works, but the module to be used really needs to be specified).

WFLY-581

WFLY-576 TS: Check scripts options.

Make sure that all options to build.sh and integration-tests.sh are
being processed correctly (likewise for the Windows versions)

WFLY-580

WFLY-576 TS: Implement test reporting.

Make sure test reports are being produced correctly (this may involve
chasing up Paul Gier for the surefire release).

WFLY-579

WFLY-576 TS: Check regressions

make sure that the modified testsuite does not introduce any
regressions. This includes tests run agains the default server profile
as well as the preview server profile. (There are currently failures
against the preview server profile in the integration tests)

WFLY-578

WFLY-576 TS: Compare sets of tests run in both versions for respective groups

Make sure that the set of tests run by the modified testsuite is the
same as the set of tests run by the unmodified testsuite (this is
because the includes/excludes have been rearranged in some cases,
especially in the integration module).

COMB="new-DallTests-clean"
DIR=~/work/AS7/TS-compare/$COMB
mkdir -p $DIR/allReports
find testsuite/ -name TEST-* > $DIR/list.txt
for i in `find testsuite/ -name *TestCase*.txt`; do cp $i $DIR/allReports ; done
for i in `find testsuite/ -name TEST-*`;         do cp $i $DIR/allReports ; done
WFLY-577

WFLY-576 Tests must fail the build by default.

To override, use:
<surefire.test.failure.ignore>false</surefire.test.failure.ignore>
or
-Dmaven.test.failure.ignore=true

Cancel