Step 1 of 4: Choose Issues


Key Summary Description

WFLY-576 TS: Make Surefire test ordering configurable.


WFLY-576 TS: API changes report - Animal sniffer maven plugin

Add support for such report.


mvn animal-sniffer:build -DincludeJavaHome=false -DskipIfNoJavaHome=true

WFLY-576 TS: Dynamic way to get ports of various services.

private int getRecoveryManagerPort() throws IOException {
    /*    /socket-binding-group=standard-sockets/socket-binding=txn-recovery-environment:read-attribute(name=port)  */
    final ModelNode address = new ModelNode();
    address.add("socket-binding-group", "standard-sockets");
    address.add("socket-binding", "txn-recovery-environment");
    final ModelNode operation = new ModelNode();
    try {
        return executeOperation(operation).asInt();
    } catch (MgmtOperationException ignored) { }
    return -1;

WFLY-576 Implement fallback mechanism for AS7 properties in pom.xml

There are several places where is used fallback mechanism when some content from property is taken like:

static final String someIPproperty = System.getProperty("<some> key>", "localhost");

The final decision based on (linked) AS7 devel discussion is leaving this approach and set-up this things in pom.xml's only.

After this task will be done, old approach will be leaved at all and every new patch will be forced to not use/include a such fallback mechanism only.


WFLY-576 TS: Allow XSLT to fit any namespace versions.

Namespaces versions change during development, which leads to upstream testsuite versus EAP releases mismatch, which is undesirable as the testsuite sometimes contains more up-to-date tests.

It would be good to have XLST matching namespaces but except version.


WFLY-576 TS: Implement replacing of ...=localhost in .properties resources

There are 3 occurences of $SUBJ, see below.

Since they are in src/, it needs to be copied elsewhere and added to classpath from there.

$ git grep -A1 =localhost

WFLY-576 TS: -Dtest=*<something>* only matches one test per module.

It seems like some regression.

When I've tried run like:

mvn3 clean install -Dts.basic '-Dtest=*.jaxrs.*'

I got this result:

[pjanouse@pjanouse testsuite]$ mvn3 clean install -Dts.basic '-Dtest=*.jaxrs.*'
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.689 sec

------ basic module:
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.099 sec

So you can see - Integration basic is invoked, but only TestCases and were run although there are plenty of other TestCases fulfilled mask *.jaxrs.* - many of them are localted in testsuite/integration/basic/src/test/java/org/jboss/as/test/integration/jaxrs/ + sub=dirs.


WFLY-576 TS: Allow debugging client with -DdebugClient (only AS is possible now)


WFLY-576 TS: Some tests missing from <includes> and are not run.


WFLY-576 TS: Script to check missed tests.


WFLY-576 TS: Add $managementAddress and $managementPort to all arquillian.xml's.

<property name="managementAddress">${node0:}</property>
<property name="managementPort">${as.managementPort:9999}</property>



WFLY-576 TS: Pass node0, node1 to <ant ...> since inheritAll="true" doesn't work.


WFLY-576 TS: parametrize surefire test.redirectTestOutputToFile property


WFLY-576 TS: Fix XSLT transformations (namespaces problem)

Applying the standard XSLT copy template,

<xsl:stylesheet xmlns:xsl=""

    <xsl:template match="@*|node()">
            <xsl:apply-templates select="@*|node()"/>

to the JBoss AS 7 standalone.xml, leads to loss of parameters:

<?xml version="1.0" encoding="UTF-8"?><server xmlns="urn:jboss:domain:1.1">

instead of

<?xml version="1.0" encoding="UTF-8"?><server xmlns="urn:jboss:domain:1.1">
        <extension module=""/>
        <extension module=""/>

It's something namespace-related because ds:... attribs ARE copied:

                <ds:datasource xmlns:ds="urn:jboss:domain:1.1" jndi-name="java:jboss/datasources/ExampleDS"

WFLY-576 TS: Pass the path to docs/examples to the tests.

In AS7, the path to the config file, passed in a system property, is only allowed to be a relative path.

In the testsuite, we would make use of absolute paths.
The reason is that we reuse the configs from docs/examples, and they change quite often, so having them in resources would be hardly maintainable.
However, docs/examples is not guaranteed to be always in the same place: E.g. for EAP, or for RPM-based installation of EAP it might end up in /var/docs or such.

I would like to pass some property from maven to the tests, but that results in absolute path.


WFLY-576 TS: Make container JVM --enable-assertions configurable.

Arquillian's AS7 container has a property:

<property name="enableAssertions">false</property>

And it's true by default.
Let's make it configurable.


WFLY-576 TS: Check how JDBC drivers are copied

I created some initial impl, but that copies drivers to deployments/.

The preferred way for TS harness is to create a module, since the other way is easy to do manually in case if interrest.

What's specially needed to check is that the driver and DB config ends up in all instances of multi-node modules (cluster, multinode, etc.).


WFLY-576 TS: Make commons-lang3, commons-io and commons-codec available in tests.

To relieve people from writing methods like toHex(), defaultIfNull() and escapeHtml(),
I'd like to add commons-lang3, commons-io and commons-codec.
These are de-facto standard, well tested, mostly thread-safe.

For tests running in server, these would still need to be packaged into the deployment.


WFLY-576 TS: Check endorsed jboss-annotations-api.

(22.12.2011 04:09:13) ozizka: Speaking of hacks, how about jboss-annotations-api_1.1_spec being set as endorsed during testsuite compilation
(04:09:25) ozizka: The comment says "Big complex hack just to get @Resource(lookup="foo")."
(04:09:34) ozizka: Is that stil needed?
(04:37:14) bstansberry: I don't know. I never really knew what that was all about
(04:37:39) bstansberry: i doubt it's needed, just because it sounds so weird and dates so far back
(04:38:12) bstansberry: oh, i can guess what it's about
(04:38:50) bstansberry: wacky Sun/Oracle have the same class in both SE and EE but with extra members in EE
(04:39:20) bstansberry: "lookup" isn't in @Resource in SE


WFLY-576 TS: Find a way to retain same value of property with ${basedir} in submodules.

John Casey:

Can you explain exactly how the ${basedir} property will be used? This matters, because expression resolution happens two different ways, once for POM interpolation and again for plugin parameter injection. For more information on the differences, you can play with an expression plugin I built: (see the README for information on using it).

We need to refer to ${basedir} for two purposes:

1) To be able to pass important directories to the tests, for example:

  • AS project root dir
  • Testsuite root dir (to access global resources)
  • Integration testsuite root dir (to access int. ts. resources)
    In this case, we need to be able to define the value in parent pom and retain it for sub-modules.

2) For various operations (plugin executions) which are defined in parent pom, but inherited in sub-modules.
In this case, we really need to have it done using relative paths, e.g. ${basedir}/src/config/arquillian .

In both cases, ${basedir} is used for plugins configuration.

John Casey:

I've written a plugin called org.commonjava.maven.plugins:directory-maven-plugin that has goals:

  • execution-root = Resolves to the directory where maven was invoked
  • directory-of = uses a <project/> configuration with groupId and artifactId to find another project in the current session, and returns its basedir
  • highest-basedir = traverses the parent hierarchies of all current projects, up until it finds a parent that has been resolved instead of loaded locally. Then, it sorts these basedir paths and returns the one closest to the filesystem root. If the top two hits (highest results) are not nested within one another, it will fail.

I'd expect you to use something like these to pin down a particular project basedir reference, and inject it consistently into each project in the session. It may be that you actually need something like a combination of the last two, to reference a particular parent project that was loaded from disk (but may not be in the current session per se). If so, that's another easy one to write.

The plugin is here:

You can take a look at the for more information.


WFLY-576 TS: Way to build with a clean local repo.

This can be done either by using an empty local repo, setting it in settings.xml,


or by using the dependency:purge-local-repository goal in a profile:


                        <executions> <execution>
                           <configuration> <removeAll>true</removeAll> </configuration>
                        </execution> </executions>


However first can only be done in settings.xml, and not in a profile, and the later can be done with `mvn dependency:purge-local-repository build-helper:remove-project-artifact -Dbuildhelper.removeAll`.
So it's not to be done in the testsuite itself. Resolving.


WFLY-576 TS: -Djboss.dist=... has no effect, /build/target/jboss-as* is always used

Please check the usage of -Djboss.dist parameter provided from the command line. It seems the testsuite does not take it and still uses jboss.dist as defined in testsuite/pom.xml#<properties>. Looks like maven properties are a bit different than system properties, read for example the following:



WFLY-576 TS: Database switch (in datasources)

This is how Hibernate did it:


WFLY-576 TS: Multiple Surefire executions - FAILURE in one supresses successive.

Jason wrote that we can live with this for CR1; However having it fixed for EAP is important.

Work in progress at .
John Casey keeps an eye on this.


WFLY-576 TS: Option to fail if AS is running (~ a port is open)

<target name="check-port" description="Checks whether AS is running.">
    <condition property="jbossas.running">
        <socket server="${}" port="${jbossas.port}"/> 

WFLY-576 TS: Configurable timeouts

Testsuite run can take different amount of time on various HW (or virtuals).
All timeouts must be configurable, in the sense of being possible to prolong.

We should also divide them into categories, depending on what is expected to take long (consider slow disks, slow networks, cloud environment)
Few ideas for categories:

  • Filesystem I/O
  • Processor
  • Network I/O
  • Memory I/O
  • Database operations
    For each category, there would be a ratio, 1.0 by default, which would multiply the timeout.

So the timeout would look like:

something.setTimeout( 1000 * Timeouts.getFSIORatio() );

Other possibility is to use @Annotations to mark tests belonging to certain group, like @LongRunning or such.


WFLY-576 TS: Option to run under security manager.




There are some tests failing under globally set SM because they use their own.
These need to be exluded from running when security manager is in use.


WFLY-576 TS: Implement self-test.

Implement a script which would verify the health of the testsuite - i.e.:

  • Run scripts with various params & check the results
  • Run mvn with various params & check the results

By results, we mean:

  • Same set of tests run
  • Same results as before
  • AS is configured the same way (if not intentionally changed)

WFLY-576 TS: Check tests for hard-coded strings like URLs etc. - candidates for parametrization.


WFLY-576 TS: Update windows scripts (build.bat and integration-tests.bat)


WFLY-576 TS: (Re-)Create test group for web profile.

(20:46:10) Nihility: OndrejZizka: got a few seconds to talk testsuite
(20:46:10) smcgowan: baileyje:
(20:46:20) OndrejZizka: Nihility: Sure
(20:46:23) jharting opustil místnost (quit: Quit: Leaving.).
(20:46:54) Nihility: OndrejZizka: ok so for the beta release i need to resurect the old default profile "the web profile" and make the existing one the "full" profile
(20:47:17) Nihility: OndrejZizka: i had some long conversations with various folks about how to do that without screwing up the progress that has already been made with the testsuite
(20:48:06) Nihility: OndrejZizka: it seems that the best solution was to have a property which tells the testsuite to run against the "web" profile, using the different config, and also disabling tests using a pattern match
(20:48:27) Nihility: OndrejZizka: so basically we would have to manually kick off a web run, or have a hudson run or something
(20:49:04) Nihility: OndrejZizka: did you have any thoughts on that? I had originally hoped we would use different configs for different tests, but that seems to be a nightmare in our current maven configuration
(20:49:28) Nihility: OndrejZizka: or any other ideas would be welcome
(20:49:32) OndrejZizka: Nihility: Have you seen my last post in jboss-as7-dev?
(20:49:59) OndrejZizka: Nihility: I have modified the testsuite a bit, so I believe this task is easier to do with it now too,
(20:50:07) OndrejZizka: I only need someone to fix OSGi tests
(20:50:41) Nihility: OndrejZizka: all i have is "printing project number"
(20:50:42) OndrejZizka: After that, web profile vs. full profile can be simply a profile activating different modules
(20:51:35) OndrejZizka: Nihility: Forwarded
(20:51:46) Nihility: OndrejZizka: i mean do you think the different config per test module is reasonable?
(20:51:59) Nihility: OndrejZizka: it seemed like we would have to do a BIG resturcturing for that
(20:52:23) OndrejZizka: Nihility: Depends on what are the differences, and for which modules,
(20:52:45) OndrejZizka: Inside the integration module, the differences are not significant,
(20:53:01) OndrejZizka: the modules are used to separate tests virtually
(20:53:07) Nihility: OndrejZizka: yeah thats the sisue the integration module mixes full and non-full
(20:53:59) OndrejZizka: Nihility: Could you send me a list of which modules are non-full? And also, the config for web AS profile
(20:54:15) OndrejZizka: or a list of differences
(20:54:40) Nihility: OndrejZizka: im working under the assumption we have to run twice (i figured there was no time to restructure to that level)
(20:55:23) Nihility: OndrejZizka: yeah sure, its basically standalone but without jacorb, hornetq, and jaxr
(20:55:34) Nihility: OndrejZizka: and webservices
(20:55:55) OndrejZizka: Twice? There are two executions of surefire, which is because some tests test some persistent stuff
(20:56:43) Nihility: OndrejZizka: no i dont mean automatically run twice, i mean the user has to say something like ./ -Dweb-profile or something
(20:57:01) Nihility: OndrejZizka: to be honest i dont like that solution as much as doing a config per module
(20:57:28) OndrejZizka: Nihility: I think this could be solved too
(20:57:31) Nihility: OndrejZizka: but the config per module sounds like there is just not enough time, we need to get teh beta out this week
(20:58:20) smcgowan opustil místnost (quit: Quit: Leaving).
(20:59:02) OndrejZizka: Nihility: Check this
(20:59:03) OndrejZizka:
(20:59:29) OndrejZizka: It's basically done, only OSGi tests fail because there's some assumption about relative paths
(20:59:38) Nihility: ah intersting
(20:59:47) Nihility: so you use a maven module to activate profiles?
(20:59:51) OndrejZizka: If someone fixes that, I think we could use that for the requirements you stated
nickarls Nihility
(21:00:13) OndrejZizka: Nihility: and vice versa - this way is the most flexible
(21:00:47) OndrejZizka: Using profiles, I activate sets of modules, which define -DallTests, or a specific group;
(21:01:17) OndrejZizka: then, in these modules, there are profiles which can be disabled, and also "global" profiles which change database config etc.
(21:02:35) Nihility: ah ok so in theory
(21:02:44) Nihility: we would have a group of "full" tests
(21:02:47) Nihility: or something
(21:03:48) OndrejZizka: Nihility: yes
(21:04:16) OndrejZizka: Either as a module, or as a profile.
(21:04:24) OndrejZizka: in the integration/basic module
(21:04:28) OndrejZizka: I'd prefer the second,
(21:04:43) OndrejZizka: but if it turns out there's some problem with maven, we can fall back to module-based selection
(21:04:52) OndrejZizka: and they can run both in one run
(21:05:35) OndrejZizka: I mean, running two executions of same tests using profiles is sometimes troublesome
(21:05:52) Nihility: yeah and its time consuming
nickarls Nihility
(21:08:52) OndrejZizka: Nihility: So, do we need to run the web tests twice?
(21:09:01) OndrejZizka: in this full/non-full scenario
(21:09:16) OndrejZizka: Or in full, only the complement would be run?
(21:09:42) OndrejZizka: * complementary tests
(21:11:12) ctomc vstoupil do místnosti.
(21:11:19) Nihility: OndrejZizka: well IMO we only need to run the "full-only" tests against standalone-full.xml
(21:12:25) OndrejZizka: Nihility: Ok, that should be okay. So I'll prepare a PoC and ping you, okay?
(21:12:33) OndrejZizka: Which TZ is in?
(21:13:03) Nihility: OndrejZizka: ok, hes in munich which i think is GMT+1
(21:13:33) Nihility: OndrejZizka: we should chat with stuartdouglas about this as well
(21:16:11) maxandersen Adium@redhat/jboss/maxandersen vstoupil do místnosti.
(21:16:11) režim (+v maxandersen) od ChanServ
(21:16:18) Nihility: OndrejZizka: so according to that email the issue he brings up is that we have duplicate sources?
(21:17:50) stuartdouglas: I really don't like the 1 big ball of sources thing
(21:18:14) stuartdouglas: I think it overcomplicates things, requires additional configuration, and may cause dependency and other problems down the road
stalep stuartdouglas
nickarls Nihility
(21:22:56) OndrejZizka: stuartdouglas, Nihility, when such problems arise, there's no problem in separating them
(21:23:31) OndrejZizka: cd testsuite/integration; mv src/test/java/foo/bar module/src/test/java/foo/bar
(21:23:38) OndrejZizka:
(21:23:57) OndrejZizka: Not counting resources
(21:24:19) OndrejZizka: But I hope that tests keep their resources well separated
(21:24:33) OndrejZizka: As per Jason's recent post to dev list
(21:26:40) darranl darranl@redhat/jboss/darranl vstoupil do místnosti.
(21:26:40) režim (+v darranl) od ChanServ
(21:27:47) OndrejZizka: stuartdouglas: Either way - pushing everything to a single module doesn't work, so the question is not whether to stay with current or split,
(21:28:08) OndrejZizka: but whether to split the modules and keeps tests together, or split them "physically"
(21:28:27) OndrejZizka: I have a) done, stuart likes b), can be done too
(21:28:31) stuartdouglas: I think we should go with splitting them up into different modules, with the relevant source in each module
(21:28:38) stuartdouglas: it just seems like the simplest approach
(21:28:49) stuartdouglas: and it is also the 'standard' maven way
(21:29:12) stuartdouglas: From your point of view, what are the downsides to doing it this way?
(21:31:48) CheffPJ vstoupil do místnosti.
(21:32:26) Nihility: stuartdouglas: OndrejZizka assuming we did the split into smaller finer grained modules, would the activation thing still work (e.g. we create dummy modules that activate the other ones)?
(21:33:02) Nihility: stuartdouglas: OndrejZizka or would we just use the shell scripts?
navssurtani1 nickarls Nihility
nickarls Nihility
(21:33:53) OndrejZizka: Nihility: The separation needs was the original reason why I started this change
nickarls Nihility
(21:34:25) OndrejZizka: Nihility: So, the activation would be done by activating different profiles, which would include
(21:34:42) OndrejZizka: <modules> <module>...</module> <module>...</module> <module>...</module> ... </modules>


WFLY-576 TS: Hitting max file limit with reporting (`mvn site`)


WFLY-576 TS: Clustering tests - figure out how they will be run

The current single test is not multinode - when set up as multinode, it fails. So it runs as single node, even though being in package .cluster.


WFLY-576 TS: Move *-arquillian.xml from testsuite.integration/src/test/resources/** to some better place.


WFLY-576 TS: Add properties "jbossas.ts.integ.dir", "jbossas.ts.dir", "jbossas.project.dir" and use them in test.

Instead of implying that the test runs in certain subdirectory of the testsuite, define use these System variables passed to Surefire:

"jbossas.ts.integ.dir", "jbossas.ts.dir", "jbossas.project.dir"


WFLY-576 Propagate property node0 to system properties

Propagate property node0 (and other ones) to system properties for test being able to take IP address where the AS runs.


WFLY-576 TS: Print help banner on `mvn install` without params.


Would it be possible to somehow print out all the available options and
params that need to used, when you run mvn clean install from the
testsuite modules? Something like a help?{qoute}

Not sure if it will be possible to check the condition "There are no params", but we can print it every time as well.


WFLY-576 TS: Test output is not stored in ...-output.txt

./ clean install* -Dintegration.module -Dbasic.integration.tests


WFLY-576 TS: Generate server config for Arq instances (standalone-*.xml) using XSLT instead of keeping them in resources.


WFLY-576 TS: Move smoke and integration tests before the domain stuff.

(21:31:55) bstansberry: OndrejZizka1: I'd like to re-order the testsuite execution a bit but am a bit lost on what's driving the current order
(21:33:11) bstansberry: OndrejZizka1: it would be better if smoke and integration tests ran before the domain stuff, as the domain stuff is slow and just delays an overall failure if one of the more numerous smoke/integration tests fails{query}

WFLY-576 TS: `mvn install` and `` must run smoke tests by default.


WFLY-576 TS: Define testsuite acceptance test (a testsuite for the testsuite)

Define a procedure to perform before committing something to the testsuite.

Create scripts to help determining whether given testsuite is okay to be committed.

IN='-DallTests -Dinteg-tests -Dcluster-tests -Dbasic-tests -Dbenchmark-tests -Dsmoke-tests -Dstress-tests -Ddomain-tests -Dcompat-tests';
for GROUP in $(echo $IN | tr ";" "\n"); do
  ## Create results dir.
  mkdir -p $RES_DIR
  ## Run the testsuite.
  ./ clean -DallTests;
  ./ install $GROUP;
  ## Archive the reports.
  for REPORT in `find testsuite -name 'TEST-*.xml' -or -name '*TestCase.txt' -or -name '*TestCase-output.txt'`; do
    SUB_DIR=`dirname $REPORT | sed 's#testsuite/##' | sed 's#/target/surefire-reports##'`
    mkdir -p $RES_DIR/$SUB_DIR
  ## Run the testsuite for 2nd time to ensure that unclean run works too.
  ./ install $GROUP;
## Create a JUnitDiff report.

Also, check what's being passed to `mvn` on various params combinations:

./ -DallTests
./ -DallTests -Dfoo
./ install -DallTests
./ clean -DallTests
./ clean install -DallTests -Dfoo
./ -DallTests;
./ install -DallTests;
./ clean -DallTests;
./ clean install -DallTests -Dfoo;
./ test -DallTests -Dfoo;

WFLY-576 TS: On -Dtest=..., activate appropriate modules automatically (using package prefix) relieve people from writing `-Dintegration.module` all the time.


WFLY-576 TS: Unify activation properties' names (e.g. -DstressTests instead of -Dstress-test)


WFLY-576 TS: IPv6 testing issues (tracking)

This includes:

  • Check AS management support for IPv6
  • Check Arquillian's support
  • Config param to scripts / maven property passed to arquillian
  • Replacing "http://localhost:..." in tests with a configurable value.

QA contact: Pavel Janousek - See 4. Future work ... more than 50% missing

Java IPv6 guide:


WFLY-576 TS: Fix -Ddebug (-Djpda)

ozizka-as7/testsuite/integration $ mvn install -Dclustering.integration.tests -Djpda

doesn't work.


WFLY-576 TS: Move Arquillian configs from src/test/resources/X to some special dir.

Currently it's these:



WFLY-576 TS: Logging issues (tracking)

Some tests print to System.out, and also ex.printStackTrace(). Turn it to proper logging. Also check logging config.


WFLY-576 TS: Check tests workdirs

Some tests (also unit tests) have work dirs in ${basedir}, which leaves garbage after running.
That should change either to target/work or /tmp/... (consider what's better).


WFLY-576 TS: Get new harness to master.

  • pom.xml's
  • scripts
  • common variables script
  • integration
    + XSLT's
  • arquillian.xml's
    + Ant scripts (src/test/scripts)
    + basic... (removed stuff for removed test)
    + removed compat... and timerservice... (melded)

WFLY-576 Alternative JDKs for building and running - OpenJDK 6, 7, Sun JDK 7, IcedTea

James Perkins:

At one point there was a JIRA for OpenJDK 6, in fact I think a couple. The issue with OpenJDK 6 is in the build we use some JavaScript and OpenJDK 6 doesn't come with a JavaScript engine. I tried a couple things to get Rhino working with it as that's what the Sun JDK uses, but I think it needs to implement an SPI to get it to work. I didn't look into it much beyond that.

JDK 7 is a different issue. It's a bug in the annotation processing API which the JBoss Logging Tooling uses. I just refreshed my OpenJDK 7 update source and it looks like the bug is fixed in there. It's fixed in IcedTea as well. There could be the JavaScript issue here as well I can't actually remember, but I thought I had some successful builds with custom JDK's I compiled.

I did just try with IcedTea 7 and got some other errors. I'm building the latest upstream of OpenJDK 7 now and we'll see if it works. I'll let you know either way.

Let me know if you have any more questions on this. I did dig into a while ago a little.


WFLY-576 TS: Put all management tests into one dir (


and further separated by subsystems/interfaces:

Thanks Dominik for pointing out.


WFLY-576 TS: Pick the most acceptable test packages organization

1) Split classes to src/main and src/test ?
This seems to be decided before - integration tests suite has all classes in src/test.

(02:35:15) barreiro: When I start I done it that way so that it clear to me what was the test code, the test subject and the resources — I knew that probably I would have to change the location later.

2) Low-level Test packages micro

(02:37:51) barreiro: should I create one sub package for each test — like /web/xxxTestCase — or stick with the old schema of having web/tests and web/servlets, ... etc.
(02:38:01) ozizka: Not sure if democracy is the principle to follow here, but I'd like devs to agree upon these things. TBD next week
(02:38:41) ozizka: Depends if you reuse them.
(02:39:24) ozizka: But generally, I'd prefer the first approach -
(02:39:53) ozizka: with reused things being placed one level higher, or next to
(02:40:16) ozizka: The structure should reflect test's structure
(02:40:24) barreiro: IMO, democracy is not the way to go in a OSS project ... I'll do it the way you ask me to
(02:40:34) ozizka: Ok
(02:40:57) ozizka: But anyway - what's your preference?
(02:41:07) ozizka: Or what would suite better for your case
(02:42:42) barreiro: the ultimate goal is that someone from outside can easily understand what's going on. Like me, I just landed in this project.
(02:43:58) barreiro: surely that it should reflect the test's structure. but I would rather have a common structure in most modules ...
(02:45:30) barreiro: like, at some level there is a clear list of the tests in the package that is not mixed with other stuff — be it a list of packages or a test package with all the test classes
(02:45:43) ozizka: Well, 1st option is current practice, I'd stick with that.
(02:46:47) barreiro: and from there should be clear (as possible) what resources are used.
(02:47:34) ozizka: I was thinking about using a mechanism used in Wicket framework, if you know that,
(02:47:57) ozizka: which is, having a mechanism to load resources from places
(02:48:04) barreiro: After a while I start to understand how AS6 is structured ... the problem is that there is too much voodoo in the back that defeats it's purpose.
(02:48:18) ozizka: where the test class is loaded, or it's superclass, etc.
(02:48:45) ozizka: Eventually it could go by packages up. But perhaps too magical for a testsuite
(02:49:10) ozizka: Right. That's the voodoo.
(02:56:27) barreiro: yep. Shrinkwrap and arquilling are great for that. The magic is greatly reduced, without adding much complexity.
(02:58:10) ozizka: One other argument to have the package-per-test separation is that if we decided the other way, moving the files would be easier


WFLY-576 TS: Review smoke tests

Currently, smoke tests are in package .smoke.
I'd rather see them in appropriate package,
and differentiated by some grouping mechanism. See AS7-2086.

(02:07:46) stuartdouglas: but really, the smoke tests need review
(02:08:03) stuartdouglas: to make sure they cover lots of different stuff in a general way
(02:08:12) stuartdouglas: and also I would like to remove their dependency on the demo's
(02:08:22) stuartdouglas: my thoughts where to just have them named SmokeTestCase

Also, let's remove the "embedded" from package name.


WFLY-576 TS: Add a module for shared util classes.

To prevent people coding their own impl of what was coded twenty-times in other packages of TS.

There's already a package: ``

It could be a module to keep it's dependencies separated.

Perhaps also Apache Commons Util should be added as a dependency.


WFLY-576 TS: Tests grouping

Decide which mechanism to use to assign the tests into one or multiple groups.



Also, Arquillian itself supports grouping:



3) Maven Surefire <includes> and <excludes>


WFLY-576 TS: Consolidate documentation of the testsuite.

1) Update README.txt
2) Update
3) Finish .
4) Maybe update


WFLY-576 TS: Discuss resources location - next to .java files?

Having resources in completely different dir tree makes sense for business apps where they are often maintained by different teams (css, images, ...).
For a testsuite, where all resources are maintained by devs, usually even the same as sources which use them, there's no real reason to keep them separated.

Many devs prefer having resources as close to the sources as possible, which sometime leads to putting text content to .java sources - especially with Arquillian. See ejb/descriptor/ at .

Maven allows to keep sources and resources in the same dir tree.
TODO: Discuss with devs how they'd like this.


WFLY-576 TS: Discuss package names - vs.
"integration" gives a hint of test's affiliation right from the package name.
Less subdir levels.

Discuss (with QA?) whether they really care.


WFLY-576 TS: Check clustering and compat and spec tests

Stuart Douglas:

1) Clustering tests do not run, and as they require a different arquillian adaptor they must not be in the same maven module as other integration tests.

2) Compat tests do not run correctly. Even though they have a different hibernate version specified in the profile, when attempting to run alongside other tests the hibernate version will be overriden. Also as this modifies a build time dependency test classes need to be re-compiled each run, to make sure they are built against the correct hibernate version.

I'll merge our changes into Stuart's branch since moving poms and scripts is easier than moving Clustering and Compat dirs back to their original location.


WFLY-576 TS: Prepare for merging into master.

Current authoritative branch is at
I keep rebasing this daily to be ready to merge into master.
When all the issues under AS7-2007 are resolved, I'll create a pull request at .


WFLY-576 TS: #!/bin/sh prevents using bash construct

$ ./ clean integration-test -Dtest=*SessionFactoryTestCase*
./ 238: [[: not found
/home/ondra/work/AS7/jboss-as/tools/maven/bin/mvn -s ../tools/maven/conf/settings.xml  clean integration-test -Dtest=*SessionFactoryTestCase*

WFLY-576 clean install -DallTests doesn't work after scripts were split


WFLY-576 TS: Coverage reports

Cobertura coverage jobs in Hudson (unit-test only, see 1st comment):

We can also use Emma:

Or this?

TCK coverage reports:


WFLY-576 Maven plugin to process properties


WFLY-576 TS: Some tests are still in package


WFLY-576 TS: Check -Dtest

Make sure that -Dtest can be used to run a single test (at the
moment, this works, but the module to be used really needs to be specified).


WFLY-576 TS: Check scripts options.

Make sure that all options to and are
being processed correctly (likewise for the Windows versions)


WFLY-576 TS: Implement test reporting.

Make sure test reports are being produced correctly (this may involve
chasing up Paul Gier for the surefire release).


WFLY-576 TS: Check regressions

make sure that the modified testsuite does not introduce any
regressions. This includes tests run agains the default server profile
as well as the preview server profile. (There are currently failures
against the preview server profile in the integration tests)


WFLY-576 TS: Compare sets of tests run in both versions for respective groups

Make sure that the set of tests run by the modified testsuite is the
same as the set of tests run by the unmodified testsuite (this is
because the includes/excludes have been rearranged in some cases,
especially in the integration module).

mkdir -p $DIR/allReports
find testsuite/ -name TEST-* > $DIR/list.txt
for i in `find testsuite/ -name *TestCase*.txt`; do cp $i $DIR/allReports ; done
for i in `find testsuite/ -name TEST-*`;         do cp $i $DIR/allReports ; done

WFLY-576 Tests must fail the build by default.

To override, use: