Bugs fixed by this PR

• Resolves QuarkusTest: consider removing the test profile support for @Nested tests #45349
• Resolves Broken QuarkusTestExtension #8446
• Resolves Kotlin junit ParameterizedTest with functions as argument, not working anymore (Kotlin 2.0) #42000
• Resolves Regression in 3.13.0.CR1: Lambda expression from custom serializable interface used as @QuarkusTest method parameter fails with ClassNotFoundException #42006
• Resolves @ParameterizedTest with @MethodSource do not work when using @QuarkusTest annotation #44320
• Resolves Mockito fails to mock non-public inner class in continuous testing due to classloading issues #38987
• Resolves ConfigProvider.getConfig().getOptionalValue("..").get() does not work in JUnitExtension any more #46383
• Resolves Ecosystem CI failure NoClassDefFoundError: io/quarkiverse/pact/devmodetest/farm/FarmContractTest quarkiverse/quarkus-pact#272

Bugs created by this PR (doh!)

• Creates Dev Services should be started during startup, not augmentation #45785
• Creates Connection problems to Oracle in tests when another db is used in another resource first #46542

Outstanding issues/breaking changes (input to release notes)

• @TestProfile on @Nested tests are not supported anymore (fixes QuarkusTest: consider removing the test profile support for @Nested tests #45349).
• Version 3 of Maven Surefire and Failsafe is required, and version 2 is unlikely to work.
• Changes the order of some tests; tests should not depend on execution order if they don't set one explicitly, so this shouldn't cause customer failures, but it did cause failures for us (see, for example, Tests in opentelemetry-reactive project are sensitive to execution order #45955 and Improve MetricsTest reliability #45960)
• Dev services start in the discovery phase (Dev Services should be started during startup, not augmentation #45785, we know how to fix this quickly)
• Increased memory footprint running tests. For suites using multiple profiles and resources, more metaspace may be needed.
• Accessing ConfigProvider from a custom JUnit condition results in ServiceConfigurationError · Issue Accessing ConfigProvider from a custom JUnit condition results in ServiceConfigurationError #47081 · workaround is to set a TCCL to this.getClass().getClassLoader()
• Whatever causes [CI] - Artemis + Quarkus main quarkiverse/quarkiverse#55 (comment), diagnosis TBD

What problem is this solving?

We see a lot of problems caused by the fact that we load test classes with the deployment classloader, and then intercept the execution and reload the classes with the runtime classloader. Although the new test is loaded with the runtime classloader, its arguments are still loaded with the system classloader. To work around that we sometimes need to clone the arguments by serializing and de-serializing. This was always brittle and no longer worked at all on Java 17+ (until #40601 fixed that). We also see issues because parts of the test infrastructure see the 'wrong' instance of the class. See, for example, quarkiverse/quarkus-pact#73 and #22611.

We have several feature raised against the JUnit team to allow us more control over classloading. The first of this features was introduced in JUnit 5.10, and allows an interceptor to be registered before any tests are launched. This interceptor can set a thread context classloader, which is then used by JUnit to load tests.

My experiments with this feature were thoroughly disappointing. It turns out, setting a TCCL early in the test lifecycle doesn't really help us, because we overwrite our 'early' TCCL with other TCCLs later in the test lifecycle. The following diagram shows some of the places we set the TCCL.

Source: https://excalidraw.com/#json=HFPHIKx8wv0iiyXgNhAzw,8IlEmPcMRvm9pfCGShdClQ

What if we just used one of the existing interception points to set the 'right' classloader, before tests are loaded? If the tests were loaded with our preferred classloader, we wouldn’t need to intercept the factory. Loading the tests with the runtime classloader needs us to move some of our app initialisation earlier in the lifecycle, but I don't think there's any fundamental barrier to this. (We would have had to do this with a solution based on the new JUnit Launcher Interceptor anyway.)

The logic for starting Quarkus needs to be in the test discovery phase, rather than in the extension. This allows us to create the runtime classloader before the test is loaded. The JUnitTest runner already knows about the Quarkus Extension, so it’s only a small extra bit of knowledge to do some of the startup actions.

This only gets us part of the way, though. @stuartwdouglas raised the point that if we have to set only a single classloader, that's not very flexible, because we have a runtime classloader for each test profile. A Quarkus test run doesn't just use one classloader, it uses several. Every resource/unique profile triggers an app relaunch, which means a new classloader. What I've done to handle this is create a FacadeClassLoader. It takes the classloading requests, and then either routes them on to the quarkus application (for vanilla @QuarkusTests), or, if there's a profile/resource, it makes a new app + classloader and sends the request to that.

What we used to before was load a throwaway copy of the the test, pass it to JUnit discovery, let JUnit launch it, and then intercept the execution, figure out what profiles+resources the test declares, create a quarkus app with that information, start the quarkus app, reload the test with the runtime classloader of the quarkus app (and clone its parameters), and execute the test.

The new model is load a throwaway copy of the the test, figure out what profiles+resources the test declares, create a quarkus app with that information, reload the test with the runtime classloader of the quarkus app, pass the ‘right’ class to JUnit discovery, let JUnit launch it, and then intercept the execution, start the quarkus app, and execute the test.

One fundamental limitation of "load tests with the classloader used to execute them" is that a single test cannot run with multiple classloaders, which means it cannot support multiple profiles. We know some people do use this feature, but we also know there have been suggestions that we drop support for it, since it is complex to support (#45349). There is an easy workaround, which is to use one test per profile.

Thoughts on serialization and cloning

A big initial goal of this PR was to get rid of the xstream serialization, since it didn't work on Java 17+. #40601 fixes this issue by switching to use the JBoss marshaller for serialization. Does that mean this work item isn't needed any more? No, although it does mean its benefits are smaller. Here's why it's still useful:

• Even with the JBoss serializer, higher-level test infrastructure (such as @TestTemplate) does not see Quarkus bytecode transformations done by extensions
• Although the JBoss serializer works a lot better than xstream with the Java 17 access restrictions (as in, it works), serialization may continue to be a challenge going forward. See https://bugs.openjdk.org/browse/JDK-8164908 for some context. Most serializers use sun.misc.unsafe, but unsafe is shrinking. It seems certain the JDK team will have to come up with some solution and API to open up access for serializers, but the final design could have security implications (perhaps opening up access in a blanket way), or performance implications (reflection fun), or user experience implications (a need to manually set flags such as --enable-serialization?). If we can avoid serialization, we avoid all that.