Fixes #51269.
The current problem in CI (and for users) was caused by #50092, which bumped surefire from 3.5.3 to 3.5.4. We didn't catch it because (a) the tests for maven reruns wasn't merged and (b) they didn't lock the version of surefire to the one used by the rest of the build, so they wouldn't have caught the regression. I've confirmed that when I do add the version, the tests start failing, and this PR gets them passing again.
The root cause was that apache/maven-surefire#863 changed how surefire handled LauncherSessions so that one session was created for the whole process, rather than one for each test run. That meant that on runs after the first one, the rerun tests would still be loaded with the 'old' classloader and attached to the previous app. I worried this would be hard to fix the hooking the end of the test execution allows us to close everything at the right point.