After debugging this test failure using a Windows agent, I found that as I had initially suspected, KeystoreRealmTestCase fails when the test directory does happen to exist and has a file that's been left around by a previous test (the file that is left around is target\test-classes\ca\jks\ocsp-responder.keystore). WFLY-14487 tried to fix this by attempting to delete the test directory if it does happen to exist. However, from debugging with the Windows agent, I can now see that the deletion actually fails with an IOException that says "Unable to delete file". This seems to be due to a Windows related bug where the file can't be deleted because it is being used by another process. The test that leaves around the problematic file does attempt to delete it as part of its cleanup. However, the deletion is not always successful.

To avoid this failure, we should use a unique name for the test directory for KeystoreRealmTestCase so other tests do not affect this one.