When the tests-compatibility target is run on Windows, the JVM gets out of permgen memory.

The target is run in a forked junit with this parameter:
<jvmarg value="-XX:MaxPermSize=512m"/>

In EAP 4.3 this is set to 768m.

The solution is to increase it to 768m even for EAP 5.