1. They're running JBoss as a Windows service so they used service.bat to install the service. (they were running Tomcat as a service before)
2. The run.bat calls the run.conf.bat to set the startup variables.
3. run.conf.bat sets JAVA_OPTS with memory settings, but only if JAVA_OPTS is empty. If there's anything at all in that variable, the whole script is skipped.
4. Which is bad, because the service.bat sets a user variable of JAVA_OPTS to -Xrs.
So, when you start the server as a service with anything other than a trivial application, you bomb out with OutOfMemoryErrors. Our solution was to set memory on JAVA_OPTS in the run.conf.bat regardless. But that's not a scalable solution because other users might set JAVA_OPTS per environment and already have memory settings set up. Works for here though, so at least Trilogy is OK for now.