It looks like MessageContextJAXWS is opening streams to wsdl files (which in this case are local for services provided by this container) and never closing them, causing our machine to run out of file handles if the wsdl is requested many times in rapid succession. We got the following exception:

009-04-28 16:21:53,689 ERROR
[org.jboss.ws.core.jaxws.SOAPFaultHelperJAXWS] SOAP request exception
org.jboss.ws.WSException: Cannot open:
file:/var/lib/jbossas/server/production/data/wsdl/regnum-service-ear-1.1.ear/regnum-service-ejb-1.1.jar/RegNumServic
+eBeanService7046.wsdl
at
org.jboss.ws.core.jaxws.handler.MessageContextJAXWS.setOperationMetaData(MessageContextJAXWS.java:143)
at
org.jboss.ws.core.server.ServiceEndpointInvoker.invoke(ServiceEndpointInvoker.java:177)

We suspected that the nodes were either running out of file handles or
disk space. Turns out its file handles. Looking at the class, line
123:

123 public void setOperationMetaData(OperationMetaData opMetaData)

It does a wsdlSource.openStream() but never closes it.

We watched the file handles grow on this wsdl file and never shrink
during the operation of the script. You can check on this with a
script like this:

lsof -p `pgrep java` | awk '

' | sort | uniq -c | sort -n |
tac | head