We had OutOfMemoryError issue with JBoss Messaging 1.4.7 test suite on RHEL 6 platform.

Problem occurred during the fail-over tests where test suite starts 6 servers on the local machine.

Exception:
java.lang.OutOfMemoryError: unable to create new native thread.

Test suite was unable to start new thread. Server with RHEL 6 was clean installation (not configured or tuned) and problem was caused by OS configuration option ,,max user processes". Default value for this configuration option is 1024 and this is not enough for JBoss Messaging test-suite.

Default value:
ulimit -a
max user processes (-u) 1024

We have increased this configuration value to 30754 and issue is gone:
ulimit -a
max user processes (-u) 30754

If you have same problem, you can change this configuration by adding following line to /etc/security/limits.conf.
[your_user_name] soft nproc 30754