We have written client/server application. we have deployed the server jars into JBoss Application server. The Jboss service running successfully in many client/server machine. But Jboss service has been hanged after 5-6 days recursively. then we need to restart the service to come out of this problem again and again. The following error message that we got from the server log.
Note : We are using wrapper.exe to start and Stop the services of JBoss AS.
2011-02-24 00:56:17,095 ERROR [STDERR] Exception in thread "RequestController-5"
2011-02-24 00:56:17,096 ERROR [STDERR] java.lang.OutOfMemoryError: unable to create new native thread
2011-02-24 00:56:17,096 ERROR [STDERR] at java.lang.Thread.start0(Native Method)
2011-02-24 00:56:17,096 ERROR [STDERR] at java.lang.Thread.start(Unknown Source)
2011-02-24 00:56:17,096 ERROR [STDERR] at org.jacorb.poa.RPPoolManager.addProcessor(Unknown Source)
2011-02-24 00:56:17,096 ERROR [STDERR] at org.jacorb.poa.RPPoolManager.getProcessor(Unknown Source)
2011-02-24 00:56:17,096 ERROR [STDERR] at org.jacorb.poa.RequestController.processRequest(Unknown Source)
2011-02-24 00:56:17,096 ERROR [STDERR] at org.jacorb.poa.RequestController.run(Unknown Source)
2011-02-24 09:46:48,863 ERROR [STDERR] Exception in thread "Listener:40584"
2011-02-24 09:46:48,863 ERROR [STDERR] java.lang.OutOfMemoryError: unable to create new native thread
2011-02-24 09:46:48,863 ERROR [STDERR] at java.lang.Thread.start0(Native Method)
2011-02-24 09:46:48,864 ERROR [STDERR] at java.lang.Thread.start(Unknown Source)
2011-02-24 09:46:48,864 ERROR [STDERR] at com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:135)
We have also mentioned the run.conf file details
rem JVM memory allocation pool parameters. Modify as appropriate.
set JAVA_OPTS=%JAVA_OPTS% -Xms128m -Xmx512m
rem With Sun JVMs reduce the RMI GCs to once per hour
set JAVA_OPTS=%JAVA_OPTS% -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000
rem JPDA options. Uncomment and modify as appropriate to enable remote debugging.
rem set JAVA_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y %JAVA_OPTS%
java.lang.OutOfMemoryError: unable to create new native thread
Unfortunately the OOM Exception is a bit confusing, see Java API for this
It mean that the JVM is out of system resource, in your case could not start a new thread.
There are two reasons:
- restriction to thread resources or filedescriptor
- could not get memory for thread stack (this memory is not included in your -Xmx512m, it will be allocated extra)
1) As I said -Xmx does not change the threadstacksize
2) It's more difficult ...
You should not start threads by yourselve running JEE under AS control!
If you don't do this your code should be ok.
Different might be the MBean handling.
Do you check whether you have enough memory system wide? What OS you use?
1) I have checked the memory system: It is Ok.
2) OS is Windows server 2008 (SP2)
Please find the memory information below:
Total physical memory: 4086 MB
Available Physical memory: 976 MB
Page File Max size: 8428 MB
Available size: 4137 MB
Page file In use: 4291 MB
Do we need to check any system related things? because this problem i am able to see in some particular machines and in some machines it is working fine and all the configurations common for both.
Waiting for the reply,
Thank's in advance.
Many thanks for given some useful links and ideas.
But i have a question.
Is it possible to set Xss or XX:ThreadStackSize in run.bat file?
By default ThreadstackSize is 1MB. This could be one of the problem.
Note : We also restarted service each tiimes. Its always solved the error for certain time - around one week.
As it's explain on this blog post http://www.mastertheboss.com/jboss-monitoring/how-to-solve-javalangoutofmemoryerror-unable-to-create-new-native-thread
You have to align JAVA_OPTS -Xss with the max-user-processes limit configuration for the system user.
You can have this information with ulimit command :
open files (-n) 4096
max user processes (-u) 1024