-
1. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
wdfink Mar 28, 2012 2:29 AM (in response to marcusdidiusfalco)You should check your system whether the user has enough file-descriptors (see ulimit).
A long open socket might not point directly to a leak because the connections are pooled and a close() might only signal that the connection is not longer used and go back to the pool (the socket is still keept open).
As the message is the problem can be somewhere and not because of open sockets, it might be open files db-connections or the amount of all toghether.
-
2. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
marcusdidiusfalco Apr 2, 2012 6:49 AM (in response to wdfink)We have increase the number of file-descriptors.
No we get
2012-04-02 10:54:36,458 DEBUG [org.jboss.mq.il.uil2.SocketManager] Exiting on unexpected error in read task
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.$$YJP$$start0(Native Method)
at java.lang.Thread.start0(Thread.java)
at java.lang.Thread.start(Thread.java:640)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.addThread(PooledExecutor.java:521)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:893)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:359)
at java.lang.Thread.run(Thread.java:662)
In the profiling tool I see that UIL2.SocketManager.WriteTask Threads steadily accumulate.
I have set
<category name="org.jboss.mq">
<priority value="TRACE"/>
</category>
and get a lot of
2012-04-02 09:45:08,561 TRACE [org.jboss.mq.il.uil2.SocketManager] start called
java.lang.Exception: Start stack trace
at org.jboss.mq.il.uil2.SocketManager.start(SocketManager.java:112)
at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:171)
at java.lang.Thread.run(Thread.java:662)
Is this normal? Or does it point to my problem?
Thanks,
Hans
-
3. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
peterj Apr 2, 2012 10:03 AM (in response to marcusdidiusfalco)Which OS are you using? Is it 32-bit or 64-bit? If 64-bit, are you using a 32-bit or 64-bit JVM?
The error "java.lang.OutOfMemoryError: unable to create new native thread" means that there is not enough memory available in your address space to create another thread. Depending on your OS and JVM, a new thread might use anywhere from 512KB to 1MB of memory. You can adjust the thread memory usage via the -XX:ThreadStackSize=n JVM option. You can also decrease your heap size or permgen size to free up more address space for your threads (that is, if you have overprovisioned those areas).
As far as logging the MQ, I am not familiar enough with JBossMQ to know what it does when you turn logging to TRACE (but you could grab the source for org.jboss.mq.il.uil2.SocketManage and find out for yourself), but obviously you don't want that. Put the priority back to DEBUG or higher.
-
4. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
marcusdidiusfalco Apr 2, 2012 10:51 AM (in response to peterj)I have already set the ThreadStackSize to 128 kb. This only postpones the problem somehow. The system only crashes a few minutes later.
I have to solve the problem of accumulating threads.
But thanks for your reply.
Hans
-
5. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
sven.woerz May 4, 2012 4:41 AM (in response to peterj)Hi Peter,
I stumbled upon this discussion when googling for a problem we face on one of our environments.
We are currently moving a JBoss 4.2.2 Server from a 32bit Win2008 to a 64bit Win2088 system. Setup of the system is not our task. We are only responsible for JBoss.
On teh nuew system we get "java.lang.OutOfMemoryError: unable to create new native thread" errors. Especially intresting to me is your question "Is it 32-bit or 64-bit? If 64-bit, are you using a 32-bit or 64-bit JVM?" cause this is exactly the setup we have on the new system.
Meanwhile we are running on a 64bit JVM (1.6) and it seems to be fine now. We did not change any other JVM settings.
So, do you have an idea why it works on a 64bit JVM? Is there simply more address space?
Regards
Sven
-
6. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
wdfink May 4, 2012 5:31 AM (in response to sven.woerz)Yes,
in detail, a 32bit VM can not address more than ~4GB on Windows (other OS different).
This mean JVM code, threadstack, PermGen and Heap (young and old gen)
If you set now 3GB for heap and 256m for Permgen there is not a real big amount of memory for JVM code and threadstack.
If many threads created you will get a OOM.
With a 64bitVM this problem does not exists (or even far far away).
You might come to the same situation if you (and others) consume all memory of the system and the JVM can not allocate more.
-
7. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
marcusdidiusfalco May 4, 2012 7:54 AM (in response to marcusdidiusfalco)We have found the problem (or rather our partners)
Actually 4.2.2. is used.
And in the client which sends JMS messages to our application instances of the InitialContext were not closed.
I cannot show the exact code because we don't have access to the code of the client application.
-
8. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
sven.woerz May 4, 2012 8:26 AM (in response to marcusdidiusfalco)Hans-Michael Rupp wrote:
We have found the problem (or rather our partners)
Actually 4.2.2. is used.
And in the client which sends JMS messages to our application instances of the InitialContext were not closed.
I cannot show the exact code because we don't have access to the code of the client application.
Thanks for this hint, Hans-Michael. I'll check our code to see if InitialContext is handled proprely!
-
9. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
sven.woerz May 4, 2012 8:46 AM (in response to wdfink)Wolf-Dieter Fink wrote:
Yes,
in detail, a 32bit VM can not address more than ~4GB on Windows (other OS different).
This mean JVM code, threadstack, PermGen and Heap (young and old gen)
If you set now 3GB for heap and 256m for Permgen there is not a real big amount of memory for JVM code and threadstack.
If many threads created you will get a OOM.
With a 64bitVM this problem does not exists (or even far far away).
You might come to the same situation if you (and others) consume all memory of the system and the JVM can not allocate more.
Thanks for your reply, Wolf-Dieter!
Ok, on a 32bit JVM you cannot address more than 4GB. But - and sorry for bothering - I still do not understand why on a 64bit JVM we are fine as we did not change JVM settings when moving from 32bit JVM to 64bit JVM. The -Xmx parameter is set to 1.5GB in both JVMs. So JVM has max 1.5GB on 32bit and 64bit, or am I wrong?
-
10. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
raneves May 4, 2012 9:08 AM (in response to marcusdidiusfalco)if you using SO linux, set the fallow command: ulimit -n 65536.
this solved you problem.
on windows, see the regedit configurations for this parameters.
-
11. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
wdfink May 4, 2012 9:54 AM (in response to sven.woerz)As im wrote (ok it's implicit)
-Xmx control the heap size
this will be only a part of the memory consumption (you will find other threads discussing that if you search for it)
So the memory is in total:
JVM code | Heap | PermGen | ThreadStack
and other (minor) stuff
-
12. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
peterj May 4, 2012 10:18 AM (in response to wdfink)Here is a description of all of the memory consumed by a JVM:
http://www.coderanch.com/t/111262/Linux-UNIX/read-memory-usage-process-running#560894
When running a 32-bit JVM on Windows, you have at most 2GB available for all of the things mentioned in that post. When running a 64-bit JVM, you have a much larger adress space (terabytes, I think) so there is more room to hold all of the things mentioned in that post. Therefore when a new thread was created it no longer bumped up against the 2GB limit.
-
13. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
sven.woerz May 8, 2012 6:00 AM (in response to peterj)Ok. Now it becoms quite clear how things are. Thank you you all for helping me to understand what's going on.