13 Replies Latest reply on May 8, 2012 6:00 AM by sven.woerz

    JBoss 4.2.3 JMS java.net.SocketException: Too many open files

    marcusdidiusfalco

      Hello,

       

      on an application which relies heavily on JMS I get

      2012-03-22 16:36:30,496 11904541 WARN [org.jboss.mq.il.uil2.UILServerILService] (UILServerILService Accept Thread:) Failed to setup client connection

      1. java.net.SocketException: Too many open files

                  at java.net.PlainSocketImpl.socketAccept(Native Method)

                  at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)

                  at java.net.ServerSocket.implAccept(ServerSocket.java:462)

                  at java.net.ServerSocket.accept(ServerSocket.java:430)

                  at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:159)

                  at java.lang.Thread.run(Thread.java:662)

      after some time.

      This log entry is repeated forever and the application server must be stopped.


      I have already looked through the code and it seems that all JMS connections, sessions and senders are close in finally blocks after use.

       

      Using YourKit for Profiling I get

       

      openjmssockets1.jpg

      openjmssockets2.jpg

       

      I would be very greatful for any suggestions how to fix this or how to further analyze this problem.

       

      Thanks,

       

      Hans

        • 1. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
          wdfink

          You should check your system whether the user has enough file-descriptors  (see ulimit).

          A long open socket might not point directly to a leak because the connections are pooled and a close() might only signal that the connection is not longer used and go back to the pool (the socket is still keept open).

           

          As the message is the problem can be somewhere and not because of open sockets, it might be open files db-connections or the amount of all toghether.

          • 2. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
            marcusdidiusfalco

            We have increase the number of file-descriptors.

            No we get

            2012-04-02 10:54:36,458 DEBUG [org.jboss.mq.il.uil2.SocketManager] Exiting on unexpected error in read task

            java.lang.OutOfMemoryError: unable to create new native thread

                at java.lang.Thread.$$YJP$$start0(Native Method)

                at java.lang.Thread.start0(Thread.java)

                at java.lang.Thread.start(Thread.java:640)

                at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.addThread(PooledExecutor.java:521)

                at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:893)

                at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:359)

                at java.lang.Thread.run(Thread.java:662)

             

            In the profiling tool I see that UIL2.SocketManager.WriteTask Threads steadily accumulate.

             

            I have set

            <category name="org.jboss.mq">

                  <priority value="TRACE"/>

               </category>

            and get a lot of

            2012-04-02 09:45:08,561 TRACE [org.jboss.mq.il.uil2.SocketManager] start called

            java.lang.Exception: Start stack trace

                at org.jboss.mq.il.uil2.SocketManager.start(SocketManager.java:112)

                at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:171)

                at java.lang.Thread.run(Thread.java:662)

            Is this normal? Or does it point to my problem?

             

            Thanks,

             

            Hans

            • 3. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
              peterj

              Which OS are you using? Is it 32-bit or 64-bit? If 64-bit, are you using a 32-bit or 64-bit JVM?

               

              The error "java.lang.OutOfMemoryError: unable to create new native thread" means that there is not enough memory available in your address space to create another thread. Depending on your OS and JVM, a new thread might use anywhere from 512KB to 1MB of memory. You can adjust the thread memory usage via the -XX:ThreadStackSize=n JVM option. You can also decrease your heap size or permgen size to free up more address space for your threads (that is, if you have overprovisioned those areas).

               

              As far as logging the MQ, I am not familiar enough with JBossMQ to know what it does when you turn logging to TRACE (but you could grab the source for org.jboss.mq.il.uil2.SocketManage and find out for yourself), but obviously you don't want that. Put the priority back to DEBUG or higher.

              • 4. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                marcusdidiusfalco

                I have already set the ThreadStackSize to 128 kb. This only postpones the problem somehow. The system only crashes a few minutes later.

                I have to solve the problem of accumulating threads.

                But thanks for your reply.

                 

                Hans

                • 5. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                  sven.woerz

                  Hi Peter,

                   

                  I stumbled upon this discussion when googling for a problem we face on one of our environments.

                  We are currently moving a JBoss 4.2.2 Server from a 32bit Win2008 to a 64bit Win2088 system. Setup of the system is not our task. We are only responsible for JBoss.

                  On teh nuew system we get "java.lang.OutOfMemoryError: unable to create new native thread" errors. Especially intresting to me is your question "Is it 32-bit or 64-bit? If 64-bit, are you using a 32-bit or 64-bit JVM?" cause this is exactly the setup we have on the new system.

                  Meanwhile we are running on a 64bit JVM (1.6) and it seems to be fine now. We did not change any other JVM settings.

                  So, do you have an idea why it works on a 64bit JVM? Is there simply more address space?

                   

                  Regards

                  Sven

                  • 6. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                    wdfink

                    Yes,

                    in detail, a 32bit VM can not address more than ~4GB on Windows (other OS different).

                    This mean JVM code, threadstack, PermGen and Heap (young and old gen)

                    If you set now 3GB for heap and 256m for Permgen there is not a real big amount of memory for JVM code and threadstack.

                    If many threads created you will get a OOM.

                     

                    With a 64bitVM this problem does not exists (or even far far away).

                    You might come to the same situation if you (and others) consume all memory of the system and the JVM can not allocate more.

                    • 7. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                      marcusdidiusfalco

                      We have found the problem (or rather our partners)

                      Actually 4.2.2. is used.

                      And in the client which sends JMS messages to our application instances of the InitialContext were not closed.

                      I cannot show the exact code because we don't have access to the code of the client application.

                      • 8. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                        sven.woerz

                        Hans-Michael Rupp wrote:

                         

                        We have found the problem (or rather our partners)

                        Actually 4.2.2. is used.

                        And in the client which sends JMS messages to our application instances of the InitialContext were not closed.

                        I cannot show the exact code because we don't have access to the code of the client application.

                         

                        Thanks for this hint, Hans-Michael. I'll check our code to see if InitialContext is handled proprely!

                        • 9. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                          sven.woerz

                          Wolf-Dieter Fink wrote:

                           

                          Yes,

                          in detail, a 32bit VM can not address more than ~4GB on Windows (other OS different).

                          This mean JVM code, threadstack, PermGen and Heap (young and old gen)

                          If you set now 3GB for heap and 256m for Permgen there is not a real big amount of memory for JVM code and threadstack.

                          If many threads created you will get a OOM.

                           

                          With a 64bitVM this problem does not exists (or even far far away).

                          You might come to the same situation if you (and others) consume all memory of the system and the JVM can not allocate more.

                           

                          Thanks for your reply, Wolf-Dieter!

                          Ok, on a 32bit JVM you cannot address more than 4GB. But - and sorry for bothering - I still do not understand why on a 64bit JVM we are fine as we did not change JVM settings when moving from 32bit JVM to 64bit JVM. The -Xmx parameter is set to 1.5GB in both JVMs. So JVM has max 1.5GB on 32bit and 64bit, or am I wrong?

                          • 10. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                            raneves

                            if you using SO linux, set the fallow command: ulimit -n 65536.

                            this solved you problem.

                            on windows, see the regedit configurations for this parameters.

                            • 11. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                              wdfink

                              As im wrote (ok it's implicit)

                               

                              -Xmx control the heap size

                              this will be only a part of the memory consumption (you will find other threads discussing that if you search for it)

                               

                              So the memory is in total:

                              JVM code | Heap | PermGen | ThreadStack

                              and other (minor) stuff

                              • 12. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                                peterj

                                Here is a description of all of the memory consumed by a JVM:

                                http://www.coderanch.com/t/111262/Linux-UNIX/read-memory-usage-process-running#560894

                                 

                                When running a 32-bit JVM on Windows, you have at most 2GB available for all of the things mentioned in that post. When running a 64-bit JVM, you have a much larger adress space (terabytes, I think) so there is more room to hold all of the things mentioned in that post. Therefore when a new thread was created it no longer bumped up against the 2GB limit.

                                • 13. Re: JBoss 4.2.3 JMS java.net.SocketException: Too many open files
                                  sven.woerz

                                  Ok. Now it becoms quite clear how things are. Thank you you all for helping me to understand what's going on.