12 Replies Latest reply on Sep 15, 2010 1:08 PM by leonko

    java.net.SocketException: Too many open files on Red Hat lin

    kiranhk

      Dear All,
      We are using JBoss 4.2.2GA on a RHEL with Apache as Webserver.
      We keep running out of open files whenever after just hrs since the JBoss server is started.

      We also increased the ulimit max open files number ffrom 1024 to 8192 but still we seem to be maxing out this number after a few hrs or when doing load testing. The application that is running on the JBoss server makes HTTP connections to other systems using Apache HTTPClient and the code is releasing the connections using the HTTPClient.

      can somebody pls help with this issue.

      JBoss 4.2.2 GA, RHELinux -- Quad Code 8GBRAM,

      There has been some internal JBoss errors which started occurring at a rapid pace and the server.log went up from 900kb to 350mb in a span of 30-60mins. The following are the errors in the server.log. this error is the only one that is logged in this whole 350mb server.log. there has been no other exception/error logged



      2009-06-22 16:20:39,512 ERROR [org.apache.tomcat.util.net.JIoEndpoint] Socket accept failed

      java.net.SocketException: Too many open files

      at java.net.PlainSocketImpl.socketAccept(Native Method)

      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

      at java.net.ServerSocket.implAccept(ServerSocket.java:453)

      at java.net.ServerSocket.accept(ServerSocket.java:421)

      at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)

      at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:309)

      at java.lang.Thread.run(Thread.java:619)

      2009-06-22 16:20:39,512 ERROR [org.apache.tomcat.util.net.JIoEndpoint] Socket accept failed

      java.net.SocketException: Too many open files

      at java.net.PlainSocketImpl.socketAccept(Native Method)

      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

      at java.net.ServerSocket.implAccept(ServerSocket.java:453)

      at java.net.ServerSocket.accept(ServerSocket.java:421)

      at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)

      at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:309)

      at java.lang.Thread.run(Thread.java:619)



      thanks in advance

      Cheers
      Kiran

        • 1. Re: java.net.SocketException: Too many open files on Red Hat
          vons1234

          try to see what file handles are open using following command

          lsof -p PID


          I did face same problem while using Jboss 4.3 on SunOS.
          We were using jboss webservices module and while load testing it was throwing same exception for WSDL

          java.net.SocketException: Too many open files


          I believe there is a patch available for this. We used that patch and that resolved the problem.

          • 2. Re: java.net.SocketException: Too many open files on Red Hat
            kiranhk

            Yes. we are also hitting a Webservices system and we post xml over http instead of WSDL.

            There are 2 things that i have done to overcome this problem right now.

            1) turned off the deployment scanner in the jboss-service.xml

            2) added a executor tag for the server.xml so that it will close the TCP connections after there is idle time.

            jboss-4.2.2.GA/server/ottserver/deploy/jboss-web.deployer/server.xml

            <Executor name="appThreadpool" namePrefix="activeThread-" maxThreads="1000" minSpareThreads="30" maxIdleTime="30



            for now we seem to have jumped over this max open files problem.

            But there is a new problem which has cropped up now.

            After running load tests for 3/4 hours using JMeter at the end of the 4th hour we saw a sudden spike in the jboss process where its hitting 100% CPU and this didn’t come back down after we stopped the requests from jmeter to the server. we had to kill the server to revive it.

            any ideas as to what seems to be the problem.


            also can you pls give me more info abt the patch.

            thanks in advance.

            Cheers
            Kiran

            • 3. Re: java.net.SocketException: Too many open files on Red Hat
              vons1234

              Do you really need maxThreads="1000" in your tomcat connector??

              You should not set it that high, that might be reason your CPU is spiking when you do the load test, it will keep creating new threads for your incoming request.

              Even if you are expecting 500 concurrent request, i think 50 threads should be able to handle that.
              Also you can use acceptCount which is a queue for all your incoming connection requests when all possible request processing threads are in use.

              Refer to following URL for the bug which i was talking about
              https://jira.jboss.org/jira/browse/JBWS-2325

              • 4. Re: java.net.SocketException: Too many open files on Red Hat
                leonko
                You are resolved your problem?
                • 5. Re: java.net.SocketException: Too many open files on Red Hat
                  kiranhk

                  Yes.. I was able to resolve the problem. the executor was not mentioned which will close the connections/files that are open

                   

                   

                  Navigate to "jboss-5.1.0.GA/server/ottserver/deploy/jbossweb.sar" and edit server.xml as mentioned below.

                   

                  Look for following xml snippet.

                   

                  <Connector port="9090" address="${jboss.bind.address}" maxThreads="250" maxHttpHeaderSize="8192" emptySessionPath="true" protocol="HTTP/1.1" enableLookups="false" redirectPort="8443" acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" />

                   

                  Replace this snippet with below one.

                   

                  <Executor name="appThreadpool" namePrefix="activeThread-" maxThreads="150" minSpareThreads="3" maxIdleTime="30000" />

                      <Connector executor="appThreadpool" port="9090" address="${jboss.bind.address}"          maxThreads="150" maxHttpHeaderSize="8192"

                           minSpareThreads="3" maxSpareThreads="20"          emptySessionPath="true" protocol="HTTP/1.1"

                           enableLookups="false" redirectPort="8443" acceptCount="100"      connectionTimeout="120000" disableUploadTimeout="true" />

                   

                   

                  Look for following xml snippet.

                   

                  <Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3"  emptySessionPath="true" enableLookups="false" redirectPort="8443" />

                   

                  Replace this snippet with below one.

                   

                  <Connector executor="appThreadpool" port="8009" address="${jboss.bind.address}" protocol="AJP/1.3" emptySessionPath="true" enableLookups="false" redirectPort="8443" />

                   

                  hope that helps.

                   

                  thanks

                  Kiran

                  • 6. Re: java.net.SocketException: Too many open files on Red Hat
                    leonko

                    Yes. On jboss 4.2.x I resolve this problem  too.

                    I just compile new jbossweb.jar from svn (2.0.x). It have bugfix.

                    At now I have CLOSE_WAIT = 0 and happy))

                    • 7. Re: java.net.SocketException: Too many open files on Red Hat
                      sarran2003

                      Hi Kiran,

                       

                      i have tried the same way which you explained. but seems that i am getting again same exception.

                       

                      is there any new patch is available in Jboss for this issue?

                       

                      Please let me know how to resolve this issue...

                       

                      Thanks in Advance.

                       

                      Regards,

                      Silambarasan

                      • 8. Re: java.net.SocketException: Too many open files on Red Hat
                        leonko

                        compile this: http://anonsvn.jboss.org/repos/jbossweb/branches/2.0.x/

                        or use this patch https://jira.jboss.org/secure/attachment/12316577/patch.txt with JBOSSWEB_2_0_0.

                        IF  patch work you seen close_wait arount zero:

                        netstat -a | grep CLOSE_WAIT .

                        • 9. Re: java.net.SocketException: Too many open files on Red Hat
                          sarran2003

                          Thanks for your reply....

                           

                          Sorry for asking this...do we have any binary available for the latest with fix for jbossweb.2.0.x?

                           

                          it would be great if i get jar instead of source code.

                           

                          here is the exception we are getting in our jboss. but finally all the transcations getting broken and the server got hang-up...

                           

                          04:55:34,751 ERROR [org.jboss.naming.Naming] Naming accept handler stopping
                          java.net.SocketException: Too many open files
                              at java.net.PlainSocketImpl.socketAccept(Native Method)
                              at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

                           

                           

                          java.net.SocketException: Broken pipe
                              at java.net.SocketOutputStream.socketWrite0(Native Method)

                           

                           

                          I believe the main issue is with the "too many open files", probably this could be solved with the latest jbossweb...

                           

                          Cheers,

                          Silambarasan

                          • 10. Re: java.net.SocketException: Too many open files on Red Hat
                            leonko

                            I make myself binary and test it for jboss 4.2.1 and jboss 4.2.3.

                             

                             

                            just replace jbossweb.jar in deploy/jboss-web.deployer/ folder

                            • 11. Re: java.net.SocketException: Too many open files on Red Hat
                              sgtw_speed

                              We're having the same problem on JBoss 4.2.1 GA - we compiled jbossweb 2.0.0.x and replaced the jbossweb.jar file under jboss-web.deployer as suggested, but getting the following error on startup.  Are you sure no other change is needed?

                               

                              org.jboss.deployment.DeploymentException: - nested throwable: (java.lang.reflect.UndeclaredThrowableException)

                               

                              Caused by: java.lang.NoClassDefFoundError: org/apache/catalina/Lifecycle

                               

                              It's odd, because Lifecycle is definitely found within the jbossweb.jar file.

                              • 12. Re: java.net.SocketException: Too many open files on Red Hat
                                leonko

                                Shane Weaver

                                No, I do not have any problems like this