1 2 Previous Next 22 Replies Latest reply on Jan 28, 2009 7:33 AM by immortalex

    Socket Read Timeout on stressing calls for validating produc

    vink

      Hello,

      I'm working on JBoss 4.2.2.GA with JBoss Remoting 2.4.0.SP1.
      I've a stress test in Java which places call on Stateless EJB. After some minutes of stress test, I'm encountered with following exception. Please suggest what configuration changes do I make in order to tune it to my 24x7 production environment.

      Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
      at $Proxy0.listSources(Unknown Source)
      at MarshalTest.main(MarshalTest.java:44)
      Caused by: org.jboss.remoting.InvocationFailureException: Socket timed out. Waited 5000 milliseconds for response while calling on InvokerLocator [socket://NOICLT13407:3873/?clientMaxPoolSize=50&invokerDestructionDelay=5000&timeout=5000]; nested exception is:
      java.net.SocketTimeoutException: Read timed out
      at org.jboss.remoting.transport.socket.SocketClientInvoker.handleException(SocketClientInvoker.java:130)
      at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:886)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:160)
      at org.jboss.remoting.Client.invoke(Client.java:1708)
      at org.jboss.remoting.Client.invoke(Client.java:612)
      at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:107)
      ... 2 more
      Caused by: java.net.SocketTimeoutException: Read timed out
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(Unknown Source)
      at java.io.BufferedInputStream.fill(Unknown Source)
      at java.io.BufferedInputStream.read(Unknown Source)
      at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.refill(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.read(Unknown Source)
      at java.io.ObjectInputStream.read(Unknown Source)
      at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1258)
      at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:833)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:160)
      at org.jboss.remoting.Client.invoke(Client.java:1708)
      at org.jboss.remoting.Client.invoke(Client.java:612)
      at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:107)
      at $Proxy0.listSources(Unknown Source)
      at MarshalTest.main(MarshalTest.java:44)
      at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:74)
      ... 10 more

        • 1. Re: Socket Read Timeout on stressing calls for validating pr
          vink

          On increasing timeout value of ejb3 connector, this exception is not coming. But, on running my stress tests, my application is throwing java.lang.OutOfMemoryError: unable to create new native thread

          Is there any place, where we can specify number of sockets, which can be closed by our application.

          If I'm running only 4 clients on my machine, why it is complaining about sockets. Why it doesn't reuse the same socket?

          • 2. Re: Socket Read Timeout on stressing calls for validating pr
            ron_sigal

             

            "vink" wrote:

            But, on running my stress tests, my application is throwing java.lang.OutOfMemoryError: unable to create new native thread

            Is there any place, where we can specify number of sockets, which can be closed by our application.


            Why do you think the OOM problem is related to the number of sockets? Is it an org.jboss.remoting.transport.socket.ServerThread that can't be created?

            In any case, you can specify the number of open sockets on the server with the parameter "maxPoolSize": each ServerThread in the thread pool manages a single socket.

            "vink" wrote:

            If I'm running only 4 clients on my machine, why it is complaining about sockets. Why it doesn't reuse the same socket?


            Again, why do you think it's complaining about sockets?

            • 3. Re: Socket Read Timeout on stressing calls for validating pr
              vink

              I've commented this, becqause of following behaviour;

              - On running my client which is requesting SLSB, the netstat tool is showing several sockets going in TIME_WAIT state. And, number of such sockets increases as time passes by & it finally results into the following exception;

              org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://NOICLT13407:3873/]
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:771)
              at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:160)
              at org.jboss.remoting.Client.invoke(Client.java:1708)
              at org.jboss.remoting.Client.invoke(Client.java:612)
              at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
              at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
              at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
              at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
              at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:107)
              at $Proxy0.getSourcesInfo(Unknown Source)
              at com.barco.cmstest.dependency.MarshalTest.call(MarshalTest.java:82)
              at com.barco.cmstest.dependency.MarshalTest.main(MarshalTest.java:51)
              Caused by: java.net.BindException: Address already in use: connect
              at java.net.PlainSocketImpl.socketConnect(Native Method)
              at java.net.PlainSocketImpl.doConnect(Unknown Source)
              at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
              at java.net.PlainSocketImpl.connect(Unknown Source)
              at java.net.SocksSocketImpl.connect(Unknown Source)
              at java.net.Socket.connect(Unknown Source)
              at org.jboss.remoting.util.SecurityUtility.connect(SecurityUtility.java:969)
              at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:203)
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1084)
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:757)
              at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:160)
              at org.jboss.remoting.Client.invoke(Client.java:1708)
              at org.jboss.remoting.Client.invoke(Client.java:612)
              at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
              at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
              at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
              at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
              at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:107)
              at $Proxy0.getSourcesInfo(Unknown Source)
              at com.barco.cmstest.dependency.MarshalTest.call(MarshalTest.java:82)
              at com.barco.cmstest.dependency.MarshalTest.main(MarshalTest.java:51)
              at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:74)
              ... 11 more



              N:\>netstat -a | findstr 3873
              TCP noiclt13407:1472 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:1782 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:1790 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2055 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2219 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2256 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2289 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2296 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2336 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2351 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2373 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2375 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2376 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2389 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2397 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2420 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2477 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2479 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2522 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2528 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2529 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2551 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2573 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2627 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2628 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2629 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2630 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2632 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2669 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2691 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2748 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2749 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2750 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2764 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2768 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2772 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2775 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2785 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2823 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2824 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2830 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2842 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2920 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2941 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2942 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:2988 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3035 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3050 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3058 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3130 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3134 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3175 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3205 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3257 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3315 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3338 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3435 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3436 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3489 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3490 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3503 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3512 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3513 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3538 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3549 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3557 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3637 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3734 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3740 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3743 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3744 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3745 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3746 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3747 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3748 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3749 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3750 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3751 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3752 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3753 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3754 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3755 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3756 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3757 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3758 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3781 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3873 noiclt13407.barco.com:0 LISTENING
              TCP noiclt13407:3944 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:3988 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4032 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4099 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4103 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4138 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4146 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4149 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4155 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4163 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4189 noiclt13407.barco.com:3873 TIME_WAIT
              TCP noiclt13407:4390 noiclt13407.barco.com:3873 TIME_WAIT

              • 4. Re: Socket Read Timeout on stressing calls for validating pr
                ron_sigal

                Hi Vinay,

                You have described problems on this thread and two other threads, which is making it hard for me to get a unified picture of what's going on. To try to clarify things, I'm consolidating the discussions on this thread. Here, I believe, are the phenomena you have experienced:

                1. This thread:

                "vink" wrote:

                On increasing timeout value of ejb3 connector, this exception is not coming. But, on running my stress tests, my application is throwing java.lang.OutOfMemoryError: unable to create new native thread


                Is it on the client side or the server side where you can't create a new thread? I believe the thread dumps you've shown me are from the server side, and I don't see anything particularly unusual.

                "vink" wrote:

                And, number of such sockets increases as time passes by & it finally results into the following exception;
                ...
                java.net.BindException: Address already in use: connect
                ...


                If you're running on a Windows machine, you could be running up against a configurable limit Windows puts on the number of open sockets. For example, the parameter "MaxUserPort", which defaults to 5000, limits the user ports to values between 1024 and 5000. You can read more at, for example, "Microsoft Windows 2000 TCP/IP Implementation Details" - http://technet.microsoft.com/en-us/library/bb726981.aspx#EDAA.

                2. "JBREM-877: New Socket Connection is being Created for Every" - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=126382&start=10

                "vink" wrote:

                On using invokerDestructionDelay in ejb3 connector configuration, my system goes slow & slow and run out of resources(native threads) in few minutes.


                Again, client side or server side?

                "vink" wrote:

                When removing this attribute, the system don't run out of resources, but many sockets are visible in TIME_WAIT state. As a result of this, my system run out of sockets & after a while I get sporadic disconnects.


                Client side or server side? And, could you describe "sporadic disconnects" more precisely?

                3. "Stream Corrupted Exception on Server side" - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=147905

                "vink" wrote:

                After several hours of running I'm getting following exception
                ...
                Caused by: java.io.StreamCorruptedException: invalid stream header: 77011679


                Clearly, this exception is occurring on the server side, but the actual problem might be occurring on the client side. Do you see anything unusual happening on the client side when this exception is thrown?


                ===============================================
                I have a couple of general questions.

                1. Am I right in inferring that you're running a stress test and you're seeing performance deteriorating on the client side?

                2. Can you create a simple version of your application that demonstrates the same problems and that I can run for myself?

                Thanks,
                Ron

                • 5. Re: Socket Read Timeout on stressing calls for validating pr
                  vink

                  Hello Ron,

                  Please find my comments below.
                  Issue 1:
                  On providing invokerDestructionDelay=10000, my system run out of threads. I mean my client side. Every call results in the creation of new thread, it seems old invoker thread is not getting reused.

                  Secondly, even if I'm working on windows; I'm not creating new sockets. I'm just placing a call to SLSB, now if it creates new socket or thread for each & every request, I think can only be treated as design issue.

                  • 6. Re: Socket Read Timeout on stressing calls for validating pr
                    vink

                    Issue 2:
                    Using default configuration, i.e no extra parameter to ejb3 connector configuration, many tcp connections on my system shows TIME_WAIT state. If you create a very simple RMI client & place invocation at an interval of 100 ms. And, you like to stress your server with several clients (in my case it extends upto 64).

                    I can understand the socket limit on windowsbased computer. But, if we see one time input which is 64 clients making 64 simultaneous requests then an efficient server must not consume more than 64 sockets. The other sockets has no caller for which it will wait.

                    The best use case for SLSB is to open a socket for a request & close it after finish.

                    • 7. Re: Socket Read Timeout on stressing calls for validating pr
                      vink

                      Issue 3:
                      Stream corrupted exception, it is very hard to reproduce. Last time when I change the log level of remoting to TRACE, after sometime my client faces disconnects with server. It normally appears when I run my client overnight, where a call is being placed every 5 seconds. It appears as if server releases sockets slowly & many calls are hitting & then server is unable to process their requests.

                      If there is some way either to restrict the logs, or to provide extra information; I would welcome to support it.

                      • 8. Re: Socket Read Timeout on stressing calls for validating pr
                        vink

                        In the meanwhile, I tried overnight tests on my system with invokerDestructionDelay=10000, & I face this exception on server. I cannot understand what it means, but it is linked with socket; so I thought it may be worth to share;

                        Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not execute query
                        at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
                        at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:73)
                        at com.barco.cms.eventmgt.EventQueuePoolImpl.createEventQueue(EventQueuePoolImpl.java:163)
                        at com.barco.cms.openapi.sessionmgt.SessionManager.createSession(SessionManager.java:109)
                        at com.barco.cms.openapi.OpenAPIImpl.startSession(OpenAPIImpl.java:148)
                        at com.barco.cms.openapi.PerspectiveMgtAPIImpl.startSession(PerspectiveMgtAPIImpl.java:426)
                        at sun.reflect.GeneratedMethodAccessor229.invoke(Unknown Source)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:597)
                        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                        at com.barco.cms.openapi.PerspectiveMgtAPIImpl.initialSteps(PerspectiveMgtAPIImpl.java:269)
                        ... 32 more
                        Caused by: org.hibernate.exception.JDBCConnectionException: could not execute query
                        at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:74)
                        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
                        at org.hibernate.loader.Loader.doList(Loader.java:2223)
                        at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
                        at org.hibernate.loader.Loader.list(Loader.java:2099)
                        at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)
                        at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
                        at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
                        at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
                        at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
                        at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:64)
                        ... 42 more
                        Caused by: org.postgresql.util.PSQLException: An I/O error occured while sending to the backend.
                        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:218)
                        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451)
                        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:350)
                        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
                        at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:90)
                        at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:236)
                        at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:186)
                        at org.hibernate.loader.Loader.getResultSet(Loader.java:1787)
                        at org.hibernate.loader.Loader.doQuery(Loader.java:674)
                        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
                        at org.hibernate.loader.Loader.doList(Loader.java:2220)
                        ... 50 more
                        Caused by: java.net.SocketException: Socket closed
                        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
                        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
                        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:78)
                        at org.postgresql.core.PGStream.SendChar(PGStream.java:174)
                        at org.postgresql.core.v3.QueryExecutorImpl.sendBind(QueryExecutorImpl.java:829)
                        at org.postgresql.core.v3.QueryExecutorImpl.sendOneQuery(QueryExecutorImpl.java:1053)
                        at org.postgresql.core.v3.QueryExecutorImpl.sendQueryPreamble(QueryExecutorImpl.java:373)
                        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:189)
                        ... 60 more

                        • 9. Re: Socket Read Timeout on stressing calls for validating pr
                          ron_sigal

                          Hi Vinay,

                          "vink" wrote:

                          Issue 1:
                          On providing invokerDestructionDelay=10000, my system run out of threads. I mean my client side. Every call results in the creation of new thread, it seems old invoker thread is not getting reused.


                          Well, I don't think this is a Remoting problem, because the socket transport doesn't create new threads on the client side. My conjecture is that there is some other problem, and setting "invokerDestructionDelay=10000" is just reducing the consumption of Remoting resources enough to allow this other problem to show up. What kind of threads are you seeing on the client side?

                          "vink" wrote:

                          The best use case for SLSB is to open a socket for a request & close it after finish.


                          Actually, that is the default behavior, but it's expensive to recreate sockets. The "invokerDestructionDelay" parameter is meant to prevent the closing of existing sockets when they are likely to be reused in the near future. By the way, if a socket is in the TIME_WAIT state, then it *has* been closed.

                          "vink" wrote:

                          n the meanwhile, I tried overnight tests on my system with invokerDestructionDelay=10000, & I face this exception on server. I cannot understand what it means, but it is linked with socket; so I thought it may be worth to share;

                          Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not execute query


                          This problem is somewhere between hibernate and your database. Definitely not a Remoting issue.

                          -Ron

                          • 10. Re: Socket Read Timeout on stressing calls for validating pr
                            immortalex

                             

                            "vink" wrote:

                            Issue 1:
                            On providing invokerDestructionDelay=10000, my system run out of threads. I mean my client side. Every call results in the creation of new thread, it seems old invoker thread is not getting reused.

                            I have the same issue. I think reason is invokerDestructionTimer in org.jboss.remoting.Client, which is created and started in disconnect() but isn't stopped in connect().
                            My situation is not artifically created stress test (but, of course, not every day working scheme). Worst thing is that I can't remove invokerDestructionDelay from Connector configuration. I'm using SSL, and without destruction delay every remote call works so slow...

                            • 11. Re: Socket Read Timeout on stressing calls for validating pr
                              ron_sigal

                               

                              "ImmortAlex" wrote:

                              invokerDestructionTimer in org.jboss.remoting.Client, which is created and started in disconnect()


                              Good catch, Alexey. EJB3 creates a new Client with each invocation, so there you are. I've created JIRA issue JBREM-1083 "Each Client creates a new invokerDestructionTimer", and I'll fix it in the next release.

                              "ron.sigal@jboss.com" wrote:

                              Well, I don't think this is a Remoting problem, because the socket transport doesn't create new threads on the client side.


                              Ah, my mistake. org.jboss.remoting.Client does. Thanks, Alexey.


                              • 12. Re: Socket Read Timeout on stressing calls for validating pr
                                vink

                                Hello Ron,

                                Can we expect this fix for Remoting 2.4.x.

                                I migrated to this version with a hope of better stability as visible from release notes.

                                • 13. Re: Socket Read Timeout on stressing calls for validating pr
                                  vink

                                  You have addressed my problem in other thread; http://www.jboss.com/index.html?module=bb&op=viewtopic&t=147397

                                  But, the JIRA issue created by you don't talk about its fix in 2.4.x, whereas you have exposed invokerDestructionDelay in this version.

                                  • 14. Re: Socket Read Timeout on stressing calls for validating pr
                                    vink

                                    You have addressed my problem in other thread; http://www.jboss.com/index.html?module=bb&op=viewtopic&t=147397

                                    But, the JIRA issue created by you don't talk about its fix in 2.4.x, whereas you have exposed invokerDestructionDelay in this version.

                                    1 2 Previous Next