11 Replies Latest reply on Mar 19, 2012 11:19 PM by freeliuade

    meet hang on call a ejb, add a timeout property in jboss-server.xml?

    freeliuade

      Hi,

      I'm doing a nagative case for my application, a client to call a ejb server. (client and server is on different machine), when ejb call reach server side, unplug the server machine network. (by using debug to suspend server side to know the ejb server is on processing client call)

       

      client will wait forever. following is the client side thread dump, block at java.net.SocketInputStream.socketRead0(Native Method), and I debugged it, the timeout value send to native api is 0, means never timeout.

       

      C:\Documents and Settings\hyliu>jstack 18320
      2011-12-28 14:32:26
      Full thread dump Java HotSpot(TM) Client VM (17.0-b17 mixed mode):
      
      "RMI RenewClean-[pek-wkst71btb:1101]" daemon prio=6 tid=0x171ec000 nid=0x19f0 in Object.wait() [0x17
      3af000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              - waiting on <0x02eb00d0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
              - locked <0x02eb00d0> (a java.lang.ref.ReferenceQueue$Lock)
              at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
              at java.lang.Thread.run(Thread.java:619)
      
      "RMI Scheduler(0)" daemon prio=6 tid=0x171ddc00 nid=0x4070 waiting on condition [0x1735f000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x07f746b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
              at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
              at java.lang.Thread.run(Thread.java:619)
      
      "GC Daemon" daemon prio=2 tid=0x171da400 nid=0x43c4 in Object.wait() [0x1730f000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              - waiting on <0x07f74038> (a sun.misc.GC$LatencyLock)
              at sun.misc.GC$Daemon.run(GC.java:100)
              - locked <0x07f74038> (a sun.misc.GC$LatencyLock)
      
      "RMI RenewClean-[pek-wkst2cwxc:1101]" daemon prio=6 tid=0x171d9800 nid=0x471c in Object.wait() [0x17
      2bf000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              - waiting on <0x07f73cb0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
              - locked <0x07f73cb0> (a java.lang.ref.ReferenceQueue$Lock)
              at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
              at java.lang.Thread.run(Thread.java:619)
      
      "Low Memory Detector" daemon prio=6 tid=0x16bc8400 nid=0x3c78 runnable [0x00000000]
         java.lang.Thread.State: RUNNABLE
      
      "CompilerThread0" daemon prio=10 tid=0x16bb2000 nid=0x2090 waiting on condition [0x00000000]
         java.lang.Thread.State: RUNNABLE
      
      "JDWP Command Reader" daemon prio=6 tid=0x16baec00 nid=0x45b4 runnable [0x00000000]
         java.lang.Thread.State: RUNNABLE
      
      "JDWP Event Helper Thread" daemon prio=6 tid=0x16bad400 nid=0x2bb0 runnable [0x00000000]
         java.lang.Thread.State: RUNNABLE
      
      "JDWP Transport Listener: dt_socket" daemon prio=6 tid=0x16baa800 nid=0x26c0 runnable [0x00000000]
         java.lang.Thread.State: RUNNABLE
      
      "Attach Listener" daemon prio=10 tid=0x16b9a800 nid=0x4704 waiting on condition [0x00000000]
         java.lang.Thread.State: RUNNABLE
      
      "Signal Dispatcher" daemon prio=10 tid=0x16b99400 nid=0x403c runnable [0x00000000]
         java.lang.Thread.State: RUNNABLE
      
      "Finalizer" daemon prio=8 tid=0x16b89000 nid=0x3cbc in Object.wait() [0x16cff000]
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              - waiting on <0x07f20a30> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
              - locked <0x07f20a30> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
      
      "Reference Handler" daemon prio=10 tid=0x16b84400 nid=0x40f8 in Object.wait() [0x16caf000]
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              - waiting on <0x07f20958> (a java.lang.ref.Reference$Lock)
              at java.lang.Object.wait(Object.java:485)
              at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
              - locked <0x07f20958> (a java.lang.ref.Reference$Lock)
      
      "main" prio=6 tid=0x00847800 nid=0x449c runnable [0x0091f000]
         java.lang.Thread.State: RUNNABLE
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.read(SocketInputStream.java:129)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
              - locked <0x02afb570> (a java.io.BufferedInputStream)
              at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
              at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:242
      9)
              at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499)
              at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571)
              at java.io.ObjectInputStream.read(ObjectInputStream.java:820)
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1000)
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:574)
              at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
              at org.jboss.remoting.Client.invoke(Client.java:1634)
              at org.jboss.remoting.Client.invoke(Client.java:548)
              at org.jboss.invocation.unified.interfaces.UnifiedInvokerHAProxy.invoke(UnifiedInvokerHAProxy.java:188)
              at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365)
              at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197)
              at org.jboss.proxy.ejb.RetryInterceptor.invoke(RetryInterceptor.java:176)
              at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
              at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
              at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
              at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
              at $Proxy2.getDomainInfo(Unknown Source)
              at com.qilin.EJBLookupTest.lookupEJB(EJBLookupTest.java:78)
              at com.qilin.EJBLookupTest.lookup(EJBLookupTest.java:58)
              at com.qilin.EJBLookupTest.main(EJBLookupTest.java:33)
      
      "VM Thread" prio=10 tid=0x16b81c00 nid=0x43c8 runnable
      
      "VM Periodic Task Thread" prio=10 tid=0x16bca000 nid=0x4314 waiting on condition
      
      JNI global references: 2172
      

       

       

      and after some google search, add a timeout property (blue part) in server side jboss-service.xml can get a timeout. and tried it works

       

                     <!-- The following are specific to socket invoker -->

                     <!-- <attribute name="numAcceptThreads">1</attribute>-->

                     <!-- <attribute name="maxPoolSize">303</attribute>-->

                     <!-- <attribute name="clientMaxPoolSize" isParam="true">304</attribute>-->

                     <attribute name="timeout" isParam="true">600000</attribute>

                     <attribute name="socketTimeout" isParam="true">600000</attribute>

                     <attribute name="serverBindAddress">${jboss.bind.address}</attribute>

                     <attribute name="serverBindPort">4446</attribute>

                     <!-- <attribute name="clientConnectAddress">216.23.33.2</attribute> -->

                     <!-- <attribute name="clientConnectPort">7777</attribute> -->

       

      My question are:

      1. is it a right way to add this timeout. what's any side effect?

      2. why jboss hasn't a default timeout value for this?

       

      BTW, the ejb is a cluster ejb, have multiple nodes run on server side. unpluged one of nodes's network when using debug to get the client call is load balanced to this node.

       

      jboss version is 4.2.2GA

        • 1. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
          ron_sigal

          re: "1. is it a right way to add this timeout. what's any side effect?"

           

          Yes, it is.  In a very early version of Remoting, "socketTimeout" was used instead.  Setting "timeout" will cause that value to be passed to all read() calls on an InputStream obtained from a Socket.

           

          re: "2. why jboss hasn't a default timeout value for this?"

           

          Actually, the socket transport has a default read() timeout of 30 minutes.  I'm not sure why you are seeing 0. 

           

          -Ron

          1 of 1 people found this helpful
          • 2. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
            freeliuade

            Hi, thanks for your answer. I'm struggling this for a long time.

             

            It looks like to me this "socketTimeout" doesn't work as expect.

             

            see diagram1, the socketTimeout has been set. and in diagram2, this parameter doesn't be put to socket read call. BTW, my jboss version is 4.2.2GA

             

             

            diagram1.JPG

            diagram2.JPG

            • 3. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
              freeliuade

              another question:

              any way to configure timeout against a specified ejb method?

              • 4. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
                ron_sigal

                re: "It looks like to me this "socketTimeout" doesn't work as expect."

                 

                Right.  "socketTimeout" is no longer used.  "timeout" is the correct parameter.

                 

                re: "any way to configure timeout against a specified ejb method?"

                 

                No.  "timeout" applies to invocations on all methods.

                 

                -Ron

                • 5. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
                  freeliuade

                  ok, thanks very much!

                   

                  BTW, does this feature got enhancement in jboss later version?, if yes, after which version?  the enhancement possiable are: 1. add a default timeout, 2. let user can specify different timeout value in ejb method level.

                  • 6. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
                    ron_sigal

                    re: "does this feature got enhancement in jboss later version?"

                     

                    There are two possible answers, in general: (1) for Remoting 2 (AS 4,5,6) and (2) Remoting 3 (AS 7).  Unfortunately, I don't know enough about Remoting 3 to answer.

                     

                    With respect to Remoting 2:

                     

                    re: "add a default timeout,"

                     

                    I'm not really sure what you mean.  The code itself has a hard coded default timeout of 30 minutes.  If you set the "timeout" parameter, that value becomes the timeout value for all invocations.

                     

                    re: "let user can specify different timeout value in ejb method level"

                     

                    Remoting 2 is currently in maintenance mode and there will be no new features.

                    1 of 1 people found this helpful
                    • 7. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
                      freeliuade

                      re: "add a default timeout,"

                       

                      I'm not really sure what you mean.  The code itself has a hard coded default timeout of 30 minutes.  If you set the "timeout" parameter, that value becomes the timeout value for all invocations.

                       

                      ----------------

                      Hi, about this, a litter bit confuse, in my testing, if not set "timeout" parameter, my ejb client will stick be there always if unplug network of ejb server machine when ejb server on processing ejb call.

                      so it seems NO default timeout .

                       

                      another question is: if add "timeout" parameter, except EJB call, any other jboss services will get impact? (i.e. any other jboss components are based on Remoting)

                      • 8. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
                        ron_sigal

                        re: "in my testing, if not set "timeout" parameter, my ejb client will stick be there always if unplug network of ejb server machine when ejb server on processing ejb call.

                        so it seems NO default timeout . "

                         

                        The 30 minutes default might seem like forever.  Are you sure that, if you wait 30 minutes, the invocation won't time out?

                         

                        re: "if add "timeout" parameter, except EJB call, any other jboss services will get impact?"

                         

                        Any service that uses the connector upon which you set the "timeout" parameter will be affected.  However, I don't believe any other service uses the EJB2 connector.

                         

                        -Ron

                        • 9. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
                          freeliuade

                          re: "in my testing, if not set "timeout" parameter, my ejb client will stick be there always if unplug network of ejb server machine when ejb server on processing ejb call.

                          so it seems NO default timeout . "

                           

                          The 30 minutes default might seem like forever.  Are you sure that, if you wait 30 minutes, the invocation won't time out?

                           

                          -----------------

                          sure, following is my steps, it's a nagative test case for simulate a case happened in our production enviorment, you may want help review it.

                          1. a ejb server machine, a ejb client machine. open debug on the ejb server

                          2. debug the ejb method to let ejb processing stick. now unplug the ejb server machine network.

                          3. wait ... wait .... (more than one night)

                          4. the next day morning, client still stick on there, with jps/jstack call print the call stack as the top of discussion.

                           

                          also, you can see the diagram1 in earlier reply, the impl.getTimeout() is 0, which would means it will always wait.

                          • 10. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
                            ron_sigal

                            re: "you can see the diagram1 in earlier reply, the impl.getTimeout() is 0"

                             

                            Hmmm, I'm not sure how that happens.  But, if, as you say, setting "timeout" works ("add a timeout property (blue part) in server side jboss-service.xml can get a timeout. and tried it works"), then the rest is just theoretical, right?

                            • 11. Re: meet hang on call a ejb, add a timeout property in jboss-server.xml?
                              freeliuade

                              ok, thanks!