Threads getting blocked on MicroRemoteClientInvoker.establishLease and MicroRemoteClientInvoker.terminateLease
Hi Team,
We are running 2 Jboss Nodes (4.2.2 GA) in a cluster with Jboss Messaging 1.4.4.GA and Jboss remoting ( 2.2.3).
After running the setup for a month we see the application dragging. Every operation related to JMS queue (either dispatching a message to queue to retrieving a message from JMS queue) is taking considerable time approx. 7/10 mins.
When we analysed the thread dumps, we found that all the threads are getting into BLOCKED state as either of following place:
Name: Thread-100406008 Id: 103030603
State: BLOCKED
Lock name: java.lang.Object@74cbe038
Lock owner: Thread-100402565 Lock owner id: 103026056
Total blocked: 1 Total waited: 1
Stack trace:
org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:442)
org.jboss.remoting.Client.setupClientLease(Client.java:1845)
org.jboss.remoting.Client.connect(Client.java:1732)
org.jboss.remoting.Client.connect(Client.java:613)
org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)
java.security.AccessController.doPrivileged(Native Method)
org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)
sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:87)
org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:82)
com.mypack.common.msg.MessageFactory.handleDispatchMessage(MessageFactory.java:378)
com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:342)
com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:326)
com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:148)
com.mypack.dam.tasks.FileChangeNotificationTask.dispatchMessage(FileChangeNotificationTask.java:780)
com.mypack.dam.tasks.FileChangeNotificationTask.performTaskNow(FileChangeNotificationTask.java:581)
com.mypack.dam.tasks.DeleteFileFromStorageTask$1.run(DeleteFileFromStorageTask.java:402)
java.lang.Thread.run(Thread.java:662)
Name: Thread-100402494 Id: 103025979
State: BLOCKED
Lock name: java.lang.Object@74cbe038
Lock owner: Thread-100402565 Lock owner id: 103026056
Total blocked: 2 Total waited: 1
Stack trace:
org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:353)
org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:348)
org.jboss.remoting.Client.disconnect(Client.java:633)
org.jboss.jms.client.remoting.JMSRemotingConnection.stop(JMSRemotingConnection.java:418)
org.jboss.jms.client.container.ConnectionAspect.handleClose(ConnectionAspect.java:192)
sun.reflect.GeneratedMethodAccessor300.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)
org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)
org.jboss.jms.client.delegate.ClientConnectionDelegate.close(ClientConnectionDelegate.java)
org.jboss.jms.client.JBossConnection.close(JBossConnection.java:132)
com.mypack.common.msg.MessageFactory.handleDispatchMessage(MessageFactory.java:390)
com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:342)
com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:326)
com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:148)
com.mypack.dam.tasks.FileChangeNotificationTask.dispatchMessage(FileChangeNotificationTask.java:780)
com.mypack.dam.tasks.FileChangeNotificationTask.performTaskNow(FileChangeNotificationTask.java:581)
com.mypack.dam.tasks.DeleteFileFromStorageTask$1.run(DeleteFileFromStorageTask.java:402)
java.lang.Thread.run(Thread.java:662)
The stack trace of Lock owner thread is as below :
Name: Thread-100402565 Id: 103026056
State: RUNNABLE
Lock name: null
Lock owner: null Lock owner id: -1
Total blocked: 17 Total waited: 7
Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.FilterInputStream.read(FilterInputStream.java:66)
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:957)
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:590)
org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422)
org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
org.jboss.remoting.LeasePinger.sendClientPing(LeasePinger.java:369)
org.jboss.remoting.LeasePinger.addClient(LeasePinger.java:175)
org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:448)
org.jboss.remoting.Client.setupClientLease(Client.java:1845)
org.jboss.remoting.Client.connect(Client.java:1732)
org.jboss.remoting.Client.connect(Client.java:613)
org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)
java.security.AccessController.doPrivileged(Native Method)
org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
Can you please suggest what could lead to this condition?
Please be informed that upgrade to any later version of JBOSS is not an option for us because of some constraints.
Restarting both the nodes fixes the problem for one month approx.
Thanks,
Veena