1 Reply Latest reply on Nov 9, 2013 9:38 PM by ron_sigal

Big readwrite lock in the jboss messaging service [EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire]

raghavev Oct 8, 2013 6:50 AM

Hi All,

Greetings !!!

Problem :

---------------------

Things are getting stuck stuck on the JBM side. There is a big readwrite lock in the jboss messaging server. All message delivery and sending involves getting the read lock [EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire]. Adding a new connection needs the write lock.

I see in the jstacks (an extract from the jstack as shown below) a massive pile up of server side threads that need the read lock

The locked resource 0xfffffffd3cdaa0d0 is not in RUNNABLE state in any of the threads.Several threads are hanging in waiting state in order to acquire the reader or writer lock eventhough there is no other thread holding on the lock.

Jstack_info:

------------------------

"XXXX" prio=3 tid=0x000000015b0eb800 nid=0x733 in Object.wait() [0xfffffffc5a539000]

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:485)

at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)

- locked <0xfffffffd3cdaa0d0> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)

at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2302)

at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:520)

at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:777)

at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)

at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)

at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)

at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)

at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)

at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)

at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)

at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)

at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)

at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)

at org.jboss.remoting.ServerInvoker$4.run(ServerInvoker.java:2171)

at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:147)

at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1746)

at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)

at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)

at org.jboss.util.threadpool.BasicThreadPool.execute(BasicThreadPool.java:487)

at org.jboss.util.threadpool.BasicThreadPool.runTaskWrapper(BasicThreadPool.java:206)

at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:226)

at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:220)

at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:2182)

at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:937)

at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:106)

at org.jboss.remoting.Client.invoke(Client.java:2077)

at org.jboss.remoting.Client.invoke(Client.java:880)

at org.jboss.remoting.Client$9.run(Client.java:934)