JMS Message Queueing bottleneck
We are using jboss-4.0.2. I have around 1500 queues into my application. There are seperate threads for each queues. All thread dumps have the usual pattern that most of threads are waiting for the following lock. Even the threads that work on seperate Queues have to wait for the same lock. Is there anyway, I can prevent this locking/bottleneck ? Atleast it will be great if threads working on different queues don't have to wait for the same lock. Application will then be able to scale well.
"MessageProcessor1" prio=5 tid=0x03ea7a38 nid=0xee4 waiting for monitor entry [4fdde000..4fddfc30]
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getPool(JBossManagedConnectionPool.java:660)
- waiting to lock <0xb5c6d410> (a java.util.HashMap)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:514)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:395)
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:297)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:447)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1368)
at org.jboss.mq.pm.jdbc2.PersistenceManager.loadFromStorage(PersistenceManager.java:1176)
at org.jboss.mq.server.MessageCache.loadFromStorage(MessageCache.java:402)
at org.jboss.mq.server.MessageReference.makeHard(MessageReference.java:350)
- locked <0xd8cc50c8> (a org.jboss.mq.server.MessageReference)
at org.jboss.mq.server.MessageReference.getMessage(MessageReference.java:155)
- locked <0xd8cc50c8> (a org.jboss.mq.server.MessageReference)
at org.jboss.mq.server.MessageReference.getHeaders(MessageReference.java:248)
at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:508)
- locked <0xc239efd0> (a java.lang.Object)
- locked <0xc239f000> (a org.jboss.mq.server.ReceiversImpl)
at org.jboss.mq.server.JMSQueue.receive(JMSQueue.java:136)
at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:222)
at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager.java:656)
at org.jboss.mq.server.JMSServerInterceptorSupport.receive(JMSServerInterceptorSupport.java:226)
at org.jboss.mq.security.ServerSecurityInterceptor.receive(ServerSecurityInterceptor.java:100)
at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:570)
at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:226)
at org.jboss.mq.il.jvm.JVMServerIL.receive(JVMServerIL.java:244)
at org.jboss.mq.Connection.receive(Connection.java:909)
at org.jboss.mq.SpyMessageConsumer.receive(SpyMessageConsumer.java:398)
- locked <0xd8d43620> (a java.util.LinkedList)
.........
.........
Following is the stack trace of the thread holding the lock. This thread is trying to remove the message from the Queue.
"MessageUser520" prio=5 tid=0x03eaa9f8 nid=0xee8 runnable [4fc9e000..4fc9fc30]
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$SubjectActions.hashCode(JBossManagedConnectionPool.java:918)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$SubjectKey.hashCode(JBossManagedConnectionPool.java:803)
at java.util.HashMap.hash(HashMap.java:261)
at java.util.HashMap.get(HashMap.java:317)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getPool(JBossManagedConnectionPool.java:662)
- locked <0xb5c6d410> (a java.util.HashMap)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:514)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:395)
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:297)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:447)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1368)
at org.jboss.mq.pm.jdbc2.PersistenceManager.remove(PersistenceManager.java:1025)
at org.jboss.mq.server.BasicQueue.acknowledge(BasicQueue.java:573)
at org.jboss.mq.server.JMSQueue.acknowledge(JMSQueue.java:123)
at org.jboss.mq.server.ClientConsumer.acknowledge(ClientConsumer.java:328)
at org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:519)
at org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:503)
at org.jboss.mq.server.JMSServerInterceptorSupport.acknowledge(JMSServerInterceptorSupport.java:198)
at org.jboss.mq.server.TracingInterceptor.acknowledge(TracingInterceptor.java:481)
at org.jboss.mq.server.JMSServerInvoker.acknowledge(JMSServerInvoker.java:198)
at org.jboss.mq.il.jvm.JVMServerIL.acknowledge(JVMServerIL.java:214)
at org.jboss.mq.Connection.send(Connection.java:1139)
at org.jboss.mq.SpySession.doAcknowledge(SpySession.java:175)
- locked <0xd8898e50> (a java.util.ArrayList)
at org.jboss.mq.SpyMessage.doAcknowledge(SpyMessage.java:320)
at org.jboss.mq.SpyMessageConsumer.preProcessMessage(SpyMessageConsumer.java:829)
at org.jboss.mq.SpyMessageConsumer.receive(SpyMessageConsumer.java:401)
- locked <0xdc3b81e8> (a java.util.LinkedList)
Many thanks in Advance. Any input will be greatly appreciated.
Regards,
Sudarshan