Hornetq stops responding
bhuvangupta Aug 7, 2012 12:26 PMWe are using JMS at the backend for a mobile application which caters for about 5000 concurrent connections at the moment.
We have a pool of 100 queues where all these apps are connected via message selectors to pull their messages.
Since the mobile networks are unreliable, they keep (re)connecting, disconnecting quite often. Whenever a app client is disconnected, we close the corresponding
JMS Queue Receiver to stop processing the messages for that client.
Because of these reconnection and creation/closure of sessions/consumers so frequently , the JMS server stops responding every few hours.
When this happens we see following exceptions in the logs:
* [Old I/O server worker (parentId: 207182797, [id: 0x0c595bcd, /10.142.211.140:5445])] 7-Aug 15:58:19,28 WARNING [QueueImpl] Timed out on waiting for MessageCount
java.lang.IllegalStateException: Timed out on waiting for MessageCount
at org.hornetq.core.server.impl.QueueImpl.getMessageCount(QueueImpl.java:828)
at org.hornetq.core.server.impl.ServerSessionImpl.executeQueueQuery(ServerSessionImpl.java:526)
at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:222)
at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:508)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:556)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:517)
at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:533)
at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:73)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372)
at org.jboss.netty.channel.StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(StaticChannelPipeline.java:534)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)
at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)
at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372)
at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:367)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:90)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
When I take thread dump of the server, lot of threads are blocked at Queue level operations
Old I/O server worker (parentId: 424711235, [id: 0x19509443, /10.142.211.140:5445])" prio=10 tid=0x0000000002082800 nid=0x7d13 waiting for monitor entry [0x00007fa1d3c0c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.hornetq.core.server.impl.QueueImpl.removeConsumer(QueueImpl.java:576)
- waiting to lock <0x000000050099fe88> (a org.hornetq.core.server.impl.QueueImpl)
at org.hornetq.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:342)
at org.hornetq.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1074)
at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:427)
at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:508)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:556)
- locked <0x00000007a3791488> (a java.lang.Object)
"Old I/O server worker (parentId: 424711235, [id: 0x19509443, /10.142.211.140:5445])" prio=10 tid=0x0000000001f9f000 nid=0x7cf2 waiting for monitor entry [0x00007fa1d9464000]
java.lang.Thread.State: BLOCKED (on object monitor) at org.hornetq.core.server.impl.QueueImpl.addConsumer(QueueImpl.java:557)
- waiting to lock <0x000000050099fe88> (a org.hornetq.core.server.impl.QueueImpl)
at org.hornetq.core.server.impl.ServerConsumerImpl.<init>(ServerConsumerImpl.java:178)
at org.hornetq.core.server.impl.ServerSessionImpl.createConsumer(ServerSessionImpl.java:346)
at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:214)
at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:508)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:556)
- locked <0x00000007a380ee08> (a java.lang.Object)
"hornetq-failure-check-thread" prio=10 tid=0x00007fa1e44f5800 nid=0x6364 waiting for monitor entry [0x00007fa1dd6a6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.hornetq.core.server.impl.QueueImpl.removeConsumer(QueueImpl.java:576)
- waiting to lock <0x000000050099fe88> (a org.hornetq.core.server.impl.QueueImpl)
at org.hornetq.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:342)
at org.hornetq.core.server.impl.ServerSessionImpl.doClose(ServerSessionImpl.java:306)
- locked <0x0000000511a284e8> (a org.hornetq.core.server.impl.ServerSessionImpl)
at org.hornetq.core.server.impl.ServerSessionImpl.access$100(ServerSessionImpl.java:86)
at org.hornetq.core.server.impl.ServerSessionImpl$1.done(ServerSessionImpl.java:1053)
The version of hornetq being used is 2.2.14 (hornetq-2.2.14.Final).
Any clue on what could be wrong ?