Strange netty error when sending a lot of messages
rnicholson10 Oct 9, 2009 12:17 PMOk, I'm back on the Beta5 release. We are seeing an unusual error with netty. At the time this error occurs we will have 10s of thousands of tcp connections from localhost to localhost all in a TIME_WAIT state (using "netstat -an" to see these).
tcp 0 0 127.0.0.1:33727 127.0.0.1:5445 TIME_WAIT tcp 0 0 127.0.0.1:33721 127.0.0.1:5445 TIME_WAIT tcp 0 0 127.0.0.1:33725 127.0.0.1:5445 TIME_WAIT tcp 0 0 127.0.0.1:33724 127.0.0.1:5445 TIME_WAIT tcp 0 0 127.0.0.1:33722 127.0.0.1:5445 TIME_WAIT tcp 0 0 127.0.0.1:33728 127.0.0.1:5445 TIME_WAIT tcp 0 0 127.0.0.1:33726 127.0.0.1:5445 TIME_WAIT tcp 0 0 127.0.0.1:33723 127.0.0.1:5445 TIME_WAIT
We are sending 65K messages one after the other. Each message is dependent on the last one so they must be sent in sequence. Messages are sent via a core bridge using a message group.
What we think is happening is that netty cannot open another localhost connection as there are too many in the TIME_WAIT state.
The first error we get is a JMS exception on the MDB which should be consuming the messages.
2009-10-09 16:22:26,253 ERROR [STDERR] (pool-19-thread-1) Problem with creating connection or session. 2009-10-09 16:22:26,253 DEBUG [org.jboss.ejb3.interceptors.aop.InterceptorSequencer] (Thread-340 (group:HornetQ-client-global-threads-21795015)) aroundInvoke [advisedMethod=public void com.paddypower.phase.engine.bean.mdb.EngineMDB.onMessage(javax.jms.Message), unadvisedMethod=public void com.paddypower.phase.engine.bean.mdb.EngineMDB.onMessage(javax.jms.Message), metadata=null, targetObject=com.paddypower.phase.engine.bean.mdb.EngineMDB@1c572d0, arguments=[Ljava.lang.Object;@534e5d] 2009-10-09 16:22:26,253 ERROR [STDERR] (pool-19-thread-1) Messages processing will be not started. Fix queue problem and redeploy application.
Shortly after we get a lot of errors like the following:
2009-10-09 16:22:26,371 ERROR [STDERR] (pool-19-thread-15) Problem with creating connection or session. 2009-10-09 16:22:26,384 ERROR [STDERR] (pool-19-thread-15) Messages processing will be not started. Fix queue problem and redeploy application. 2009-10-09 16:22:26,384 INFO [STDOUT] (pool-19-thread-15) SentToHandler 2009-10-09 16:22:26,385 SEVERE [org.hornetq.integration.transports.netty.NettyConnector] (pool-19-thread-9) Failed to create netty connection java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(Unknown Source) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:145) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:109) at org.jboss.netty.channel.Channels.connect(Channels.java:762) at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:195) at org.jboss.netty.bootstrap.ClientBootstrap$Connector.channelOpen(ClientBootstrap.java:287) at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:197) at org.jboss.netty.channel.socket.nio.NioClientSocketChannel.<init>(NioClientSocketChannel.java:88) at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.newChannel(NioClientSocketChannelFactory.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.newChannel(NioClientSocketChannelFactory.java:93) at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:235) at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:199) at org.hornetq.integration.transports.netty.NettyConnector.createConnection(NettyConnector.java:363) at org.hornetq.core.client.impl.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:903) at org.hornetq.core.client.impl.ConnectionManagerImpl.getConnectionWithRetry(ConnectionManagerImpl.java:783) at org.hornetq.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:280) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:976) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:721) at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:710) at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:729) at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:307) at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:302) at com.paddypower.phase.engine.core.SingleDestMessageSenderTask.run(SingleDestMessageSenderTask.java:54) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
Once more, any ideas?