We are using JBoss App Server 4.2.3.GA, JBoss Messaging 1.4.5 GA and JBoss Remoting 2.2.3 SP1. In our application, there are a lot of Message listeners running on the client side, these message listeners will receive messages from queue/topic deployed in JBoss Messaging


Configuration:

We created our own JMS Connection factory which uses the default remoting connector. As you know, the default remoting connector is configured to use the bisocket transport. We didn't change the default value of the remoting connector

<invoker transport="bisocket">

 <!-- There should be no reason to change these parameters - warning!
 Changing them may stop JBoss Messaging working correctly -->
 <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="dataType" isParam="true">jms</attribute>
 <attribute name="socket.check_connection" isParam="true">false</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">4457</attribute>
 <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
 <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>

 <attribute name="numberOfCallRetries" isParam="true">1</attribute>
 <attribute name="pingFrequency" isParam="true">214748364</attribute>
 <attribute name="pingWindowFactor" isParam="true">10</attribute>
 <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
 <!-- End immutable parameters -->

 <attribute name="stopLeaseOnFailure" isParam="true">true</attribute>

 <!-- Periodicity of client pings. Server window by default is twice this figure -->
 <attribute name="clientLeasePeriod" isParam="true">10000</attribute>
 <attribute name="validatorPingPeriod" isParam="true">10000</attribute>
 <attribute name="validatorPingTimeout" isParam="true">5000</attribute>
 <attribute name="registerCallbackListener">false</attribute>

 <attribute name="timeout" isParam="true">0</attribute>

 <!-- Number of seconds to wait for a connection in the client pool to become free -->
 <attribute name="numberOfRetries" isParam="true">10</attribute>

 <!-- Max Number of connections in client pool. This should be significantly higher than
 the max number of sessions/consumers you expect -->
 <attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>

 <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -
->
 <attribute name="callbackTimeout">10000</attribute>
 <attribute name="secondaryBindPort">4557</attribute>

 <!-- Use these parameters to specify values for binding and connecting control connections to
 work with your firewall/NAT configuration
 <attribute name="secondaryBindPort">xyz</attribute>
 <attribute name="secondaryConnectPort">abc</attribute>
 -->

 </invoker>

During we run our application, we open the JBoss web console to monitor the value of currentClientPoolSize under "Jboss.remoting" JMX MBean.


How to reproduce this issue:

1. Run 5 message listeners in the client side to receive messages from JBoss Messaging, then we observe the value of currentClientPoolSize is 10

2. After processing several messages, we manually pull out the ethernet cable. The value of currentClientPoolSize is still 10.

3. We run another 5 message listeners in client side, then the value of currentClientPoolSize will become 20

4. After we do the same operations above several times, the value of currentClientPoolSize will increase continuously. Once the value of currentClientPoolSize is equal to the MaxPoolSize, then the subsequent incoming client requests will hang, and we will encounter the following exception in server side

2009-10-20 18:08:09,655 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure

java.net.SocketException: Connection reset

 at java.net.SocketInputStream.read(SocketInputStream.java:168)

 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

 at java.io.BufferedInputStream.read(BufferedInputStream.java:235)

 at java.io.FilterInputStream.read(FilterInputStream.java:66)

 at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:859)

 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:545)

 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406)

 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)

Conclusion: JBoss Messaging won't close the failure connections if they are caused by manually pulling out ethernet cable. As a result, the value of currentClientPoolSize will increase continuously and finally the new client requests will hang

Note: If we killed the process of message listener in client side, then the value of currentClientPoolSize will decrease to 0 immediately, it seems that the server could detect the failure connection and perform the corresponding resource releasing.