6 Replies Latest reply on Nov 2, 2007 12:50 PM by nkhadakk

    Sockets left in CLOSE_WAIT on server

    engluer

      I was trying to cleaning close a connection to a JBoss server. I have tried the following on both JBoss 4.2.2 with Messaging and JBoss 5 b2 and in both cases when the client closes the connection a socket is left in the CLOSE_WAIT state on the server. In both cases I was using the BiSocket transport


      import javax.jms.Connection;
      import javax.jms.ConnectionFactory;
      import javax.jms.Destination;
      import javax.jms.JMSException;
      import javax.jms.Message;
      import javax.jms.MessageConsumer;
      import javax.jms.MessageListener;
      import javax.jms.Session;
      import javax.naming.InitialContext;
      import javax.naming.NamingException;
      
      /*
       *
       * Note monitor server connections with
       * watch "/usr/sbin/lsof -p [pid of Jboss server] | grep -c CLOSE_WAIT"
       *
       */
      
      public class SimpleSubscriber implements MessageListener {
      
       private Session subSession;
       private Connection connection;
       private String destName;
      
       private String password;
       private String username;
       private ConnectionFactory conFactory;
       private InitialContext ic;
       private MessageConsumer subscriber;
      
       public SimpleSubscriber(String destName, String username, String password) {
       this.destName = destName;
       this.username = username;
       this.password = password;
       }
      
       private void connect() throws NamingException, JMSException {
       if (ic == null) {
       ic = new InitialContext();
       }
      
       //Look up JMS connection Factory
       conFactory = (ConnectionFactory)ic.lookup("/ConnectionFactory");
      
       //Lookup Destination
       Object obj = ic.lookup(destName);
       Destination topic = (Destination)obj;
      
       //Create JMS connection
       connection = conFactory.createConnection(username, password);
      
       //Create two JMS session objects
       subSession = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
       subscriber = subSession.createConsumer(topic);
       subscriber.setMessageListener(this);
      
       //Start the connection
       connection.start();
       }
      
       public void close() throws JMSException {
       subscriber.close();
       subSession.close();
       connection.close();
       }
      
       public static void main(String [] args) {
       String username = null;
       String password = null;
       String dest= null;
      
       if (args.length != 3) {
       printUsage();
       return;
       }
      
       dest= args[0];
      
       username = args[1];
       password = args[2];
      
       final SimpleSubscriber subscriber = new SimpleSubscriber(dest, username, password);
      
       System.out.println("Connecting to JBoss...");
       try {
       subscriber.connect();
       } catch (NamingException e) {
       e.printStackTrace();
       return;
       } catch (JMSException e) {
       e.printStackTrace();
       return;
       }
      
       System.out.println("Connected");
      
       System.out.println("Sleeping");
      
       try {
       Thread.sleep(5000);
       } catch (InterruptedException e) {}
       System.out.println("Disconnecting");
      
       try {
       subscriber.close();
       } catch (JMSException e) {
       e.printStackTrace();
       }
      
       System.out.println("Closed");
      
       try {
       Thread.sleep(5000);
       } catch (InterruptedException e) {}
      
       System.out.println("Done");
       }
      
       private static void printUsage() {
       System.out.println("Subscriber <topic> [username] [password]"
       + "\ntopic: Name of topic to connect to pings with"
       + "\nusername: Username to connect with"
       + "\npassword: Password to connect with");
       }
      
       public void onMessage(Message arg0) {
       System.out.println("Received Message: " + arg0);
       }
      
      }
      




      Th above example connects waits, then disconnects and waits, then shuts down. Am I not closing the connection correctly? I am using the following to check for socket state.
      watch "/usr/sbin/lsof -p [pid of Jboss server] | grep -c CLOSE_WAIT"
      


      The server we are using has client connecting and disconnecting quite rapidly and we quickly run out of file descriptors because of all the CLOSE_WAIT sockets left open.

        • 1. Re: Sockets left in CLOSE_WAIT on server
          engluer

          [JBREM-774] Seems to be related I will try upgrading to Remoting 2.2.2 SP1

          • 2. Re: Sockets left in CLOSE_WAIT on server
            engluer

            Upgrading has solved the issue.

            • 3. Re: Sockets left in CLOSE_WAIT on server
              ron_sigal

              You've made my day!

              • 4. Re: Sockets left in CLOSE_WAIT on server
                nkhadakk

                JBoss AS 4.2.1GA / JBM 1.3.0GA

                I copied the above sample file and ran it in our environment. Sure enough, there is a CLOSE_WAIT ed socket left hanging.
                I now upgraded from default remoting that comes with JBoss AS 4.2.1 (2.2.1 GA) to 2.2.2 SP1 remoting and i still see this issue.

                This issue is manifesting in our environment when the JMS provider is brought down (leaving the JMS remote clients running) and started back up. We see a bunch of sockets left hanging in the CLOSE_WAIT state.

                I would appreciate any help in this regard

                • 5. Re: Sockets left in CLOSE_WAIT on server
                  engluer

                  Out of the box JBoss AS 4.2.1 and I would believe 4.2.2 run JBoss MQ.

                  We updated our versions of JBoss AS to use Jboss messaging 1.4.0.GA. The installation isn't that hard, but you have to be careful about making sure you also update the jboss remoting jar to 2.2.0 or else it won't work.

                  We have successfully run the code without close waits with Messaging and updated remoting but have not tested it with MQ. My suggestion would be to upgrade to Messaging if at all possible. This requires a change on the clients as well as the server.

                  Hope this helps.

                  • 6. Re: Sockets left in CLOSE_WAIT on server
                    nkhadakk

                    Just upgrading to JBoss Remoting 2.2.2.SP1 and keeping to older JBoss AS(4.2.1) and JBM(1.3) did not fix this issue. We had to upgrade across the board to JBM 1.4SP1 + JBoss AS 4.2.2 GA + JBoss Remoting 2.2.2.SP1 and this issue seems to be alleviated significantly.

                    In some fringe cases we still see a couple of CLOSE_WAIT s hanging around. These fringe cases have to do with scenarios where the message producer is sending messages with the JMS provider restarting AND the Message consumer listening to older sockets that should have been discarded when the JMS provider restarted.

                    These cases are quite fringe and require a certain amount of application code bugs (to allow listeners on older sockets) to recreate....