10 Replies Latest reply: Jan 9, 2012 4:47 AM by Hushen Savani RSS

javax.jms.JMSException: Timed out waiting for response when sending packet 30

Hushen Savani Newbie

Hi,

   

     I have my application running on JBossAS-5.1.0 with Hornetq-2.2.5. My application is running on Solaris SPARC Machine. The application works fine with lower number of messages. i.e. 50. But when it increases to 400+, following exception is coming:

 

 

2011-12-29 10:32:35,154 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732)) javax.jms.JMSException: Timed out waiting for response when sending packet 30
2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276)
2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:695)
2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:264)
2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:589)
2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:694)
2011-12-29 10:32:35,157 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:121)
2011-12-29 10:32:35,157 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:116)
2011-12-29 10:32:35,157 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at com.elitecore.messageservice.ejb.base.utils.hornetq.HornetQ.startConnection(HornetQ.java:84)

 

Following are my observations:

 

1) There is no network latency as consumer and producer are on the same machine.

2) This works fine on Linux machine for very higher number of messages. i.e. 250000+ Messages.

3) All the HornetQ Configurations are same on Solaris & Linux Machine.

 

Following are my configurations:

 

hornetq-configuration.xml:

 

<!--
  ~ Copyright 2009 Red Hat, Inc.
  ~  Red Hat licenses this file to you under the Apache License, version
  ~  2.0 (the "License"); you may not use this file except in compliance
  ~  with the License.  You may obtain a copy of the License at
  ~     http://www.apache.org/licenses/LICENSE-2.0
  ~  Unless required by applicable law or agreed to in writing, software
  ~  distributed under the License is distributed on an "AS IS" BASIS,
  ~  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
  ~  implied.  See the License for the specific language governing
  ~  permissions and limitations under the License.
  -->


<configuration xmlns="urn:hornetq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">


   <!--  Don't change this name.
         This is used by the dependency framework on the deployers,
         to make sure this deployment is done before any other deployment -->
   <name>HornetQ.main.config</name>


   <log-delegate-factory-class-name>org.hornetq.integration.logging.Log4jLogDelegateFactory</log-delegate-factory-class-name>
   
   <bindings-directory>${jboss.server.data.dir}/hornetq/bindings</bindings-directory>


   <journal-directory>${jboss.server.data.dir}/hornetq/journal</journal-directory>
   
   <journal-min-files>10</journal-min-files>


   <journal-type>ASYNCIO</journal-type>


   <large-messages-directory>${jboss.server.data.dir}/hornetq/largemessages</large-messages-directory>


   <paging-directory>${jboss.server.data.dir}/hornetq/paging</paging-directory>
   
   <message-counter-enabled>true</message-counter-enabled>
   
   <message-counter-max-day-history>7</message-counter-max-day-history> 
   
   <!-- sample the queues every minute (1ms) -->
          <message-counter-sample-period>29000</message-counter-sample-period>


   <connectors>
      <connector name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
<!--         <param key="host"  value="${jboss.bind.address:localhost}"/> -->
           <param key="host"  value="192.168.1.76"/> 
           <param key="port"  value="${hornetq.remoting.netty.port:5445}"/> 
      </connector>
      
      <connector name="netty-throughput">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
    <!--     <param key="host"  value="${jboss.bind.address:localhost}"/> -->
          <param key="host"  value="192.168.1.76"/> 
         <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
         <param key="batch-delay" value="50"/>
      </connector>


      <connector name="in-vm">
         <factory-class>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</factory-class>
         <param key="server-id" value="${hornetq.server-id:0}"/> 
      </connector>


   </connectors>


   <acceptors>   
      <acceptor name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
        <!-- <param key="host"  value="${jboss.bind.address:localhost}"/> -->
          <param key="host"  value="192.168.1.76"/> 
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </acceptor>
      
      <acceptor name="netty-throughput">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
        <!--  <param key="host"  value="${jboss.bind.address:localhost}"/> -->
          <param key="host"  value="192.168.1.76"/> 
         <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
         <param key="batch-delay" value="50"/>
         <param key="direct-deliver" value="false"/>
      </acceptor>


      <acceptor name="in-vm">
        <factory-class>org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory</factory-class>
        <param key="server-id" value="0"/>
      </acceptor>


   </acceptors>


   <security-settings>
      <security-setting match="#">
         <permission type="createNonDurableQueue" roles="guest"/>
         <permission type="deleteNonDurableQueue" roles="guest"/>
         <permission type="consume" roles="guest"/>
         <permission type="send" roles="guest"/>
      </security-setting>
   </security-settings>


   <address-settings>
      <!--default for catch all-->
      <address-setting match="#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>3221225472</max-size-bytes>       
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
          <page-size-bytes>104857600</page-size-bytes>
         <address-full-policy>PAGE</address-full-policy>


      </address-setting>
   </address-settings>


</configuration>

 

 

hornetq-jms.xml:

 

<configuration xmlns="urn:hornetq"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">


   <connection-factory name="NettyConnectionFactory">
      <xa>true</xa>
                    <client-failure-check-period>-1</client-failure-check-period>
      <connectors>
         <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
         <entry name="/XAConnectionFactory"/>
      </entries>
   </connection-factory>
   
   <connection-factory name="NettyConnectionFactory">
      <xa>false</xa>
                    <client-failure-check-period>-1</client-failure-check-period>
      <connectors>
         <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
         <entry name="/ConnectionFactory"/>
      </entries>


   </connection-factory>
   
   <connection-factory name="NettyThroughputConnectionFactory">
      <xa>true</xa>
                    <client-failure-check-period>-1</client-failure-check-period>
      <connectors>
         <connector-ref connector-name="netty-throughput"/>
      </connectors>
      <entries>
         <entry name="/XAThroughputConnectionFactory"/>
      </entries>
   </connection-factory>
   
   <connection-factory name="NettyThroughputConnectionFactory">
      <xa>false</xa>
          <client-failure-check-period>-1</client-failure-check-period>
      <connectors>
         <connector-ref connector-name="netty-throughput"/>
      </connectors>
      <entries>
         <entry name="/ThroughputConnectionFactory"/>
      </entries>


   </connection-factory>
   
   <connection-factory name="InVMConnectionFactory">
      <xa>true</xa>
                    <client-failure-check-period>-1</client-failure-check-period>
      <connectors>
         <connector-ref connector-name="in-vm"/>
      </connectors>
      <entries>
         <entry name="java:/XAConnectionFactory"/>
      </entries>
   </connection-factory>


   <connection-factory name="InVMConnectionFactory">
      <xa>false</xa>
                    <client-failure-check-period>-1</client-failure-check-period>
      <connectors>
         <connector-ref connector-name="in-vm"/>
      </connectors>
      <entries>
         <entry name="java:/ConnectionFactory"/>
      </entries>
   </connection-factory>


   <queue name="DLQ">
      <entry name="/queue/DLQ"/>
   </queue>
   
   <queue name="ExpiryQueue">
      <entry name="/queue/ExpiryQueue"/>
   </queue>


          <queue name="testQueue">
      <entry name="/queue/testQueue"/>
          </queue>
  
</configuration>

 

Pl. have some pointers on the same. Thanks.

  • 1. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Yong Hao Gao Master

    Any error message at the server side?

     

    Also are you creating connections/sessions each time you send a message?

     

    Howard

  • 2. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Hushen Savani Newbie

    Hi Howard,

     

    Any error message at the server side?

     

    Following log messages are coming on server side:

     

    011-12-29 18:29:26,814 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /127.0.0.1:45505. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

    2011-12-29 18:29:26,814 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a82f4a9c-321c-11e1-80a5-b1c6e8464921

    2011-12-29 18:29:26,815 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a82f4a9c-321c-11e1-80a5-b1c6e8464921

    2011-12-29 18:29:26,815 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a82f98bd-321c-11e1-80a5-b1c6e8464921

    2011-12-29 18:29:26,815 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a82f98bd-321c-11e1-80a5-b1c6e8464921

    2011-12-29 18:29:26,822 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /127.0.0.1:45506. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

    2011-12-29 18:29:26,826 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a830aa2f-321c-11e1-80a5-b1c6e8464921

    2011-12-29 18:29:26,826 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a830aa2f-321c-11e1-80a5-b1c6e8464921

    2011-12-29 18:29:26,826 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a830f850-321c-11e1-80a5-b1c6e8464921

    2011-12-29 18:29:26,826 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a830f850-321c-11e1-80a5-b1c6e8464921

     

    However, with this warn messages, it was working fine on Linux Machine.

     

    Also are you creating connections/sessions each time you send a message?

     

    Yes, Connections/Sessions are created each time.

  • 3. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Yong Hao Gao Master

    Re: However, with this warn messages, it was working fine on Linux Machine.

     

    This error messages suggests that your client didn't close the connections properly. They should be closed when is not used.

     

    Re: Yes, Connections/Sessions are created each time.

     

    This is very inefficient. You should cache connections/sessions/consumers/producers whenever possible. Because creating those objects are expensive calls.

  • 4. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Hushen Savani Newbie

    Hi Howard,

    This is very inefficient. You should cache connections/sessions/consumers/producers whenever possible. Because creating those objects are expensive calls.

     

    I am recreating sessions and connection each time as my application is running on JEE Application Server. And according to following wiki article, it is not an anti-pattern:

     

    http://community.jboss.org/wiki/ShouldICacheJMSConnectionsAndJMSSessions

     

    The mystry is that the same configuration and the same code is working very fine on Linux machine. No such "timed out" exception appeared even once. But on Solaris, it appears.

     

    Can there be any possible solution for this exception?

  • 5. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Yong Hao Gao Master

    com.elitecore.messageservice.ejb.base.utils.hornetq.HornetQ.startConnection(HornetQ.java:84)

     

    I ask the question because I see this line in the error message. If your application is running inside JEE app server, you should use JCA api to get JMS resources. However this line seems to me you directly create the connections, which are not managed by JCA. Is that true?

     

    Re: Can there be any possible solution for this exception?

     

    We need to find out what's the cause first and then come up with a solution. So far I'm not clear what's happened.

  • 6. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Hushen Savani Newbie

    Hi Howard,

     

    Following is the code snippet where I start the connection:

     

    @Resource(mappedName = "java:JmsXA")

              private ConnectionFactory _cfConnFactory = null;

     

     

     

              public void startConnection(Context initialContecxt, String hornetqQueue) throws BusinessTaskException {//NamingException, JMSException {

     

                        String strMethodName ="startConnection():: ";

     

                        System.out.println(strLogPreffix + strMethodName + "HornetQ Queue Name:: " + hornetqQueue);

     

                        try {

                                  this._ctx = initialContecxt;

     

                                  String queue = "";

                                  if(hornetqQueue!=null) {

                                            //Set HornetQ Queue

                                            queue = "/queue/" + hornetqQueue;

                                            this._queue = (Queue)_ctx.lookup(queue);

     

     

                                            //Set Connection

                                            _cfConnFactory = (ConnectionFactory) _ctx.lookup("/ThroughputConnectionFactory");

                                            this._conn = _cfConnFactory.createConnection();

     

                                            //Set session

                                            this._session = _conn.createSession(false, Session.CLIENT_ACKNOWLEDGE);

                                            _producer = _session.createProducer(_queue);

                                _producer.setDisableMessageID(true);

                                _producer.setDisableMessageTimestamp(true);

                               

                                //Start Service

                                if(_conn!=null)

                                                      _conn.start();

                                            else

                                                      System.err.println("[ HornetQ:: startConnection() ]Error in Connection!");                           

                                    

                                  }

                                  else {

                                            System.err.println("[HornetQ :: startConnection() Queue Name is not provided.]");

                                  }

                        }

                        catch (Exception e) {

                                  e.printStackTrace();

                        }

              }

     

    com.elitecore.messageservice.ejb.base.utils.hornetq.HornetQ.java is a POJO which is being called from an MDB on its onMessage().

     

    If you see problem in this code, pl. provide the reference for the correction.

     

    Thanks.

  • 7. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Yong Hao Gao Master

    //Set Connection

    _cfConnFactory = (ConnectionFactory) _ctx.lookup("/ThroughputConnectionFactory");

    this._conn = _cfConnFactory.createConnection();

     

    This doesn't sound right to me. You should look up something like "JmsXA". like

     

    _cfConnFactory = (ConnectionFactory) _ctx.lookup("/JmsXA");

    this._conn = _cfConnFactory.createConnection();

     

    This way you will get a managed connection. And then each time after the message is sent you should close it. like:

     

    this._conn.close();

     

    This doesn't necessarily close the jms connection underneath but it allows the connection resource to be reused.

     

    Howard

  • 8. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Andy Taylor Master

    once youve mapped the factory like

     

    @Resource(mappedName = "java:JmsXA")

              private ConnectionFactory _cfConnFactory = null;

     

    you dont have to look it up, thi is actually doingg that for you.

     

    In your code you are using a different (non JCA) factory so a new connection is being created for every message processed and then not closed. These are onl;y closed via GC so depending on the VM its liikely you will get memory problems causing timeouts etc

  • 9. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Hushen Savani Newbie

    Thanks a lot Howard and Andy. Issue resolved. I have shifted above code in a Session Bean instead of a POJO and taken care of not looking up the connection factory on each call.

     

    Happy New Year Guys! :-)

  • 10. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
    Hushen Savani Newbie

    Hi Guys,

     

         With reference to the above problem, as Howard and Andy suggested, I have moved message producing logic to a Session Bean from the POJO, so that I will not have to look up the connection factory on each message, and also, I get rid of connection/session/producer switching anti-pattern if I use a session bean. Hence, moved to session bean, all went well. But, when a great load is generated on application, it shows the following exception:

     

    2012-01-08 21:38:16,726 ERROR [com.elitecore.messageservice.ejb.base.utils.logger.EJBLogger] (Thread-610 (group:HornetQ-client-global-
    threads-1848021541)) [ PROCESSING_ENTITY_LISTNER_MDB ] enqueueProcessingEntity():: javax.naming.NameNotFoundExceptionjavax.naming.Name
    NotFoundException: HornetQSessionBean not bound
    

     

    How come this session bean automatically gets unbound if I produce so many messages using this session bean?

     

    I doubt that session bean pools are exhausted if I look up this session bean for producing messages heavily.

     

    Please suggest your pointers on the same... Thanks!