0 Replies Latest reply on May 31, 2012 3:03 PM by samhendley

    InvalidDestinationException using temporary queues in hornetQ cluster

    samhendley

      Hello, I am working on some POC projects using spring integration and hornetQ. I built out my example and it works perfectly when I have a single node serving my requests. As soon as I add another jboss instance with a clustered hornetQ instance I start getting failures.

       

       

      Here is the exception thrown on the handler side, it appears that it cannot publish back to the temporary queue. This error only occurs on the other node that didn't issue the request.

       

      <pre>

      18:38:04,391 WARN  [org.springframework.jms.listener.DefaultMessageListenerContainer] (org.springframework.jms.listener.DefaultMessageListenerContainer#0-9) Setup of JMS message listener invoker failed for destination 'testQueue2' - trying to recover. Cause: Destination dbe17fa4-587a-46bf-9202-9d381b7c99c7 does not exist

      18:38:04,402 INFO  [stdout] (Thread-3 (HornetQ-client-global-threads-15240740)) Removing clusterName=jms.tempqueue.221f172b-9e38-4453-a3d4-a8cf329976bfd53508e9-a9db-11e1-bb73-080027000cb0 on ClusterConnectionImpl@9262213[nodeUUID=199cc33b-aa68-11e1-9329-001aa0397847, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192-168-100-5, address=jms, server=HornetQServerImpl::serverUUID=199cc33b-aa68-11e1-9329-001aa0397847]

      18:38:04,408 INFO  [stdout] (Thread-3 (HornetQ-client-global-threads-15240740)) Removing clusterName=jms.tempqueue.157dc876-9911-4a33-ac54-6e6f2aabe641d53508e9-a9db-11e1-bb73-080027000cb0 on ClusterConnectionImpl@9262213[nodeUUID=199cc33b-aa68-11e1-9329-001aa0397847, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192-168-100-5, address=jms, server=HornetQServerImpl::serverUUID=199cc33b-aa68-11e1-9329-001aa0397847]

      18:38:04,421 INFO  [stdout] (Thread-3 (HornetQ-client-global-threads-15240740)) Removing clusterName=jms.tempqueue.278b801d-8269-46e5-9658-0d176cab6b8fd53508e9-a9db-11e1-bb73-080027000cb0 on ClusterConnectionImpl@9262213[nodeUUID=199cc33b-aa68-11e1-9329-001aa0397847, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192-168-100-5, address=jms, server=HornetQServerImpl::serverUUID=199cc33b-aa68-11e1-9329-001aa0397847]

      18:38:04,428 WARN  [org.springframework.jms.listener.DefaultMessageListenerContainer] (org.springframework.jms.listener.DefaultMessageListenerContainer#0-13) Execution of JMS message listener failed, and no ErrorHandler has been set.: javax.jms.InvalidDestinationException: Destination 07cf88df-4991-4c66-b440-97a177c4ba68 does not exist

              at org.hornetq.jms.client.HornetQSession.createProducer(HornetQSession.java:337)

              at org.springframework.integration.jms.ChannelPublishingJmsMessageListener.sendReply(ChannelPublishingJmsMessageListener.java:390) [spring-integration-jms-2.1.0.RELEASE.jar:]

              at org.springframework.integration.jms.ChannelPublishingJmsMessageListener.onMessage(ChannelPublishingJmsMessageListener.java:294) [spring-integration-jms-2.1.0.RELEASE.jar:]

              at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:535) [spring-jms-3.0.7.RELEASE.jar:3.0.7.RELEASE]

              at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495) [spring-jms-3.0.7.RELEASE.jar:3.0.7.RELEASE]

              at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) [spring-jms-3.0.7.RELEASE.jar:3.0.7.RELEASE]

              at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) [spring-jms-3.0.7.RELEASE.jar:3.0.7

      </pre>

       

       

      On the issuing side here is the timeout exception we get (included just for context of how request was getting done)

       

      <pre>

      19:10:52,900 WARN  [org.springframework.integration.ws.SimpleWebServiceInboundGateway] (http--0.0.0.0-8080-28) failure occurred in gateway sendAndReceive: org.springframework.integration.MessageTimeoutException: failed to receive JMS response within timeout of: 5000ms

              at org.springframework.integration.jms.JmsOutboundGateway.handleRequestMessage(JmsOutboundGateway.java:389) [spring-integration-jms-2.1.0.RELEASE.jar:]

              at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:97) [spring-integration-core-2.1.0.RELEASE.jar:]

              at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) [spring-integration-core-2.1.0.RELEASE.jar:]

              at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source) [:1.6.0_20]

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_20]

              at java.lang.reflect.Method.invoke(Method.java:616) [rt.jar:1.6.0_20]

      </pre>

       

       

      It appears to work most of the time, looks like it fails around 5% of the time the request is handled on the other node. I have found this ticket from 2010 that seems like a likely culprit https://issues.jboss.org/browse/HORNETQ-286.

       

       

      I can't imagine that a bug this obvious is still sitting around so I assume I must be doing something stupid. Any recommendations for what I should be looking for to help figure this out?