We are trying to integrate different parts of our system written in different languages using a messaging backbone. We have chosen HornetQ as a message broker and running it under the JBoss AS 7.0.2.
On a very high level the system looks like as follows:
- Server1 (Delphi) is listening on Queue1 and sends to Queue2 via STOMP
- Server2 (Java) is listening on Queue2 and sends to Queue1 via native HornetQ protocol using core API (not JMS)
The problem is that we are experiencing weird disconnects and suspect the STOMP side. The weirdness is in the fact that we do not see any error messages in logs, but HornetQ is creating something which looks very much like a second buffer/consumer on the Queue1 which results in Server1 receiving every second message which obviously is breaking the application level protocol. Please forgive me if I’m not explaining myself clearly. Has anyone been experiencing something similar? Are there any suggestions as to how we could diagnose/troubleshoot the problem to better understand of what is going on?
JBoss xml for HornetQ looks like this:
<netty-acceptor name="netty" socket-binding="messaging"/>
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
<in-vm-acceptor name="in-vm" server-id="0"/>
<netty-acceptor name="stomp-acceptor" socket-binding="stomp">
<param key="protocol" value="stomp"/>
Sometimes when we restart jBoss we see messages like this:
07:50:55,657 WARN [org.jboss.messaging] (MSC service thread 1-7) failed to destroy jms queue: RuleEngineOutQueue: HornetQException[errorCode=104 message=Cannot delete queue jms.queue.RuleEngineOutQueue on binding jms.queue.RuleEngineOutQueue - it has consumers = org.hornetq.core.postoffice.impl.LocalQueueBinding]
at org.hornetq.core.server.impl.HornetQServerImpl.destroyQueue(HornetQServerImpl.java:1083) [hornetq-core-2.2.7.Final.jar:]
at org.hornetq.jms.server.impl.JMSServerManagerImpl.destroyQueue(JMSServerManagerImpl.java:710) [hornetq-jms-2.2.7.Final.jar:]
at org.jboss.as.messaging.jms.JMSQueueService.stop(JMSQueueService.java:68) [jboss-as-messaging-7.0.2.Final.jar:7.0.2.Final]
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1909) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1872) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_26]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_26]
at java.lang.Thread.run(Thread.java:662) [:1.6.0_26]
I can understand the meaning of this, but can this add to our troubles?
I’d highly appreciate ANY idea/insight on this. I’m at lost, but the issue is very time-critical.
I'm not sure how's the integration on AS7, but I believe it will call destroy once the queue is stopped. Destroy doesn't mean delete the queue messages, but it means undeploy...etc.
It seems that there's a dependency issue here, since Stomp should be stopped before these queues, since Stomp could be holding consumers, causing the issue you are seeing.
I guess you should open a JIRA on AS7 project.
Workaround would be to kill the server when using stomp (if AS7)
Just a heads up for the community.
Our STOMP connectivity issues have gone away 3 weeks ago since we switched off to ActiveMQ. Also ActiveMQ has a very simple yet powerful administration console. I cannot imagine MQ-related development without having such a tool. Just a glimpse on the screen gives you a clear picture as to what is going on in your system in communication area. And yes, I know that there is an extensive management API in HornetQ - we are very close to creating our own monitoring/testing tool.
To HornetQ team: our team is ready to help with all means in helping to resolve the STOMP connectivity issue as we DO want to use JBoss AS in our implementations including integration of the heterogeneous systems. For non-Java application the STOMP protocol perhaps the only option here.