10 Replies Latest reply on May 8, 2013 8:13 AM by ataylor

    Setting up a message cluster with single consumer (grouped?)

    mobi-tim

      I've been struggling with HornetQ for the past few days. Read the manual, studied the examples etc.

      But I can't seem to get the results i'm wishing for:

       

      We've got several servers with hornetq as a node sharing a queue (testqueue).

      Messages can be produced and consumed on the same server/nodes (through JMS / Spring). This is working fine.

      But we would like to consume all queues with a single consumer on a different server without having to connect to each individual server.

       

      I figured hornetq clustering would give us one big virtual queue (which could be consumed by one consumer), but what we got are different queues who are reached by a 'round robin' like system and filled individually.

      Now we also have to connect to all the queues with multiple consumers (?) Which is not wat we want.

       

      So after a lot of research I tried to automatically send all the messages from the different queues to one single queue. So we can use only one consumer (or more) but this consumer would always get all the messages available in the cluster.

      I used the clustered group-example. But when I send a message to the HornetQ instance having configured:

       

      <grouping-handler name="my-grouping-handler">

            <type>REMOTE</type>

            <address>jms</address>

            <timeout>5000</timeout>

      </grouping-handler>

       

       

      We get this error:

      java.lang.IllegalStateException: no response received from group handler for Group-0.jms.queue.testQueue

       

      Doing the same on the server with the group handler:

         <type>LOCAL</type>

      Will keep the message locally and not send it to another HornetQ server.. but will not give us the above error.

       

      I can't find *anything* on this problem, so this is a death end for now.

       

      So I tried some stuff from another topic I found

      But configuring it like that will keep all messages locally on the servers, they are not send to the queue in the cluster which do have a consumer ready.

       

      I can not bend my mind around this issue, there must be a solution for this? Are there any pointers in the right direction? Or are we using the wrong queue technology for our issue?

        • 1. Re: Setting up a message cluster with single consumer (grouped?)
          jbertram

          So your main goal is to, "consume all queues with a single consumer on a different server without having to connect to each individual server," right?

           

          In order to do that just make sure all your nodes are clustered properly and have a non-zero redistribution-delay.

          • 2. Re: Setting up a message cluster with single consumer (grouped?)
            mobi-tim

            This is correct. And i'm already testing with these settings.

            But if i produce a message on one of the other servers (using the XML configuration attched to this reply) the messages are not showing up on the server currently consuming.

            Only the messages who are produced locally are consumed.

             

            Maybe i'm missing something, but these are all code snippets used at the moment:

             

            The code i'm using to consume in spring is:

                    connection = connectionFactory.createConnection();
                    Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
                    messageConsumer = session.createConsumer(testQueue);
                    messageConsumer.setMessageListener(this);
                    connection.start();
            

            This is all part of the class implementing the MessageListener. And the connection factory used is configured in Spring:

                <bean name="connectionFactory" class="org.hornetq.jms.client.HornetQXAConnectionFactory" >
                    <constructor-arg value="false" />
                    <constructor-arg>
                        <bean name="transportConfiguration" class="org.hornetq.api.core.TransportConfiguration">
                            <constructor-arg value="org.hornetq.core.remoting.impl.netty.NettyConnectorFactory" />
                            <constructor-arg>
                                <map key-type="java.lang.String" value-type="java.lang.Object">
                                    <entry key="host" value="127.0.0.1" />
                                    <entry key="port" value="5445" />
                                </map>
                            </constructor-arg>
                        </bean>
                    </constructor-arg>
                </bean>
            

             

            Code snippet to produce a message:

                    Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
                    TextMessage message = session.createTextMessage("hello, this is " + ip.getHostAddress());
                    MessageProducer producer = session.createProducer(testQueue);
                    producer.send(message);        
            

            Using the same Spring configured connection factory.

             

            The HornetQ servers cleary state finding each other (in the log):

            INFO [BridgeImpl]  Bridge ClusterConnectionBridge [...] connected

             


            • 3. Re: Setting up a message cluster with single consumer (grouped?)
              ataylor

              Justin is correct. as far as grouping is concerned, this is just really for pinning consumers to producers

              • 4. Re: Setting up a message cluster with single consumer (grouped?)
                ataylor

                your confid has a redistribution-delay of non zero so messages won't be redistributed

                • 5. Re: Setting up a message cluster with single consumer (grouped?)
                  jbertram

                  Actually my comment was in error.  The redistribution-delays needs to be > -1.  As the manual states, "A delay of zero means the messages will be immediately redistributed."  A delay of "-1" means messages won't be redistributed.  His config looks fine to me.

                   

                  We have an example demonstrating this in the HornetQ distribution.  See here.  Does this example work for you, Tim?

                  • 6. Re: Setting up a message cluster with single consumer (grouped?)
                    ataylor

                    also, try it without Spring first to get it to work, Spring can tend to muddy the waters a bit. you can also use the admin console (or CLI if you are using AS7) to see where the messages actually are and what consumers there are.

                    • 7. Re: Setting up a message cluster with single consumer (grouped?)
                      mobi-tim

                      Yes i can run the example.

                      I do notice there is a consumer on both queues (in the example). Is this mandatory?

                      The documentation also states:

                       

                      It often makes sense to introduce a delay before redistributing as it's a common case that a consumer closes but another one quickly is created on the same queue, in such a case you probably don't want to redistribute immediately since the new consumer will arrive shortly.

                       

                      This implies you would need a consumer disconnecting before the messages are redistrubuted..

                      But in my setup all nodes except for one are consumer-less.

                      • 8. Re: Setting up a message cluster with single consumer (grouped?)
                        ataylor

                        redistribution happens when the last consumer disconnects from a queue or when the queue is empty and a remote consumer joins.

                         

                        And no, it will work with just one consumer. have you inspected the queues to see where messages are and what consumers are available?

                        • 9. Re: Setting up a message cluster with single consumer (grouped?)
                          mobi-tim

                          Since there was more time this week to do another test, we used the 2.3.0-final version this time.

                           

                          But the issue stays the same.

                           

                          Both servers configured in a grouped cluster with the same queues, they see each other (I see the Bridge-ClusterConnectionBridge in the log).

                          Only one of them has a consumer, but the messages will stay on the server on which they were produced.. they are not automatically moved to the other server with the consumer.

                           

                          I checked there status with JMS, and I can see the server building up messages in the queue, and the consumer counter is on 0.

                          The other server keeps consuming messages only produced on the same server, but the cluster isn't moving the messages on the consumerless server to the one with the consumer.

                           

                          I tuned/changed every single option mentioned in the documentation about clustering messages.. but nothing seems to help.

                          forward-when-no-consumers, max-hops, redistribution-delay, they have been set to every different value

                           

                          My only "hope" would be the configuration mentioned in 38.7.2 (chain cluster) of the documentation, it's excatly the configuration im trying to set up (except for them not being in the same subnet, because they are). But this chapter is without any configuration examples. Or can I find them somewhere else?

                          • 10. Re: Setting up a message cluster with single consumer (grouped?)
                            ataylor

                            just a word on grouping, if a consumer is pinned to a specific consumer on a specific node then messages with that group id will always be sent to that node, eveb when the consumer dies. We dont redistribute messages, once messages are re distributed they are out of order so in short you dont redistribute with message grouping, it defeats the purpose. There is a best practice section in the user manual that basically says that consumers should be evenly distributed in a symmetric fashion to avoid this, see http://docs.jboss.org/hornetq/2.3.0.Final/docs/user-manual/html_single/index.html#d0e5706