10 Replies Latest reply on Aug 4, 2011 4:15 AM by ataylor

    Need help understanding clustering

    dspiess

      We're trying to set up a HornetQ stand-alone symmetric cluster, and it's not behaving quite like we expected.

       

      We have 4 nodes set up in our JMS queue cluster (A, B, C ,D).  When we were using JNDI to look up server A, we can post messages to it and they get balanced across the four nodes.  So if we sent 1000 messages, each would have 250 which is what we expected. 

       

      When we use JNDI to consume the messages like below, we have to run it 4 times to pull the messages.

       

      initialContext = hornetConsumer.getContext(0);
      ConnectionFactory cf = (ConnectionFactory)initialContext.lookup("/ConnectionFactory");
      Queue queue = (Queue)initialContext.lookup("/queue/Queue");
      connection = cf.createConnection();
      Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
      MessageConsumer messageConsumer = session.createConsumer(queue);
      connection.start();
      while ((messageReceived = (TextMessage) messageConsumer.receive(5000)) != null) {
           // do stuff with message
      } 
      

       

      Each time we run this, we get 250 messages.  This seems to make sense since we are connecting to server A, which is forwarding the connection to one server. Right?

       

      We thought if we'd change the code to use the DiscoveryGroup to create a connection, we'd get all 1000 messages.  We'd like to treat the cluster as one "super queue".

       

      We changed our code to

       

      DiscoveryGroupConfiguration groupConfiguration = new DiscoveryGroupConfiguration("231.7.7.7", 9876);
                        ConnectionFactory cf = (ConnectionFactory)HornetQJMSClient.createConnectionFactoryWithoutHA(groupConfiguration, JMSFactoryType.CF);
      Thread.sleep(2000);
      Queue queue = HornetQJMSClient.createQueue("Queue");
      connection = cf.createConnection();
      Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
      MessageConsumer messageConsumer = session.createConsumer(queue);
      connection.start();
      while ((messageReceived = (TextMessage) messageConsumer.receive(5000)) != null) {
           // do stuff with message
      }
      

       

      This pulled messages again from one queue at a time, but one queue was always ignored.  Since we were using the DiscoveryGroup, we thought we'd be connecting to the queues as a group.

       

      So I have the following questions:

       

      1)  Is the DiscoveryGroup code doing the same thing as the JNDI code, and just connecting to one server at time?

      2)  Why does the second code listing only hit three of the servers in the cluster for messages?  If we post 1000 messages, we get 250 x 3 back.

      3)  Is there a way to pull from a queue cluster and get 1000 messages when we post 1000 messages?

        • 1. Re: Need help understanding clustering
          dspiess

          To add one more thing, we double checked and in the second example we are losing the messages.  If we place 1000 messages on, we get 750 back.

          • 2. Re: Need help understanding clustering
            clebert.suconic

            Daniel Spiess wrote:

             

            To add one more thing, we double checked and in the second example we are losing the messages.  If we place 1000 messages on, we get 750 back.

            You probably have a consumer in one of the nodes but it's not looking up for the messages.

             

             

            Messages are load balanced to each one of your nodes on the cluster.

            • 3. Re: Need help understanding clustering
              dspiess

              I'm not sure I understand your reply.  Where is the consumer supposed to be looking up?

               

              It looks like the queue cluster is working like this for message consumption:

               

              A consumer says "Hey I need a connection"

              One of the nodes in the cluster says "Here I am"

              They connect

              The consumer says "I want messages"

              The node says "I don't have any" even though other nodes may have messages

               

              Since the code for establishing a connection is the same, I can kind of understand why this is.  It seems though that if one server can respond faster, the others will be ignored.  We tested this and can direct connections to a particular server by increasing the traffic to the other servers.

               

              Is there a way to connect only if the server has messages to consume?

              • 4. Re: Need help understanding clustering
                clebert.suconic

                Messages are load balanced only if there is a consumer on that node.

                 

                You probably created a consumer in a node and you are not consuming from it.

                • 5. Re: Need help understanding clustering
                  dspiess

                  Clebert Suconic wrote:

                   

                  Messages are load balanced only if there is a consumer on that node.

                   

                  That doesn't seem to match what we're seeing.  We have a cluster as described above (symmetric-standalone-x4).  We run a program which is very similar to the examples, but we can control the number of messages.  We send 1000 messages.  At this point no consumer has connected to the cluster.  There is no consumer app running, and no consumer code in the producer app.

                   

                  Then at a later time, we run a consumer app which pulls messages off the queue.  When we connect using JNDI directly to a server we pull down 250, each time we run the consumer app.  If we connect to the cluster by defining a discovery group, we get 250 messages from the queue the first time.  The subsequent times depend on if we "got lucky" and connected to a queue with messages.  It seems we are connecting to the first available server, whether it has messages or not.

                   

                   

                  Clebert Suconic wrote:

                   

                  You probably created a consumer in a node and you are not consuming from it.

                   

                  The only consumer we're creating is in the consumer app code.  The program runs, and then ends.  Are you saying there might be a consumer instantiated on the server which isn't pulling messages, just blocking?

                  • 6. Re: Need help understanding clustering
                    clebert.suconic

                    You are connecting to a single node with Discovery Group.

                     

                    What if you didn't use Discovery Group to connect to the consumer. Maybe run a consumer on each node?

                    • 7. Re: Need help understanding clustering
                      dspiess

                      Clebert Suconic wrote:

                       

                      What if you didn't use Discovery Group to connect to the consumer. Maybe run a consumer on each node?

                       

                      We'd like to avoid the consumers knowing which node they are connecting to.  I think we're going to be stuck connecting to a single node, since UDP won't work outside our vlan.

                       

                      That means we're connecting via JNDI to a single node, and getting a fraction of the messages each time. 

                       

                      Is there a code example showing how to get the server your connection factory has returned from the cluster?  We want to make sure the connection factory is returning a different node each time when we get a connection.

                      • 8. Re: Need help understanding clustering
                        ataylor

                        Not sure exactly whats happening but heres a few things to check.

                         

                        When you send the messages they should only be load balanced to nodes where consumers exist unless forwardWhennoConsumers=true, can you verify what you are doing here.

                         

                        secondly once nodes are load balanced they will only be re distributed to other nodes if the redistribution delay (on address settings) is greater than -1 (which is the default).

                         

                        One thing should always try to do is to make your cluster symmetrical, that is, consumers and producers on all nodes as this gives you best performance. Moving messages around the cluster is costly and should be avoided.

                        • 9. Re: Need help understanding clustering
                          dspiess

                          Andy Taylor wrote:

                           

                          Not sure exactly whats happening but heres a few things to check.

                           

                          When you send the messages they should only be load balanced to nodes where consumers exist unless forwardWhennoConsumers=true, can you verify what you are doing here.

                           

                          secondly once nodes are load balanced they will only be re distributed to other nodes if the redistribution delay (on address settings) is greater than -1 (which is the default).

                           

                          Neither were explicitly stated, so we're thinking its forward=true and redist delay = -1?

                           

                          So if messages aren't load balanced unless it has a consumer attached, what's the point of a queue cluster?  Our consumers are just Java apps which will end if there are no messages, so they won't hold permanent connections to the queue node.  Is this the wrong way of think of consumers?  It also seems that consumers are tied to particular nodes in all the examples.  We wanted the consumers to not care which node was responding to a request.  Again, is this the wrong way to think about HornetQs?

                          • 10. Re: Need help understanding clustering
                            ataylor

                            Neither were explicitly stated, so we're thinking its forward=true and redist delay = -1?

                            forward is false by default so messgaes won't be load balanced when no consumers exist.

                             

                            So if messages aren't load balanced unless it has a consumer attached, what's the point of a queue cluster?

                            whats the point of sending messages to a node where there are no consumers, they won't be consumed, load balancing happens round consumers not nodes.

                             

                            Our consumers are just Java apps which will end if there are no messages, so they won't hold permanent connections to the queue node.  Is this the wrong way of think of consumers?

                            That sounds wrong, how are you checking to see if there are messages? just create your consumers and let them wait for messages.

                             

                            It also seems that consumers are tied to particular nodes in all the examples.  We wanted the consumers to not care which node was responding to a request.  Again, is this the wrong way to think about HornetQs?

                            I'm not really sure what you mean here, a consumer connects to a node and stays connected to it.