1 2 Previous Next 18 Replies Latest reply on Feb 11, 2010 5:57 AM by timfox

    Clustering a Topic

    schwanitzb

      I have my application working with a topic message producer in JBoss and a simple stand-alone topic message consumer running on the same machine. Here are code snippets:

       

      JBoss message producer:

       

      public static void sendMessage(String topicName, String connectionFactoryName, Map<String, Object> msgContents) throws WebFontJMSException {

       

      if (null==topicName || topicName.length()==0) {

       

      throw new IllegalArgumentException("topicName cannot be null or empty");

      }

       

      if (null==connectionFactoryName || connectionFactoryName.length()==0) {

       

      throw new IllegalArgumentException("connectionFactoryName cannot be null or empty");

      }

       

      if (null==msgContents || msgContents.size()==0) {

       

      throw new IllegalArgumentException("msgContent cannot be null or empty");

      }

       

       

      try {

      Context initialContext =

      new InitialContext();

       

       

      try {

      String topicJNDIName = Topics.

      TOPIC_JNDI_PREFIX + topicName;

      Topic topic = (Topic) initialContext.lookup(topicJNDIName);

       

      TopicConnectionFactory topicConnectionFactory = (TopicConnectionFactory) initialContext.lookup(connectionFactoryName);

       

      TopicConnection topicConnection = topicConnectionFactory.createTopicConnection();

       

       

      try {

      TopicSession topicSession = topicConnection.createTopicSession(

      false, Session.AUTO_ACKNOWLEDGE);

       

      javax.jms.TopicPublisher topicPublisher = topicSession.createPublisher(topic);

       

      sendMessage(topicSession, topicPublisher, msgContents);

      }

      finally {

      topicConnection.close();

      }

      }

       

      finally {

      initialContext.close();

      }

      }

       

      catch (NamingException ne) {

       

      throw new WebFontJMSException(ne);

      }

       

      catch (JMSException jmse) {

       

      throw new WebFontJMSException(jmse);

      }

      }

       

       

      Standalone message consumer:

      public

       

       

      class TopicListenerStandalone implements MessageListener {

       

      private static TopicConnectionFactory topicConnectionFactory = null;

       

       

      private TopicListenerIF topicListener = null;

       

      private TopicConnection topicConnection = null;

       

      private TopicSession topicSession = null;

       

      public TopicListenerStandalone(String topicName, String jmsHostName, TopicListenerIF topicListener) throws WebFontJMSException {

       

      if (null==topicName || topicName.length()==0) {

       

      throw new IllegalArgumentException("topicName cannot be null or empty");

      }

       

      if (null==jmsHostName || jmsHostName.length()==0) {

       

      throw new IllegalArgumentException("jmsHostName cannot be null or empty");

      }

       

      if (null==topicListener) {

       

      throw new IllegalArgumentException("topicListener cannot be null");

      }

       

      Map<String, Object> connectionParams =

      new HashMap<String, Object>();

      connectionParams.put(TransportConstants.

      HOST_PROP_NAME, jmsHostName);

      connectionParams.put(TransportConstants.

      PORT_PROP_NAME, 5445);

      TransportConfiguration transportConfiguration =

      new TransportConfiguration(NettyConnectorFactory.class.getName(), connectionParams);

       

      topicConnectionFactory = (TopicConnectionFactory) new HornetQConnectionFactory(transportConfiguration);

       

      if (null == topicConnectionFactory) {

       

      throw new WebFontJMSException("HornetQ initialization failed");

      }

       

       

      boolean success = false;

       

       

      this.topicListener = topicListener;

       

       

      try {

       

      // Directly instantiate the JMS Topic object.

      Topic topic =

      new HornetQTopic(topicName);

       

       

      // Create a JMS TopicConnection

       

      topicConnection = topicConnectionFactory.createTopicConnection();

       

       

      try {

       

      // Create a JMS TopicSession

       

      topicSession = topicConnection.createTopicSession(false, Session.AUTO_ACKNOWLEDGE);

       

       

      // Create a JMS topic subscriber with this instance as the listener

      TopicSubscriber topicSubscriber =

      topicSession.createSubscriber(topic);

      topicSubscriber.setMessageListener(

      this);

       

       

      // Start the Connection

       

      topicConnection.start();

       

      success =

      true;

      }

      finally {

       

      if (!success) {

       

      topicConnection.close();

       

      topicConnection = null;

      }

      }

      }

       

      catch (JMSException jmse) {

       

      throw new WebFontJMSException(jmse);

      }

      }

       

       

       

       

      public void onMessage(Message msg) {

       

      try {

       

      // We know this is a MapMessage, so let's cast it here

      MapMessage mapMsg = (MapMessage) msg;

       

      Map<String, Object> msgContents =

      new HashMap<String, Object>();

       

      for (Enumeration<String> e=(Enumeration<String>)mapMsg.getMapNames(); e.hasMoreElements();) {

      String mapName = e.nextElement();

      msgContents.put(mapName, mapMsg.getObject(mapName));

      }

       

      topicListener.onMessage(msgContents);

      }

       

      catch (Throwable t) {

       

      // This should never throw anything, so we'll eat all exceptions here

      t.printStackTrace();

      }

      }

       

       

       

      public void close() throws WebFontJMSException {

       

      try {

       

      topicConnection.close();

      }

       

      catch (JMSException jmse) {

       

      throw new WebFontJMSException(jmse);

      }

      }

      }

       

       

      However, I'm not able to get these same producers and consumers to work when I set up a 2 machine cluster. I run JBoss on two machines using "run.bat -c all-with-hornetq". I can see that the bridges seem to properly connect the two machines. I then start a stand-alone consumer on each of the two machines. When the message producer in JBoss creates a message in the topic, only the consumer on the same machine as the producer sees the message. I would expect that the consumers on BOTH machines would see the message.

       

      Thank you for any help you can offer.

       

      Cheers,

      Bill

       

       

        • 1. Re: Clustering a Topic
          timfox

          It's hard to see what you're trying to do by looking at your pasted code.

           

          Can you attach an actual working test program?

           

          Also, pls attach your server config.

          • 2. Re: Clustering a Topic
            schwanitzb

            Tim,

             

            Thanks for the extremely fast response!

             

            It took me a bit to get together a simple applet to repro.

             

            The server configuration I'm running is:

            1. Two JBoss 5.1.0.GA servers (one on Mac one on Win).

            2. The only tweak from the default hornetq configuration is the addition of the following block to jboss-5.1.0.GA\server\all-with-hornetq\deploy\hornetq.sar\hornetq-jms.xml:

               <topic name="exampleTopic">
                  <entry name="/topic/exampleTopic"/>
               </topic>

            I have attached a test producer & consumer named jmster.

             

            Here are the repro steps:

            1. Start jboss on two machines ("run.bat/.sh -c all-with-hornetq")

            2. Unzip the attached jmster.zip to a temp location on both machines

            3. Run two jmster instances (one on each machine) subscribed to exampleTopic:

                 - cd templocation/Common/client/jmster/build

                 - java -jar jmster.jar -mode subscribe -topic exampleTopic -host localhost

            4. On either machine use jmster to publish a message to exampleTopic:

                 - cd templocation/Common/client/jmster/build

                 - java -jar jmster.jar -mode publish -topic exampleTopic -file ..\testMessage.properties -host dummy

                 - NOTE: sorry about the dummy -host param - it is required but ignored by the app

             

            I am only seeing the message delivered to the subscriber on the local machine on which the message was published. I'd expect to see it delivered to both subscribers.

             

            Thanks so much for the help,

            Bill

            • 3. Re: Clustering a Topic
              clebert.suconic

              You didn't mention anything about Cluster Connections in your configuration:

               

               

              http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html_single/index.html#clusters.cluster-connections

               

               

              Maybe we should add a default cluster connection for jms addresses on the all configuration, which don't currently do.

               

               

              If you look at the clustered-topic example you will see this:

               

               

                 <cluster-connections>
                    <cluster-connection name="my-cluster">
                       <address>jms</address>
                            <discovery-group-ref discovery-group-name="dg-group1"/>
                    </cluster-connection>
                 </cluster-connections>
               
              
              • 4. Re: Clustering a Topic
                schwanitzb

                Celbert,

                 

                Thanks to you for the fast reply!

                 

                By default, there appears to be a <cluster-connection> just as you described it defined in hornetq.sar/hornetq-configuration.xml in the all-with-hornetq configuration. I've attached that file.

                 

                Cheers,

                Bill

                • 5. Re: Clustering a Topic
                  clebert.suconic

                  Try setting forward-when-no-consumers = true, which the default is false.

                   

                  We usually recommend it false. It's best if you keep the message on the same node. If you load balance every message sent on the node to the other node, it won't scale (It's best to consume locally).

                   

                   

                  Also, Read the documentation at http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html_single/index.html#clusters.cluster-connections

                  • 6. Re: Clustering a Topic
                    schwanitzb

                    Celbert,

                     

                    I have already tried setting forward-when-no-consumers=true. That did not seem to help.

                     

                    BTW: This is a topic that I'm trying to share across a cluster. I want messages added to the topic to be propogated to all nodes, and therefore all subscribers, in the cluster. Wouldn't this be a normal use/expectation for a topic?

                     

                    Cheers,

                    Bill

                    • 7. Re: Clustering a Topic
                      clebert.suconic

                       

                      Celbert,

                       

                       

                       

                      did you mean, Clebert? ;-)

                       

                       

                      BTW: This is a topic that I'm trying to share across a cluster. I want messages added to the topic to be propogated to all nodes, and therefore all subscribers, in the cluster. Wouldn't this be a normal use/expectation for a topic

                       

                      Duh... yes.. I made a confusion about it.

                       

                       

                      Are you sure you have proper UDP connectivity between the two nodes?

                       

                      Can you try copying the cluster-connection from the topic example

                       

                       

                       

                           <cluster-connection name="my-cluster">
                               <address>jms</address>
                               <retry-interval>500</retry-interval>
                               <use-duplicate-detection>true</use-duplicate-detection>
                               <forward-when-no-consumers>true</forward-when-no-consumers>
                               <max-hops>1</max-hops>
                               <discovery-group-ref discovery-group-name="my-discovery-group"/>
                            </cluster-connection>
                       
                      
                      • 8. Re: Clustering a Topic
                        schwanitzb

                        Clebert,

                         

                        So sorry about misspelling your name!

                         

                        I tried using the full <cluster-connection> you suggested in your last email - no change.

                         

                        Do you have a suggestion for a tool I could use to check if I have UPD connectivity between the nodes? BTW: one node is Mac the other is Win.

                         

                        Do you know if there is a way (jmx-console, etc.) to test if a HornetQ server "knows" about another HornetQ server?

                         

                        Does a log entry such as:

                        11:00:45,281 INFO  [BridgeImpl] Connecting bridge sf.my-cluster.17af35cf-07a9-11df-acb3-e158620474bb to its destination

                        11:00:46,575 INFO  [BridgeImpl] Bridge sf.my-cluster.17af35cf-07a9-11df-acb3-e158620474bb is connected to its destination

                        indicate that the nodes have UDP connectivity?

                         

                        Thanks for your help,

                        Bill

                        • 9. Re: Clustering a Topic
                          schwanitzb

                          Clebert,

                           

                          I realized I could use wireshark (installed on my Win machine) to verify UDP packets. I AM able to see UDP packets from the Mac arriving on my Win machine. So it looks like UDP connectivity is OK. Would it help if I posted the contents of the UPD packets? Essentially I am seeing a set of packets from each machine on 5 second intervals.

                           

                          Cheers,

                          Bill

                          • 10. Re: Clustering a Topic
                            timfox

                            I'd need to see all your server config, not just a snippet from one node.

                             

                            Also... does the clustered topic example in the distro work for you?

                             

                            If so, try and work out what the difference is between that and your setup.

                            • 11. Re: Clustering a Topic
                              timfox

                              You don't need wireshark to see if the nodes have formed a cluster.

                               

                              Just go the jmx-console and invoke getNodes() on the ClusterConnectionControl

                              • 12. Re: Clustering a Topic
                                schwanitzb

                                Tim,

                                 

                                I have attached a zip of all the files in the hornetq.sar directory. Please let me know if you need any other configuration files.

                                 

                                Yes, the clustered-topic example in the distro DOES work. However, please correct me if I'm wrong, it runs 2 stand-alone servers and the publisher/subscriber app on the same machine. I can't find any description in the docs on how to split it so the 2 servers are running on different machines.

                                 

                                Thanks for your help,

                                Bill

                                • 13. Re: Clustering a Topic
                                  schwanitzb

                                  Tim,

                                   

                                  Thanks for the tip on using the ClusterConnectionControl. I'm running HornetQ 2.0.0.GA and I don't see a getNodes() method (just stop and start). However, maybe this will be helpful. I've attached the outputs from the ClusterConnectionControls on both machines.

                                   

                                  Note: the IP address of the Win machine is 192.168.1.100, and the Mac is 192.168.1.104. Does the jmx output look to you like the machines are forming connections to themselves? BTW: the Nodes map looked like "{}" on the Win machine until I brought up JBoss on the Mac.

                                   

                                  Thanks for your help,

                                  Bill

                                  • 14. Re: Clustering a Topic
                                    timfox

                                    Your config doesn't look right.

                                     

                                    You can't specify a connector to connect to an address "0.0.0.0". When creating a TCP connection from a client the client needs to know the real ip address.

                                     

                                    This would explain why your cluster isn't forming.

                                     

                                    Try specifying the real ip addresses.

                                     

                                    BTW there was another thread on almost exactly this on this forum just a few days ago.

                                    1 2 Previous Next