1 2 Previous Next 16 Replies Latest reply on Jun 2, 2014 9:47 AM by ravi.teja

    HORNETQ-316 Make cluster discovery pluggable

    igarashitm

      https://issues.jboss.org/browse/HORNETQ-316

      http://community.jboss.org/thread/148457

       

      To cluster discovery get work on amazon AWS, we need 2 steps:

      1. make cluster discovery pluggable and add JGroups discovery plugin
      2. verify at least the one of GossipRouter, FILE_PING and S3_PING woks fine on amazon AWS

       

      First I'm considering about step-1. Please review my idea.

       

      • classes to add
        • org.hornetq.core.cluster.impl.DiscoveryGroupFactory
          • : if config has valid "class" attribute, create it. otherwise, create existing DiscoveryGroupImpl
        • org.hornetq.core.server.cluster.impl.BroadcastGroupFactory
          • : if config has valid "class" attribute for broadcast-group, create it. otherwise, create existing BroadcastGroupImpl
        • org.hornetq.integration.discovery.jgroups.JGroupsDiscoveryGroupImpl
        • org.hornetq.integration.discovery.jgroups.JGroupsBroadcastGroupImpl

      • behavior changes on existing classes
        • org.hornetq.core.client.impl.ServerLocatorImpl#initialize() call DiscoveryGroupFactory#createDiscoveryGroup() instaed of call new DiscoveryGroupImpl() directly.
        • org.hornetq.core.server.cluster.impl.ClusterManagerImpl#deployBroadcastGroup() call BroadcastGroupFactory#createBroadcastGroup() instead of call new BroadcastGroupImpl() directly.

       

      • hornetq-configuration.xml changes
        •    <broadcast-groups>

                <broadcast-group name="my-broadcast-group">

                   <param key="class" value="org.hornetq.integration.discovery.jgroups.JGroupsBroadcastGroupImpl"/>

                   <param key="jgroups-channel-name" value="hornetq-discovery"/>

                   <!-- attributes for BroadcastGroupImpl

                   <param key="group-address" value="231.7.7.7"/>

                   <param key="group-port" value="9876"/>

                   <param key="broadcast-period" value="100"/>

                   <param key="connector-ref" value="netty-connector"/>

                   -->

               </broadcast-group>

             </broadcast-groups>

             <discovery-groups>

                <discovery-group name="my-discovery-group">

                   <param key="class" value="org.hornetq.integration.discovery.jgroups.JGroupsDiscoveryGroupImpl"/>

                   <param key="jgroups-channel-name" value="hornetq-discovery"/>

                   <!-- attributes for DiscoveryGroupImpl

                   <param key="group-address" value="231.7.7.7"/>

                   <param key="group-port" value="9876"/>

                   <param key="refresh-timeout" value="10000"/>

                   -->

                </discovery-group>

             </discovery-groups>

       

      • TODOs
        • abstraction of org.hornetq.api.core.DiscoveryGroupConfiguration
          • currently, tightly coupled with existing DiscoveryGroupImpl
        • consider where to put the jgroups stack configuration for client, and standalone server
          • in JBossAS, we can put it in deploy/cluster/jgroups-channelfactory.sar/META-INF/jgrouls-channelfactory-stacks.xml

       

       

      any advice would be appreciated

       

      Thanks

        • 1. HORNETQ-316 Make cluster discovery pluggable
          keteracel

          here's a patch which uses JGroups for discovery rather than custom multicast. (http://paste.pocoo.org/show/318348/)

           

          Two observations:

           

          1) I think that as JGroups deals with a Channel (rather than discovery and broadcast), DiscoveryGroup and BroadcastGroup should be merged, with an option to turn broadcast off for clients.

          2) This could be completely pluggable so people can use whatever they like.

           

          So this patch just replaces the original multicast with jgroups which is enough for me to continue, allows multicast for people who want it, but ties people to jgroups (not too bad an idea).

           

          Thanks,

           

          Paul.

          • 2. HORNETQ-316 Make cluster discovery pluggable
            ataylor

            theres also static discovery that should be factored in, currently this is separate code in server locator (see StaticConnector), this basically takes a list of servers and tries them all at the same time and waits for one to connect.

            • 3. HORNETQ-316 Make cluster discovery pluggable
              clebert.suconic

              People can use JGropus as a plugin. We don't want the dependency on hornetq core.

              • 4. HORNETQ-316 Make cluster discovery pluggable
                clebert.suconic

                What you're suggesting is that StaticConnector should also be a plugin point, right?

                 

                 

                It makes sense.

                • 5. HORNETQ-316 Make cluster discovery pluggable
                  ataylor

                  Clebert Suconic wrote:

                   

                  What you're suggesting is that StaticConnector should also be a plugin point, right?

                   

                   

                  It makes sense.

                  correct

                  • 6. HORNETQ-316 Make cluster discovery pluggable
                    igarashitm

                    OK, update to deal with StaticConnector and config abstraction:

                     

                    • classes to add
                      • org.hornetq.core.cluster.DiscoveryGroupConfiguration
                        • has method : String getClassName()
                      • org.hornetq.core.cluster.impl.StaticDiscoveryGroupConfigurationImpl extends DiscoveryGroupConfiguration
                      • org.hornetq.core.cluster.impl.SimpleUDPDiscoveryGroupConfigurationImpl extends DiscoveryGroupConfiguration
                      • org.hornetq.core.cluster.impl.DiscoveryGroupFactory
                        • has method : DiscoveryGroup createDiscoveryGroup(DiscoveryGroupConfiguration)
                          • return Class.forName(DiscoveryGroupConfiguration#getClassName()).newInstance()
                      • org.hornetq.core.server.cluster.BroadcastGroupConfiguration
                        • has method : String getClassName()
                      • org.hornetq.core.server.cluster.impl.SimpleUDPBroadcastGroupConfigurationImpl extends BroadcastGroupConfiguration
                      • org.hornetq.core.server.cluster.impl.BroadcastGroupFactory
                        • has method : BroadcastGroup createBroadcastGroup(BroadcastGroupConfiguration)
                          • return Class.forName(BroadcastGroupConfiguration#getClassName()).newInstance()
                      • org.hornetq.integration.discovery.jgroups.JGroupsDiscoveryGroupConfigurationImpl extends DiscoveryGroupConfiguration
                      • org.hornetq.integration.discovery.jgroups.JGroupsBroadcastGroupConfigurationImpl extends BroadcastGroupConfiguration
                      • org.hornetq.integration.discovery.jgroups.JGroupsDiscoveryGroupImpl implements DiscoveryGroup
                      • org.hornetq.integration.discovery.jgroups.JGroupsBroadcastGroupImpl implements BroadcastGroup

                     

                    • behavior changes on existing classes
                      • ServerLocator call DiscoveryGroupFactory#createDiscoveryGroup() instaed of call new DiscoveryGroupImpl() directly
                      • ClusterManagerImpl call BroadcastGroupFactory#createBroadcastGroup() instead of call new BroadcastGroupImpl() directly
                      • add DiscoveryGroup#selectConnector()
                      • remove HornetQClient#createServerLocator{With,Without}HA(TransportConfiguration...)
                        • use HornetQClient#createServerLocator{With,Without}HA(DiscoveryGroupConfiguration) instead. pass the StaticDiscoveryGroupConfiguration as param.

                     

                    • hornetq-configuration.xml changes
                      • same as I posted first

                     

                     

                    I think that solve with above... anything else I should consider ?

                     

                    Thanks.

                    • 7. HORNETQ-316 Make cluster discovery pluggable
                      keteracel

                      That sounds great to me!

                       

                      Any ETA on when this work will be done? If you would like contributions, I'm happy to help.

                       

                      Thanks,

                       

                      Paul.

                      • 8. HORNETQ-316 Make cluster discovery pluggable
                        igarashitm

                        I'm now working on http://anonsvn.jboss.org/repos/hornetq/branches/HORNETQ-316/

                        I've just finished to write a prototype, and I'll test & debug from today. any comment would be appreciated.

                         

                        Thanks.

                        • 9. HORNETQ-316 Make cluster discovery pluggable
                          keteracel

                          Awsome.

                           

                          My only question is regarding how this works with JNDI. It looks like it'll require the jgroups config file to be in the same location on subsequent hosts that use JNDI for configuration. When I created my 'hacked' version I pushed the jgroups config file contents into JNDI so you could remove the config dependency. Not sure if that was the right thing, but thought I'd hit you up anyways.

                           

                          But yes, this is looking pretty spot on what I'd need!

                           

                          Thanks,

                           

                          Paul.

                          • 10. HORNETQ-316 Make cluster discovery pluggable
                            keteracel

                            I'm also wondering if it's worth collapsing Broadcast and Discovery into a single Class with an option to join the cluster (broadcast) or just discover the cluster. It seems that it would be more optimal to use the JChannel as a channel rather than having two of them - one for broadcast and one for discovery.

                             

                            That may be a future refactoring though

                            • 11. Re: HORNETQ-316 Make cluster discovery pluggable
                              bbansal

                              Folks,

                               

                              Is it pushed to trunk yet ? I am running hornetq-2.2.5 can it be back ported to it ?

                              • 12. Re: HORNETQ-316 Make cluster discovery pluggable
                                keteracel

                                When is this targetted for release?

                                 

                                We didn't go with HornetQ for my last project because of this issue. I'd really like to use HornetQ on my current project but again we can't rely on multicast for discovery and a static list = downtime.

                                 

                                I really appreciate the work you are doing to add this to HornetQ and am eager to start using it!

                                 

                                Paul.

                                • 13. Re: HORNETQ-316 Make cluster discovery pluggable
                                  bbansal

                                  Well,

                                   

                                  We at Groupon are going in production with a simple design which solve this problem.

                                   

                                  Producer --> Load-balancers ---> Cluster of HornetQ servers 

                                   

                                  The producer client refreshes connection every 5 mins internally and hence any new machine added or deleted is handled easily.

                                   

                                  Consumer  ------>

                                                    ------>      All hornetQ servers and consume from all

                                   

                                  Consumer connect to all HornetQ servers in parallel and consume from all, The consumer client reads a list of active servers and refresh connection every 5 mins etc.

                                   

                                  Each hornet-q machine in cluster is more or less unaware of others and performs wonderfullly. We have clients in java/ruby let me know if there is interest in open sourcing these out.

                                  • 14. Re: HORNETQ-316 Make cluster discovery pluggable
                                    keteracel

                                    That is a valid workaround. However, one of the reasons we want a HornetQ cluster is for the clustering and loadbalancing without the need for another piece of infrastructure / more network hops.

                                    1 2 Previous Next