1 2 Previous Next 18 Replies Latest reply on Apr 20, 2011 2:17 PM by akarl

    Errai in a cluster

    akarl

      Errai is quite nice when you only need to converse with a single server but many scalable web applications take advantage of clustering techniques.  Is this something on the horizon for the Errai message bus?  JBoss ESB integration?

       

      A few other notes and questions

      - Digging into ServerMessageBusImpl, I suspect that clustering support would not be incredibly difficult using either JGroups or JBossCache/Infinispan.

      - I also noticed that it may be possible to have each server instance subscribe to each other(not really scalable) or better yet, subscribe to an elected HASingleton node.  In this way messages could be passed between servers.  Was this approach a design goal or am I hacking at concepts never intended for this use?  I'm leaning towards giving this approach a try.  Does it seem feasible?

        • 1. Re: Errai in a cluster
          heiko.braun

          We don't have it on the roadmap yet. But it maybe a good time to kickoff the discussion.

          My first question would be: Why would you cluster errai? What do you strive for?

           

          - Scalability?

          - Faul tolerance?

           

          Scalability could already be achieved when using load balancing with sticky sessions.

          Faul tolerance (failover) is a different topic.

           

          But regardless of the outcome of this discussion, we should look into leveraging existing

          technologies that ship with JBoss or any other EE5 server.

          I..e before diving into the low level details like jgroups, we should evaluate usage of

          JMS, HTTP Session replication, etc.

           

          I am pretty sure we can find a suitable approach, without implementing any of the low level bits ourselves.

          • 2. Re: Errai in a cluster
            akarl

            In my case we cluster for both scalability and high availability.

             

            I don't entirely agree that scalability can be achieved using sticky sessions unless you assume that information for each session is isolated from all other sessions.  A quick example of this is a chat room.  If two users try to join a room with a given "topic" and they become load balanced to separate nodes in the cluster they will essentially be in different chat rooms as things stand now.

             

            The "One Degree of Separation" paradigm (http://www.jboss.org/errai/ErraiBus) is the limitation here.  I can certainly understand why it is important but I would propose that it be extended such that traversing a cluster on the server side is included in that one degree.

             

            Note that the Atmosphere project (https://atmosphere.dev.java.net/) has attempted to tackle this issue as well.  In fact, publishing messages to "other" sources (JMS, EJB, Cluster) seems to have been one of their reasons for doing this http://weblogs.java.net/blog/jfarcand/archive/2009/07/cluster_cluster.html.  I have not had a chance to try their method but they claim to support both Shoal (Glassfish) and JGroups clustering.  I would guess that this means they came up with an abstraction for clustering their nodes and are now able to add providers.  That sounds reasonable and would allow Errai to be extended.

            • 3. Re: Errai in a cluster
              cbrock

              I do plan on making this a priority in the coming releases.  

               

              It's worth mentioning that one community user has been successful clustering ErraiBus using Terracotta.  But obviously it would be nice for there to be some flexibility here, and the design of the bus architecture practically begs for this kind of functionality in any case.

              • 4. Re: Errai in a cluster
                akarl

                I'd like to also note that I have this accomplished using the JBoss.org 5.1 embedded JBossCache as my communications mechanism.  I can provide the code if someone else would like to use it but essentially doing this comes down to 3 things and there is really very little code related to Errai once you figure out the two hook points into the Errai server bus.

                1 - Extend the servlet implementation.  In my case I want to use the JBoss Comet servlet so I extended it in order to inject my own ServerMessageBus implementation.  Note that this could easily be configurable in the future.

                2 - Extend the ServerMessageBusImpl.  The main thing here is to be able to attach listeners to your message bus implementation.  To do this I find my communication mechanism during the ServerMessageBusImpl constructor (I just used a singleton for now) and pass a reference to the bus.

                3 - Write a communication mechanism.  I used JBossCache but you can use whatever you want.  You need to use the "addGlobalListener" method to listen for all messages in order to pass them to other nodes.  You also need to be able to listen messages from other nodes and then use the "sendGlobal" method to pass messages to your local bus.

                 

                A few things to note.  You probably want to disregard messages from the "ServerBus" subject.  The heartbeats and subscription messages are not needed on the other nodes since you are adding a listener for all messages.  I chose to create a special MessagePart called "ShareMessage" so that I can determine whether to send a given message to other nodes.  This may not be ideal but it works to limit the number of inter-node communications.  Without that mechanism there are some fun scenarios where messages get fired multiple times on other nodes.

                • 5. Re: Errai in a cluster
                  guesswt

                  Is it possible for you to share the code for Clustering . I am trying hard to make Errai work in Cluster . Not able to get it working . I use Jboss 5.1 as my app server

                  • 6. Re: Errai in a cluster
                    akarl

                    Sorry, I no longer work on that product and do not have access to the source code.  The description I have above is accurate from what I remember.  The solution is in a production environment and works quite well using JGroups via JBossCache's API. 

                    • 7. Re: Errai in a cluster
                      heiko.braun

                      Guess What wrote:

                       

                      I am trying hard to make Errai work in Cluster .

                      What are you trying to achieve?

                      Do you need to replicate state in a cluster? What state?

                      • 8. Re: Errai in a cluster
                        guesswt

                        These are the problems which I am running into .

                         

                        1. If any one of my cluster nodes go down , My errai session also expires .

                        2. I still not sure if I can send message to a different node in a cluster .

                        • 9. Re: Errai in a cluster
                          guesswt

                          Any One found a Solution ?

                          • 10. Re: Errai in a cluster
                            heiko.braun

                            If the node the GWT app was loaded from goes down, the first question is:

                             

                            - how does this relate to SOP (same origin policy)?

                             

                            SOP basically means an Ajax client can only connect to the host it was loaded from.

                             

                            The second question would then be:

                             

                            - what HTTP load balancing strategy is in place?

                             

                            Round robin basically sends the next Errai request to an active node.

                             

                            A new node needs to establish a new session, which means client bus re-connect and re-initializes.

                            It's basically like hitting the refresh button. From that point on you should be working of another node like before.

                            If there is any server side state that you rely on, make sure it's replicated and serves failover. But that basically nothing errai needs

                            to deal with.

                             

                            The bottom line is:

                            You cannot simply "send a message to a different node in a cluster". There is a HTTP connection involved that

                            needs to needs to failover. This directly impacts the same origin policy (most likely). Any server side side state should not be subject

                            to be managed by Errai.

                             

                            Hope that makes sense.

                            • 11. Re: Errai in a cluster
                              akarl

                              In my case, I was using HTTP Load balancing via Mod_Cluster with sticky sessions in place.  Mod_cluster, and for that matter most load balancer based solutions prevent any problems due to SOP because you always use the same URL.  The load balancer decides which back-end machine will process the requests.  Sticky sessions also gets around the problem of round robin requests from a single user to some extent however it actually did not help me at all in my scenario.  I actually had two different machines that needed to share Errai sessions and they were not aware of each other's HTTP sessions.  My solution that I described above was to create an Errai bus listener on each node in the cluster.  Whenever Node X would receive a message it would then fire that exact same message to the other nodes in the cluster using JGroups/JBossCache as the transport mechanism.  There were a few types of message I had to be careful of but I was able to get this working.  It was very fast even across nodes and was very reliable as well.

                              • 12. Re: Errai in a cluster
                                heiko.braun

                                Thanks Adam. That's exactly the level of detail I am asking "Guess What" for, to be able to reply to his questions.

                                 

                                So Adam has:

                                 

                                 

                                C1 --         N5
                                        LB -- 
                                C2 --         N6
                                

                                 

                                With sticky sessions keeping C1 on N5 and C2 on N6.

                                • 13. Re: Errai in a cluster
                                  guesswt

                                  Thanks Adam, Heiko . I am also  having almost the same set up .. Let me see if i can set up a Errai Bus Listener on the nodes .

                                  I am not familiar with Jgroups . I will  google around .

                                  • 14. Re: Errai in a cluster
                                    heiko.braun

                                    If I understand both of you correctly you are trying to broadcast messages created on a single node, to any GWT client on any other node, right?

                                    (Think of a chat example maybe)

                                     

                                    Before you reach for something as low level as jgroups (which works, but isn't advised) I would more likely look at the JMS integration and delegate the responsibility to distribute messages in a cluster to the JMS server implementation.

                                     

                                    Errai-JMS allows you to directly listen to JMS topics from GWT clients and send messages to them. But you might as well just use JMS from your service implementation  to broadcast messages within a cluster. IMO that more simple to maintain and portable as well.

                                    1 2 Previous Next