13 Replies Latest reply on Sep 11, 2012 4:30 AM by massimogentilini

    Clustering strategy for jBPM5

    garytse

      Hi,

       

      I'm deciding the clustering/ fail-over strategies for a project on jBPM5.  We have a 2 node application server cluster.  I've set up persistence on the StatefulKnowledgeSession so the states are persisted to DB.  So far so good - whatever process in node A can be loaded in node B, vice-versa.

       

      Q1: I plan to have a singleton ksession, one for each cluster-node A & B.  Can I share the same StatefulKnowledgeSession across both nodes A & B ?   I'm using this code below to load the StatefulKnowledgeSession (hard code session ID 1 for both nodes). 

       

      StatefulKnowledgeSession ksession = JPAKnowledgeService.loadStatefulKnowledgeSession(1, kbase, config, env);

       

      Q2: Say, for simplicity I have this process :

       

        [ start ] --> [ intermediateCatchEvent (its really a webservice) ] -->  [ print some message task ] --> [ end ]

       

      First, I register a WorkItemHandler for the intermediateCatchEvent on both nodes into the knowledge base.  Then, I started the process in node A (process instance ID 99).  Then, without any session re-loading, webservice is invoked in node B.

       

      Q3: Will the process instance 99 be found in node B by using "ksession.getProcessInstances()" ?

       

      Q4: If the process is found in node B, what happens when I call the

      processInstance99.signalEvent("webservice", message);

       

      Will both node A and B print some message?  or only node B ? or randomly A "xor" B may do the "print some message task" ?

       

      Q5: If the sharing of a single StatefulKnowledgeSession acorss 2 cluster nodes does not work, what other options do I have?  Allocate certain process to specific node ? or say, write into DB somewhere about which process instance ID is running on which node?

       

      I know there's a lot of questions there; thanks in advance for your time.

       

      Gary

        • 1. Re: Clustering strategy for jBPM5
          garytse

          Any one please help?  Has anyone able to share a single stateful knowledge session across 2 nodes with success or failures? 

          • 2. Re: Clustering strategy for jBPM5
            arkper

            No. I have tried to do so, but it appears to be an issue when Hibernate tries to update SessionInfo shared by more than one node. It fails to commit the transaction if the entity has been updated by another node. The exception is as follows:

             

             

             

             

            Hibernate: update SessionInfo set lastModificationDate=?, rulesByteArray=?, startDate=?, OPTLOCK=? where id=? and OPTLOCK=?

            Sep 7, 2011 10:39:34 PM org.drools.persistence.jta.JtaTransactionManager commit

            WARNING: Unable to commit transaction

            javax.persistence.OptimisticLockException

             

             

            : org.hibernate.StaleObjectStateException

            : Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [org.drools.persistence.info.SessionInfo#1

             

             

            It should be noted that it's just a warning, but I'm not sure if the necessary updates are re-attempted in order to persist the state accuratly.

             

            Something like Terracotta should help to cluster Knowledge Session as it has a potential to instrument the byte code and introduce the necessary locking mechanism to make the shared objects "thread"-safe even for threads running in different JVMs.

             

            Has anybody tried and succeeded integrating jBPM5 with Terracotta?

             

             

             

            • 3. Re: Clustering strategy for jBPM5
              salaboy21

              That's right and it's correct.

              If two nodes wants to do operations on the same sessionInfo one of the two will fail and the other will success. that's the way how locking works.

              Your application should add the logic to retry the operation if it fails and you don't want to miss the second modification (the one that fails).

               

              Best Regards

              • 4. Re: Clustering strategy for jBPM5
                arkper

                Are you suggesting that this try-if-fail-retry approach should be used as a way to cluster a jBPM5 application?

                • 5. Re: Clustering strategy for jBPM5
                  salaboy21

                  I'm not suggesting anything, but that's how transactional databases work. If you get two connections from different places that modifies the same row in the database one will success and the other will fail. The expcetion is clear about that:

                  javax.persistence.OptimisticLockException

                   

                  It's not a jbpm5 problem, its how databases work. If you not ensure that two thread can keep everything consistent one needs to fail.

                   

                  In which scenario are you getting that exception? what are you trying to achieve?

                   

                  Best regards

                  • 6. Re: Clustering strategy for jBPM5
                    arkper

                    I'm trying to achieve exactly what this thread is all about - have a cluster of several jBPM5 applications. They all areconnected to an enterprise service bus that can easily round-robin requests coming from the clustered web application. The exact use case is as follows:

                     

                    1. jBPM5 Node 1 gets a request to create a process instance.There are several human tasks that need to be created - so Node 1 communicates to a cluster of Task Servers to have that done.

                    2. A user completes a human task which results in a Task server Node X communicating its completion to jBPM5 Node 2.

                     

                    The point that I'm trying to make is that unless the jBPM5 nodes share the same knowledge session, Node 2 will not be able to pick up the task completion event and move the process instance alone because the process instance would only be available to the node that had created it. So, as Gary Tse had suggested in the opening message, I tried session sharing by executing the following on each node:

                     

                    StatefulKnowledgeSession ksession = JPAKnowledgeService.loadStatefulKnowledgeSession(1, kbase, config, env);

                     

                    I reported the predictable failure due to optimistic locking not being the right strategy to handle concurrent updates from multiple nodes. So this isn't an option to build a clustered jBPM5 application. So I'm going to ask you again (I've asked you the same question in a different thread dedictaed to clustering) - what is your approach to building a clustered jBPM5 application for scalability and high availability? Every high volume mission critical enterprise application requires this as a matter of very high priority.

                    • 7. Re: Clustering strategy for jBPM5
                      salaboy21

                      I will say something similar to my previous answer, probably I can help you with your problem:

                      You can create a simple mechanism that do the load for you depending on the processes that you know that are already pending for execution.

                      As you mention you need to use: StatefulKnowledgeSession ksession = JPAKnowledgeService.loadStatefulKnowledgeSession(1, kbase, config, env);

                      I'm not sure why you hard code the 1 for the session ID, but if you have multiple nodes doing that in a clustered environment one of the nodes will sucess on the process continuation. The other nodes will fail and you can probably discard that expcetions because you know that at least one node continue the process. Can you please name state the problems that you find in this approach? probably I'm missing something.. I'm just trying to help.

                       

                       

                      Best Regards

                      • 8. Re: Clustering strategy for jBPM5
                        arkper

                        Mauricio:

                         

                        First of all, thank you very much for your time and desire to help. That's truly appreciated.

                         

                        What you seem to be missing though is that in order for a cluster totry and solve the scalability problem, only ONE, but ANY one, node should be able to process ANY given valid request. In your approach, ALL nodes would try and only one would succeed to process a request. Clearly, this wouldn't address the scalability concern since ALL nodes would be trying to do the job with only one succeeding and all the other ones wasting their resources while they could have been working on other concurrent requests which would have increased the overall cluster capacity. Your approach would probably address the fail-over concern somewhat, but not at all the scalability one...

                         

                        As a community, we should push the jBPM5 team very hard to introduce the clustering support into the product. They seem to take it for granted, but in my humble opinion, it's a huge void as it makes the product completely unfit for the enterprise-grade applications.

                        • 9. Re: Clustering strategy for jBPM5
                          salaboy21

                          If I don't understand you wrong, if you change the strategy to pesimistic locking only one will start and succeed right? that can be easily done with hibernate overridings, Am I wrong?

                          Cheers

                          • 10. Re: Clustering strategy for jBPM5
                            salaboy21

                            My opinion about the community is that we are the community and we don't need to push anyone we can introduce new mechanisms if we agree on the implementation and we make those improvements as community. I'm a community member and I tried to spend time fixing bugs and adding features, but sometimes is not enough Any help is appreciated and more than welcome.

                             

                            Best regards

                            • 11. Re: Clustering strategy for jBPM5
                              newbird

                              Hi,

                              Did you get any work around or solution for this?

                              Please let me know, i am also having the same problem....

                              • 12. Re: Clustering strategy for jBPM5
                                arkper

                                We ended up resorting to process partitioning with sticky processes. All requests are round-robined across N jBPM5 nodes. Once a process instance is created on partition X it is tagged with that partition and it'll always be served by partition X. Each partition has an active and stand-by nodes for fail-over.  It's not real clustering, but it does help to scale up... Evidently, jBPM 5.1 is not designed for clustering.

                                • 13. Re: Clustering strategy for jBPM5
                                  massimogentilini

                                  My two cents:

                                   

                                  1) The availability of a simple clustering strategy for upcoming version of JBPM is a mandatory asset to gain acceptance of JBPM in the enterprise world. Is not possible to have to resort to hack or something when in the JBoss products lineup there are at least a couple of distributed cache solution that can be adopted to allow clustering

                                   

                                  2) The name itself "loadStatefulKnowledgeSession" has the implicit meaning that the session cannot be easily shared, if the session is "stateful" then it goes against the "stateless" requirement that are behind a "pure" cluster solution. There should be a way to have a different provider or to configure the JPAKnowledgeService in a way that enables an easy clustering solution and manage concurrency without resorting to database locking and retries, like having a shared cache mechanism to share the knowledge session across nodes in a way that is fault tolerant, highly available and has good performance

                                   

                                  I do not know it there is an easy way to push for a solution or to create a Jira request so that it can be voted but, from my point of view, this other enterprisey requirements are the ones that need to be pushed to make JBPM suitable for a larger adoption.

                                   

                                  Regards

                                  Massimo