9 Replies Latest reply on Oct 5, 2011 5:31 AM by jordi_alvarez

    persisted stateful sessions and knowledge base changes

    jordi_alvarez

      Hello, I am working for an organization for which we have adopted jBPM as the BPM solution. We are making use also of Drools

      Expert, which results in a nice combination.

       

      We have stateful sessions persisted in a database in order to record the state of long-time running processes (which are the goal of our

      BPM).

       

      In order to make changes to the knowledge-base associated to the stateful sessions that we have, we always redeploy the whole system (ear application + rules + process definitions). This is where we have the problem. Depending on the changes performed to the database, the system is not able to reload the persisted sessions (the change could be for example a change in the condition of a rule). This results in problems and stack traces similar to the following one:

       

      [...]

      Caused by: java.lang.RuntimeException: Unable to load session snapshot

      at org.drools.persistence.session.JPASessionMarshallingHelper.loadSnapshot(JPASessionMarshallingHelper.java:96)

      at org.drools.persistence.session.SingleSessionCommandService.initKsession(SingleSessionCommandService.java:196)

      at org.drools.persistence.session.SingleSessionCommandService.<init>(SingleSessionCommandService.java:159)

      ... 95 more

      Caused by: java.lang.ClassCastException: org.drools.reteoo.ObjectTypeNode

      at org.drools.marshalling.impl.InputMarshaller.readRightTuple(InputMarshaller.java:401)

      at org.drools.marshalling.impl.InputMarshaller.readRightTuples(InputMarshaller.java:391)

      at org.drools.marshalling.impl.InputMarshaller.readFactHandles(InputMarshaller.java:326)

      at org.drools.marshalling.impl.InputMarshaller.readSession(InputMarshaller.java:205)

      at org.drools.marshalling.impl.DefaultMarshaller.unmarshall(DefaultMarshaller.java:91)

      at org.drools.persistence.session.JPASessionMarshallingHelper.loadSnapshot(JPASessionMarshallingHelper.java:91)

      ... 97 more

       

      According to some posts (which mainly relate to drools expert), if we were using the knowledge agent we could update the already loaded sessions, which would allow to migrate in-memory sessions from the old database to the new database:

       

      http://drools.46999.n3.nabble.com/Dynamic-updates-of-stateful-sessions-td2834623.html

      http://stackoverflow.com/questions/4368148/is-it-possible-to-add-a-new-rule-to-a-running-drools-session

       

      Nevertheless, we absolutely need a way to migrate persisted sessions from the old database to the new one. Even in the situation in which we would allow hot modification of the knowledge base, we cannot have all the stateful sessions loaded in memory when this would happen.

      Is there any facility provided by Drools Expert/Flow that allows to perform this operations easily? Is there a way/configuration to

      load/save sessions without the need to store/retrieve the rete-tree (and instead recomputing it), which seems to be the problem for these

      situations.

       

      We have already posted this information into drools mailing list; but nobody has answered yet. Any help is appreciated!

       

      Many thanks in advance,

      Jordi Alvarez

       

        • 1. Re: persisted stateful sessions and knowledge base changes
          swiderski.maciej

          Have you tried Knowledge Agent? It could monitor resources and recreate the knowledge base when change is discovered. Maybe that is something that can resolve the issues you're facing.

           

           

          Jordi Alvarez wrote:

          Nevertheless, we absolutely need a way to migrate persisted sessions from the old database to the new one. Even in the situation in which we would allow hot modification of the knowledge base, we cannot have all the stateful sessions loaded in memory when this would happen.

          What you mean by old and new data base?

           

          Cheers

          • 2. Re: persisted stateful sessions and knowledge base changes
            jordi_alvarez

            Hi Maciej, we have been doing some tests with KnowledgeAgent.

             

            We managed to migrate sessions that are in memory when knowledge base changes.

             

            The problem we are facing is that we are not able to have all sessions in memory when knowledge base modifications happen. This is not possible since we are working with a lot of sessions.

             

            So, we need some kind of additional processing to that performed by KnowledgeAgent in order to update those sessions that are persisted (but not loaded in memory).

             

            We hoped that drools expert provided some facility regarding this point, but I guess the last sentence from the following paragraph extracted from drools expert documentation (still valid for recently released 5.3CR1 version) is stating that that kind of facility will come in future versions:

             

            "The Knowlege Agent provides automatic loading, caching and re-loading of resources and is configured from a
            properties files. The Knowledge Agent can update or rebuild this Knowlege Base as the resources it uses
            are changed. The strategy for this is determined by the configuration given to the factory, but it is
            typically pull-based using regular polling. We hope to add push-based updates and rebuilds in
            future versions."

             

            We are in fact now working in our own custom push solution in which we are loading and persisting sessions, and using the KnowledgeAgent in order to perform the knowledge base adaptations and making the sessions aware to those changes.

             

            With respect to old and new knowledge base, I was refering to (a) the knowledge base before the changes were applied, and (b) the knowledge base after applying the changes.

             

            thanks for your answer!

            Jordi Alvarez

            • 3. Re: persisted stateful sessions and knowledge base changes
              calca

              Jordi,

               

              How did you migrate it? Did you solve the exception? As I am having the same problem esporadically.

               

              Thanks!

               

              Demian

              • 4. Re: persisted stateful sessions and knowledge base changes
                jordi_alvarez

                Hi Demian,

                 

                we are still working on the solution. In our situation, and at this moment, we are deploying a new knowledge base every time we perform changes to it. This means that in an initial time we have KB0, for which some sessions were created; and later, we deploy a knowledge base KB1 (which is, in fact, an evolution of KB0).

                 

                The KnowledgeAgent allowed us to migrate sessions online. That is, loading KB0 sessions with KB0, and then applying a delta that evolves KB0 into KB1. The sessions associated to KB0 exhibit then the behaviour corresponding to KB1. We are also able to persist those sessions.

                 

                But we have not succeeded in reloading them after the migration has been performed when KB1 is loaded from scratch.

                 

                After a while investigating in that direction, we have abandoned it, and started working in other directions (which includes looking into the concrete way drools sessions are persisted).

                 

                So, any useful information regarding migration of persisted sessions will be highly appreciated.

                 

                regards,

                Jordi Alvarez

                • 5. Re: persisted stateful sessions and knowledge base changes
                  swiderski.maciej

                  Jordi, Damian, finally I have found some time to test it, I did not want to put this note without confirming it.

                   

                  I have pretty same issue what you describe. I built a JBoss AS service (sar) that encapsulates jBPM5 as process engine which is bound to JNDI so application can simple look it up or use dependency injection.

                  So process engine is configured to use Guvnor as knowledge base source and uses knowledge agent to load it and maintain its changes.

                  Next there is simple web application with servlet that looks up the process engine from JNDI and gets a stateful session from it. Process engine is designed to maintain the sessions in regards to internal ids and application is using its own names for the sessions, for instance it could be user id that is logged on.

                  When ever operation is invoked on the session, like start process session is disposed, meaning persisted to db. Next call will get the session from db and invoke other operations. What I tested is that I modify processes in Guvnor, next build package and snapshot and perform the operation from the web application. So the session is loaded again and the changes are automatically applied to it so it operates on the latest version of knowledge base.

                   

                  Let me know if that is what you're trying to achieve?

                   

                  Cheers

                  Maciej

                  • 6. Re: persisted stateful sessions and knowledge base changes
                    jordi_alvarez

                    Hi Maciek, in my case yes, this is what we need.

                     

                    We are now deploying without the help of guvnor, which was going to be introduced in a near future. I guess the whole behaviour can also be reproduced without guvnor. After reading your answer, it is worth advancing in this direction, and see whether the use of guvnor with packages and snapshots is the direct solution for this problem.

                     

                    Thanks very much, we will report here the success of our work.

                    Jordi

                    • 7. Re: persisted stateful sessions and knowledge base changes
                      calca

                      Hi Maciej. Yes, it is a similar case than mine too. I have (for now, changing it) one session which I reload everytime I start the app, and I have an agent to rebuild my knowledge base. From time to time, I am facing a similar exception when loading the session, which I think it is caused to some changes in kbase:

                      Caused by: java.lang.reflect.InvocationTargetException

                              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

                              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

                               at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

                              at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

                              at org.drools.persistence.jpa.KnowledgeStoreServiceImpl.buildCommanService(KnowledgeStoreServiceImpl.java:88)

                               ... 35 more

                      Caused by: java.lang.RuntimeException: Unable to load session snapshot

                              at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:96)

                              at org.drools.persistence.SingleSessionCommandService.initKsession(SingleSessionCommandService.java:193)

                               at org.drools.persistence.SingleSessionCommandService.<init>(SingleSessionCommandService.java:157)

                              ... 40 more

                      Caused by: java.lang.NullPointerException

                              at org.drools.common.ConcurrentNodeMemories.getNodeMemory(ConcurrentNodeMemories.java:65)

                               at org.drools.common.AbstractWorkingMemory.getNodeMemory(AbstractWorkingMemory.java:1035)

                              at org.drools.marshalling.impl.InputMarshaller.readFactHandles(InputMarshaller.java:327)

                              at org.drools.marshalling.impl.InputMarshaller.readSession(InputMarshaller.java:221)

                               at org.drools.marshalling.impl.DefaultMarshaller.unmarshall(DefaultMarshaller.java:92)

                              at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:91)

                              ... 42 more

                       

                      Thanks,

                       

                      Demian

                      • 8. Re: persisted stateful sessions and knowledge base changes
                        swiderski.maciej

                        Guys, one question when you construct Knowledge Agent do you set newInstance property to true or false?

                        • 9. Re: persisted stateful sessions and knowledge base changes
                          jordi_alvarez

                          Hi Maciej, I have tried it both ways, resulting in error both.

                           

                          At the present moment we have been working in a workaround in which we have patched the drools code that loads the session in order to perform a migration from one database to another. And our first experiments tell us that this is working.

                           

                          We have separated the load process in two parts: a first in which the first part of the session is read using the "old" knowledge base. This allows to build the structure associated to the rete tree correctly in the old knowledge base.

                           

                          Then, it comes a second part in which a second session is built on the "new" knowledge base, and the facts of the first session are added to it (we added an activation filter in order to avoid activations derived from this initial load to be executed). The the load process continues with the rest of the session information but over this second session (timers and processes mainly).

                           

                          This is in fact too tricky, but it has shown to work till the moment, while we explore further a definitive solution in the line you commented (guvnor, compiled packages, snaptshots, in combination with the knowledge agent), if possible without performing any drools patch.

                           

                          regards,

                          Jordi