9 Replies Latest reply on Jan 29, 2013 2:53 PM by tmag

    Multiple Sessions - Timer Tasks - some triggered twice and some missed

    tmag

      Our bpmn process has a set of timer tasks.

      • When we use a single session to create all processes & reload that single session - all timer tasks resuming for different processes without any issue.
      • When we use multiple sessions (we tried with 2 sessions)  to create processes & reload all those sessions to restore timers - some timer tasks triggering twice & some not triggered at all.

      Anyone faced this issue with timer tasks on multiple sessions?

       

      We are using 5.4.Final.

        • 1. Re: Multiple Sessions - Timer Tasks - some triggered twice and some missed
          doboss

          Shot in the dark, but are you disposing of your sessions after you use them?

          • 2. Re: Multiple Sessions - Timer Tasks - some triggered twice and some missed
            tmag

            We are not disposing the sessions - as we need them to restore timer tasks on server reboot.

            • 3. Re: Multiple Sessions - Timer Tasks - some triggered twice and some missed
              tmag

              Debugging the code - it seems SchedulerThreadPool in JDKTimerService is getting instantiated multiple times for some sessions and jobs getting scheduled twice.

              As a work around we registered our own TimerService and made this SchedulerThreadPool static to get instanatiated only once (one SchedulerThreadPool per JVM) & made the pool size configurable.

               

              Refer to https://community.jboss.org/message/741461 to register your own timer service.

               

               

              private static ScheduledThreadPoolExecutor scheduler;

               

              ..........................................

               

              public MyTimerService(int size) {

                      getSchedularInstance(size);

                  }

               

              private ScheduledThreadPoolExecutor getSchedularInstance(int size) {

                      if(scheduler == null) {

                          this.scheduler = new ScheduledThreadPoolExecutor( size );

                     } else {

                          System.out.println(" returning the old scheduler ...");

                      }

                     

                      return this.scheduler;

                  }

              • 4. Re: Multiple Sessions - Timer Tasks - some triggered twice and some missed
                tmag

                We are seeing this issue when we are testing the process restoration - bouncing & restoring the session with timer tasks multiple times (about 7 to 8  times) .

                 

                Seems there are multiple issues here - for repeating timer tasks in regular intervals.

                 

                1. Sometimes multiple instances of ScheduledThreadPool objects getting created on single session - which we tried to avoid by making scheduler static as above.

                2. ProcessContext data getting corrupted with duplicate process instance info.

                When we print out the session & process info from context  in our registered custom timer service  - its showing duplicate process instance ids as shown below.

                 

                ProcessJobContext processJobContext = (ProcessJobContext) ctx;

                StatefulKnowledgeSessionImpl ksessionImpl = (StatefulKnowledgeSessionImpl) processJobContext.getKnowledgeRuntime();

                System.out.println("**********>session id : "+ ksessionImpl.getId() + "; process id : "+ processJobContext.getProcessInstanceId());

                 

                 

                **********> session id : 0; process id : 6

                **********> session id : 0; process id : 7

                **********> session id : 0; process id : 8

                **********> session id : 0; process id : 9

                **********> session id : 0; process id : 10

                ......................

                 

                **********> session id : 0; process id : 6

                **********> session id : 0; process id : 7

                **********> session id : 0; process id : 8

                **********> session id : 0; process id : 9

                **********> session id : 0; process id : 10

                 

                These processes 6 to 10 are getting scheduled twice & executing twice.

                • 5. Re: Multiple Sessions - Timer Tasks - some triggered twice and some missed
                  swiderski.maciej

                  one thing that can lead into such situation is when you do not dispose session and then load it once again. That will mean you have two sessions (of the same id) loaded twice and thus timers will be fired twice.

                   

                  HTH

                  • 6. Re: Multiple Sessions - Timer Tasks - some triggered twice and some missed
                    tmag

                    Does that mean - we can dispose a session and reload it again to restore the timers? Our understanding is that - once we dispose a session, it cannot be reused.

                     

                    1. We have multiple sessions with timer tasks. On server crash & reboot - we are reloading all sessions to restore timers. For initial 6 to 7 reboots everything seems fine. This issue happening after that.

                    2. Even when we use a single session, in our active-active cluster environment - when server1 goes down - we are reloading that session on server2 - which is making two jbpm sessions on server2. Again leading to multiple sessions probelm as above.

                     

                    How can we overcome this? Please help.

                    • 7. Re: Multiple Sessions - Timer Tasks - some triggered twice and some missed
                      swiderski.maciej

                      Correct, you can dispose and the reload the session at any time. Once session is reloaded all timers are reinitialized and in case of some missed fires such timers will be executed almost directly after session is reloaded.

                       

                      Most important is to have only single session that runs given timers so in clustered environment that is bit tricky as you need to keep track of what timers are active on what session. There are efforts to provide more cluster friendly timer support and hopefully it will be soon available.

                       

                      HTH

                      • 8. Re: Multiple Sessions - Timer Tasks - some triggered twice and some missed
                        bpm77

                        according to your suggestion, some missed fires will be executed after session is loaded.but in my case it's never be called.

                        I have three sessions 1,2,3 and created instances p1,p2,p3 on session 1,  Once session were active all threeprocess  timertasks were fired and showing session id 1 and three process context with process id p1,p2,p3.

                        Now restarted server, in JDKTimerService schedulejob , it is always showing sessioid 0 with correct process instance id p1,p2,p3 and timertasks running fine.

                        then I have stopped java application and created new session id 2 and created process instances p4,p5,p6 on session2.

                        similiarly stopped application and  created p7,p8,p9 on sessiod id 3.

                        Now I have loaded all three sessions 1,2,3 timertask with delay of 1 minute.

                        I can see instances are running multilple time , it seems like each session is running for  p1 to p9  but behavior is not consistent.

                        sometime missing fire never runs ,sometime p1 is running for 4 times sometime p5 for 3 times like that...but in schedulejob of JDKTimerService session id is always showing 0(after startup) though session ids are 1,2,3.

                         

                        I have a table where storing sessions vs processinstanceid mapping  but dont know how to keep track of active timers.Is active timers are active processinstace?

                        Please help on this , stuck on this for days now.

                        Your suggestion is highly appreciated..

                        • 9. Re: Multiple Sessions - Timer Tasks - some triggered twice and some missed
                          tmag

                          jBPM 5.4 seems to have some issues with

                          drools-compiler-5.5.0.Final.jar

                          drools-core-5.5.0.Final.jar

                          drools-persistence-jpa-5.5.0.Final.jar

                           

                          When you have multiple sessions - its triggering timer tasks repeatedly & some timer tasks even miss.

                          If you replace these three jars with jBPM 5.3 ones - then it looks ok.