11 Replies Latest reply on Sep 15, 2011 11:02 AM by dionisior

    EJB3 Session Bean Pool

    robert.geisler

      hello...

       

      during the last few weeks we experienced some problems with session bean instantiation.
      in the past we used default session bean pool, ThreadlocalPool. this pool implementation creates instances for every newly created thread, but these instances seem to never be destroyed, not even if the threads, they were created for, die.
      so we tried StrictMaxPool pool implementation and set a maximum of 100.000. with this pool we see much less bean instances, because the pool creates them only if there are no free instances left. existing beans (released earlier) get reused later, allthough they are not destroyed until container/ server shuts down.

       

      a simple scenario: in the morning 200 users log in and read their private messages. they all start working at 8 a.m. and they are using mostly the same services. every single user needs one bean instance for one request and because all users request concurrently, StrictMaxPool creates 200 instances. later the users have lunch, so no one fires request to the server, all beans (200!) are released and waiting in the pool. but because StrictMaxPool does not destroy beans, the 200 instances are kept in server vm/ heap all the day (until shutdown!).

       

      we think this behaviour is a waste of resources! a pool should destroy instances if there a more instances in the pool than we want to reserve for further usage. because later the day our 200 users will almost never produce 200 concurrent requests again, we would like to configure the pool to just keep a maximum of 50 beans.

       

      so my questions are:

      1) why are instances not destroyed by StrictMaxPool? there is an idea of destroying instances in StrictMaxPool.release, but because pool.size will never be more than maxSize (an attemp to get one more bean always fails!), remove is never called!?

      2) is there a pool implementation that destroys instances and just holds a configurable amount of beans? or do we have to implement it on our own (TolerantMinPool ; ))?

      3) ?some more suggestions?

       

      thanks in advance

      robert

        • 1. EJB3 Session Bean Pool
          wolfc

          For the benefit of other people new to this thread, you can only use ThreadlocalPool if the threads are backed by a pool. Spawning new / infinite threads will lead to new / infinite instances. (http://community.jboss.org/message/589434#589434)

           

          The StrictMaxPool is designed to hold a maximum amount of instances. It does not care about concurrency or retention.

          I think it should be possible to extend StrictMaxPool and set the strictMaxSize semaphore higher than the maxSize. e.g. strictMaxSize = 200, maxSize = 50. That should allow for 200 concurrent requests, while the pool will size down to 50 during low load.

          • 2. EJB3 Session Bean Pool
            robert.geisler

            thank you, carlo for response.

             

            of course you are right, one could extend StrictMaxPool and enlarge strictMaxSize semaphore. but what is the benefit of using the semaphore at all? isnt it an unnecessary overhead? we believe we do not need a maximum to limit creation of instances, but we need a maximum of pooled instances!

             

            so i thought about a pool implementation backed by a simple array with length maxSize:

            1. get() looks into the array and returns and removes the last element -if available;
            2. otherwise create a new one, no maximum for creating instances (aka tolerant)!
            3. instances that are released go back into the array -if they fit into it (maxSize = array length);
            4. otherwise they are removed immediately.

             

            this seems to be quite simple to implement...?

            would you expect any issues with such an implementation?

            • 3. EJB3 Session Bean Pool
              wolfc

              Hi Robert,

               

              The semaphore is in place to prevent over-use, without it you can get too many concurrent requests which would stall the entire system. Should that condition be unlikely, but you still want to guard against it with less overhead, you should contemplate using a non-fair semaphore instead.

               

              Other than that I don't see any issues.

               

              Hmm, weird that the JDK does not have a fix-capacity queue implementation, that would fit the bill.

               

              If you want, please share the code on github. I would very much like to see it.

              • 4. EJB3 Session Bean Pool
                robert.geisler

                hello carlo,

                 

                i finally implemented a custom org.jboss.ejb3.Pool and i am going to test it during the next days.

                when all the bugs are gone, i'll post the code in JBoss forum; or on github as suggested.

                 

                but today i have a few more questions:

                 

                I) remove instances that are in use??

                i had a look at the code of org.jboss.ejb3.InfinitePool (written by you : )) and i noticed that InfinitePool holds a List<BeanContext> of >>active<< contexts. i guess, active means that these contexts are currently used (by a single thread)? whenever a context is created, the new instance is added to actives and then it stays there until it get released/ removed.

                when the pool is destroyed (e.g. the bean gets undeployed?), InfinitePool iterates through its actives and calls remove() for each of these instances/ contexts created before. but StrictMaxPool (written by Kabir Khan) for example does not maintain such a list and thus does not call remove for active instances. so instances that are in use (/ are not located in the pool) while the pool is destroyed will never be removed, right?

                is it critical? in which cases is Pool.destroy() called and is the pool required to remove such active instances?? what happens to these instances after the pool is destroyed?

                 

                II) how to synchronize pool?

                a) InfinitePool and StrictMaxPool do not use synchronized methods, but synchronized statements for changes in the pool structure. i tried to adopt this for my pool. but i wonder if synchronized methods would also do the trick? or could it cause a bottleneck??

                b) i checked sources of pool implementations in JBoss6. i noticed that org.jboss.ejb3.AbstractPool uses AtomicInteger to count create() and remove() while JBoss4 used a primitive instead. should a adopt this as well??

                 

                 

                thanks in advance.

                robert

                • 5. Re: EJB3 Session Bean Pool
                  wolfc

                  Hi Robert,

                   

                  Glad to see you're making good progress.

                  I) remove instances that are in use??

                  There is an unwritten rule that the Pool should remove its pooled instances whenever it is destroyed. StrictMaxPool doesn't follow this yet.

                   

                  The Pool itself gets destroyed on undeploy or shutdown. In essence it means that the application will be tidied up a bit cleaner.

                  II) how to synchronize pool?

                  a) InfinitePool and StrictMaxPool do not use synchronized methods, but synchronized statements for changes in the pool structure. i tried to adopt this for my pool. but i wonder if synchronized methods would also do the trick? or could it cause a bottleneck??

                  b) i checked sources of pool implementations in JBoss6. i noticed that org.jboss.ejb3.AbstractPool uses AtomicInteger to count create() and remove() while JBoss4 used a primitive instead. should a adopt this as well??

                  A Pool is called concurrently and thus thread safety must be implemented manually. Each synchronized block / method is a bottleneck for all incoming calls and should be kept to a minimum. Also because synchronized is an unfair lock on the pool, it leads to a high deviation in response times. ThreadlocalPool for example has the objective to keep thread locking to a minumum, thus giving the best throughput. Based on a pools requirements a choice must be made for modifying shared data: atomic, unfair lock or fair lock.

                  The counts in JBoss AS 4 are wrongly implemented and this was fixed. (See https://issues.jboss.org/browse/EJBTHREE-2155.)

                  • 6. EJB3 Session Bean Pool
                    robert.geisler

                    hello carlo,

                     

                    i finished implementing my "LooseMinPool" : )

                    you may find the source on sourcepod: http://www.sourcepod.com/rptzcu29-4656 .

                     

                    core pool functionality:

                    • uses an array to store released instances:
                      if a instance is requested (get()), the last element of the array will be returned (and removed from the array).
                      on release() the freed instance will be put back into the array again as the last element.
                    • the pool is cleared frequently (configured by timeout parameter):
                      on timeout all the pooled instances are checked for their last access and instances that were not used since last timeout, are removed.

                     

                    i tested the pool during the last days and it seems to work (meaning i did not recognized any fundamental errors or performance issues yet ; )).

                     

                    maybe you could take a look at the code and inform me in case i did something stupid? O: )

                    thank you for your attention!

                     

                    regards

                    robert

                    • 7. EJB3 Session Bean Pool
                      wolfc
                      • 8. Re: EJB3 Session Bean Pool
                        robert.geisler

                        hi carlo,

                         

                        thank you very much for uploading and commenting the sources.

                        i did not revise the code, yet, but i would like to discuss your statements...

                         

                        i) exception handling as logic


                        you told me not to use exception handling as logic because instantiation of exceptions "take[s] a large hit".

                         

                        well, i guess, for a single execution an if/ else may be faster than throwing/ catching an exception. but please consider how often the exception is actually thrown: only

                         

                        • if the pool is empty (arraySize == 0) and get() is called or
                        • the pool is full (arraySize == array.length) and release() is called.

                         

                        in contrast an if/ else would add an extra operation whenever get() or release() is executed.

                        that is why i excepted the exception handling logic to be the way to go.

                         

                        what do you think?

                         

                        ii) LinkedList vs. array


                        you proposed to use a LinkedList instead of cloning the array.

                         

                        i guess instantitation of arrays is more expensive than instantitation of a LinkedList.Entry, so again you may be right in case of a single execution. but please keep in mind that a LinkedList would be resized whenever get() or release() is called. LinkedList.add() creates a new LinkedList.Entry and links it to the first or last element of the list; LinkedList.remove() unlinks a LinkedList.Entry that later will be destroyed by gc.

                         

                        for an array, resizing occurs only if the pool is full (arraySize == array.length) and another element is released (and should be stored in the array). on get() no resizing (/ instantiation) is necessary.

                         

                        so i guess than an array should beat a LinkedList -in case of frequent calls?

                         

                        iii) LRU


                        you said that the pool is always sorted by LRU (last recently used) and you are right, of course.

                         

                        that means, on timeout not all elements in the pool have to be verified: i may remove the first timedout element and all elements in front of it.

                        so i suggest the following timeout algorithm: on timeout the current arraySize will be remembered as "timeout size". on next timeout this size will be used to find the first timedout element (at "timedout index"). if the element at timeout index (timeout size - 1) is not timedout, timedout index has to be less than timeout index. after timedout index is found (index of first timedout element), all elements at and int front of timedout index will later be removed. all elements beyond timedout index will remain in the pool (after they are moved to index 0 and following!). in this way iteration is reduced to the minimum and unnecessary operations are avoided.

                         

                        do you confirm?

                         

                        regards

                        robert

                        • 9. Re: EJB3 Session Bean Pool
                          dionisior

                          Hi Robert,

                           

                          Good day!

                           

                          I'm having the same problem in session bean instantiation. I just want to know how did you limit StrictMaxPool. What specific xml file did you modify?

                          Thanks in advance.

                           

                          Regards,

                          Romeo

                          • 10. Re: EJB3 Session Bean Pool
                            robert.geisler

                            hello romeo,

                             

                            i modified deploy\ejb3-interceptors-aop.xml . search for the PoolClass annotation, maxSize is the value to limit amount of instances:

                             

                            <aop>

                               ...

                               <domain name="Stateless Bean">

                                  ...

                                  <annotation expr="!class(@org.jboss.annotation.ejb.PoolClass)">

                                     @org.jboss.annotation.ejb.PoolClass (value=org.jboss.ejb3.StrictMaxPool.class, maxSize=15, timeout=10000)

                                  </annotation>

                               </domain>

                               ...

                            </aop>

                             

                            greets

                            robert

                            • 11. Re: EJB3 Session Bean Pool
                              dionisior

                              Thank you so much Robert. It really helped me a lot.