1 2 3 Previous Next 32 Replies Latest reply on Sep 5, 2009 4:54 PM by peterj

    JBoss Performance Tuning

    tdanecito

      Hi All,

      To better tune JBoss,

      1. Use the fastest JVM (Mustang I believe)
      2. Tune the JVM
      3. Use caching in Servlet container where possible
      4. Put EJB in same class loader as Servlets
      I have measure 1-2msec (P4 3Ghz) just to call across Servlet-EJB boundry in same classloader and maybe 100msec if WAR and EAR loaded in seperate classloaders

      Okay with that said and even a lot more tuning then I have time to mention I was getting total transactions in under 20msec over the Internet. Yep it can be done. I was shooting for 100,000 transactions per second in App server and I got near 50,000 on an old P4 3Ghz machine.


      Good luck,
      -Tony

        • 1. Re: JBoss Performance Tuning
          tdanecito

          Okay due to the level of intrest and the emails I have recieved I have decided every couple of days to leave some more tuning information. The info will cover all levels of a system design with emphasis on the best design based upon performance.


          With that in mind I wanted to mention that the next recommendation would be to put Apache in front as a web server since my measurements have indicated it is much faster for http then the web server functionality provided by Jboss.

          Thats the tip for today.

          Regards,
          -Tony

          • 2. Re: JBoss Performance Tuning
            tdanecito

            Okay here is the next installment in overall performance tuning.

            Today is Sun JVM setting day. I use the following for JBoss:

            -server -XX:NewSize=356m -XX:SurvivorRatio=16 -XX:MaxNewSize=356m -Xms400m -Xmx400m

            The basics are:
            1. Always set max heap size to min heap size.
            2. Always set new generation size as appropiate but again set its min an max size to be the same.
            3. Set Survivor size small relative to new generation.

            Why these setting in this manner is that you can for 2x-5x performance inprovement because the CPU is not busy monitoring resizing resources such as memory. The new generation size is set to maximum for me because if you have short lived objects it is best to keep them in the new generation area rather then going to the survivor spaces.

            The main thing to remember is you need to monitor your own usage to determine the best settings for your application.

            Good Luck All,
            -Tony

            • 3. Re: JBoss Performance Tuning
              peterj

              I have to disagree with your proposed heap settings. You have set the young generation to use most of the heap, and have allowed for only a very small tenured generation. Using your settings, every collection will be a major collection -- the Sun JVM performs a major collection if there is not enough space to hold all of the objects in the young generation during a collection (this is one of the things that we tweaked in our JVM, which is based on Sun's source -- we calculate how many objects will be tenured first before determining if we need a major collection). Major collections tend to be fairly expensive, thought that is somewhat reduced based on the size of your tenured generation.

              Recommeded heap ratios are for the tenured generation to be approximately two times the size of the young generation. Our recommended sizes are 1200MB for the heap and 400MB for the young generation. Of course, we also recommend that you have 2GB of RAM to do this. And by the way, this recommendation is only a starting point, you have to tune from there and to do that you have to gather and analyze the garbage collection statistics. I have written several white papers on this topic, they are available at www.cmg.org, but you have to be a CMG member to see them. If there is interest, I will see about posting them somewhere more accessible.

              • 4. Re: JBoss Performance Tuning
                tdanecito

                Hi PeterJ,

                Thanks for the response I appreciate it. In case you did not notice I mentioned that the heap should be adjusted to meet the needs of the application and the settings I mentioned were for my own application. What I hopefully pointed out also that the main emphasis is on getting the CPU to not be busy resizing the tenured and new generation areas. The developers/leads should always measure then measure again the performance gains for each setting under varying loads.
                I have done this and have made great gains on production systems over the years following this process.

                One of my goals was to avoid major collections which I was able to do since the objects were very short life span (message objects for interfaces mainly). Other applications will probably have singletons or caches which I expect would end up being tenured so what you say would be true.

                I would be interested in seeing any reseach papers you have written since papers do have great ideas.

                Even though I see on the web the message heap sizes depends on application usage it is really true.

                Again thanks for the feedback.
                -Tony

                • 5. Re: JBoss Performance Tuning
                  tdanecito

                  Hi All,

                  Another tip to mention is use of startup options for JBoss. I use the default startup instead of the All option. I did not need the services provided by the All option but I did need EJB support and I found a recommendation a year ago that from the JBoss CTO to reduce resources and improve performance to use the Default option. I measured performance before and after and sure enough he was right. Again if the CPU is occupied providing more services reduction of those services will improve performance.

                  Also, I forgot to answer one of the good points PeterJ mentioned regarding use of the tenured (old) generation area of the heap. If you do not need the heap space there do not size it too big because the garbage collection takes longer in the tenured area the more space you allocate even if you do not use all of it.

                  I only used 400MB for the heap because my server I was experimenting with only had 1GB. But even if it was a production server I would only size it to what is needed but probably start off with what PeterJ mentioned. Never be afraid to experiment to find the performance sweet point try new things cheaper to do it before production than find out later.

                  Got to go.
                  Regards,
                  -Tony

                  • 6. Re: JBoss Performance Tuning
                    tdanecito

                    Hi All,

                    Just one last posting regarding performance since I am getting quite busy on other interesting projects (hobbies) at home. Once you can get down to the usec response time range you might try tuning Apache Web Server. There are several recommnedations on the web but turning off logging and setting Allow Overide to None makes a difference. Turning off logging is not something I would recommend normally but it does yield results.

                    Along the same lines make sure you do not write to the console or Application logs (such as log4J) unless you have errors. Can not tell you how many times I have seen developers abuse the logs and console and the surprise in their faces when they see the time difference when setting both to reasonable levels.


                    Thanks for all the interest in this thread and good luck on all your projects!!

                    Regards,
                    Tony

                    • 7. Re: JBoss Performance Tuning

                      Hi,

                      Could you tell me how to put EJB in same class loader as Servlets. Cuurently I have an EAR, contains jar, har and war files,
                      How can I tell they are using the same classloader ?

                      Thank you very much.

                      chwang

                      • 8. Re: JBoss Performance Tuning
                        peterj

                        chwang, since your question is not relevant to the topic of this thread, you should start a new topic. Also, since your question is not performance related, you should post the question to a different forum, either [Beginners Corner] or [Installation, Configuration & Deployment] (but not to both!)

                        • 9. Re: JBoss Performance Tuning

                          Hi, Tony, Also I ahve a question
                          about set JVM,

                          you said set jvm

                          -server -XX:NewSize=356m -XX:SurvivorRatio=16 -XX:MaxNewSize=356m -Xms400m -Xmx400m

                          But I set JVM in run.bat
                          set JAVA_OPTS=%JAVA_OPTS% -Xms128m -Xmx812m

                          I don't where to set heap size in Jboos, please let me know. Thank you.

                          chwang

                          • 10. Re: JBoss Performance Tuning
                            tdanecito

                            Hi,

                            It is application dependent. I know not the best answer. For performance I would monitor the application using JConsole (JVM 1.5). If you do not have that then use verboseGC setting to print out heap contents. What you are looking for is the frequency of "tenured" gc's. If you have more than one an every 10 minutes I would increase the new generation size. Hopefully all your objects are short term and will be collected.

                            Start out with 512MB and decrease/increase based upon monitoring and some sort of load testing. Again, make sure the min/max size of the heap is the same and adjust the new/tenured area based upon some sort of load testing.


                            What JVM are you using?

                            Regards.
                            Tony Anecito
                            adanecito@yahoo.com

                            • 11. Re: JBoss Performance Tuning
                              tdanecito

                              Okay,

                              I am tuning to a new level (network) and decided to mention a few things.
                              Use a tool such as TCPOptimizer.

                              Adjust your MRU, MTU such that it meets you needs. For example if you transmit only a couple hundred bytes you can adjust it to that size say versus the standard 1500 bytes. If you transmit say 3000 bytes then the 1500 byte plus ACK can be quite an issue over long distances and the bigger the message the more chatty it can be.

                              This may not be an issue if you are down to around 20msec or less over short distances from the server (10 miles max) on the internet but I am working the long distance now (1500 miles) where a simple 1500 byte transmit and 3000 byte return take around 300 milliseconds through who knows how many routers.

                              Also, I found this good link thanks to PeterJ on another thread. It backs up what I said about reducing services and a few other tuning hints.

                              http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossASTuningSliming

                              I am down to 100,000 transactions per second per thread at the servlet layer but still working on getting the Web Service Layer to be better than 4-5 milliseconds to get through that layer thus the TCP stack tuning and some other methods I will try soon.

                              I have yet to upgrade from the old P4 to say a AMD64 X2 4200 but hopefully some day soon then I will report more on those results. My goal is to get to over 1 million transactions per second with total round trip time of less than 5 milliseconds over a distance of 10 miles.

                              Good Luck,

                              Tony Anecito
                              adanecito@yahoo.com

                              • 12. Re: JBoss Performance Tuning
                                tdanecito

                                Okay,

                                I figured out the main issue for the 1500 mile distance is the limitation due to EM fields traveling at a maximum the speed of light. The time to take the first bit of a packet is around 80msec. Thus if you have a request plus the response then 160 milliseconds is used up. If you have several routers involved that may put you up to 300 milliseconds.
                                Thus to go from coast to coast a couple of tenths of a second will be used for dynamic data.

                                Just a FYI.

                                Regards,
                                Tony Anecito
                                adanecito@yahoo.com

                                • 13. Re: JBoss Performance Tuning
                                  tdanecito

                                  Okay,

                                  More about networks. For internal users you might want to check out extended or jumbo frames. The are several white papers on the subject but the limitation is for internal sites where you have gigabit lan where the client and server hardware supports. The test results in at least one paper indicate a 2x perfomance improvement. This will reduce the CPU loading on the web container and the client.

                                  Regards,
                                  Tony Anecito
                                  adanecito@yahoo.com

                                  • 14. Re: JBoss Performance Tuning
                                    tdanecito

                                    Hi All,

                                    Just wanted to mention I am using JAX-WS 2.0 for my web service layer. Even though it is a reference implementation it still has been fast compared to axis. The important thing to know about Web Service layers for performance are:

                                    1. Use fast infoset.
                                    2. Use Document literal
                                    3. Use primatives and array of primatives instead of collections
                                    4. Keep messages small (below 1500 bytes)

                                    I would only use Web Service layer for B to B but would advise for large objects that are returned from a Web Service to use RMI if used inside a company.

                                    Have Fun,
                                    Tony Anecito
                                    adanecito@yahoo.com

                                    1 2 3 Previous Next