9 Replies Latest reply on Jul 29, 2010 1:47 AM by ruchirc

    Variation in Max Heap size

    rohit.macherla

      Hi ,

       

      I am using JBoss 4.2.3.GA server on a HP UX machine. Recently, our organization planned to buy a tool for performance monitoring of the enterprise applications. We have received reports that the maximum value of the heap is varying at times. As a direct result, I can see the CPU utilization (4 core, 3GHz machine) going up to 70% from a 30%. The funny thing is : actual heap utilization is almost constant (at 750MB)

      I have attached an image of the heap variation. (In the picture, the green line represents the max heap while the red one marks the current utilization. We can notice that about the same time, the CPU utilization also increased.)

       

      In the run.conf file, we have the following JVM parameters :

      JAVA_OPTS="-server -Xms2048m -Xmx2048m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=640m -X:MaxNewSize=640m -XX:SurvivorRatio=16 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseParallelGC -XX:ParallelGCThreads=4"

      As we can see, the whole point of making the min and max heap values to be same (2GB) was to avoid the dynamic heap memory allocation by the OS. I am unable to understand why the max heap is varying even though the actual utilization remains the same.

       

      Please let me know if I miss something.

        • 1. Re: Variation in Max Heap size
          peterj

          It looks like the JVM is ignoring the -Xms setting because at first it is using only about 1.6GB of memory space. I have noticed with JVM 5 and higher that it takes the min and max heap sizes more as suggesstions than as hard-and-fast limits.In addition, the JDK team at HP modifies the JVM sources they get from Sun to produce the version that runs on HP-UX. You might want to as the HP JDK team about this as they are the ones who could give you a definitive answer.

          1 of 1 people found this helpful
          • 2. Re: Variation in Max Heap size
            rohit.macherla

            Hi Peter,

             

            Thanks for the response. I will check with HP team.

            • 3. Re: Variation in Max Heap size
              ruchirc

              Hello,

               

              Can you provide specific details as listed below.

               

              01> What type of platform you are using

              02> What is the total RAM of the Machine

              03> how many CPUs you have on the box and what type of CPUs you have

              04> please avoid using "-server" as a input parameter to the JVM, JDK1.5 onwards dose it automatically for you, it checks the os to get this value

              05> Based on the details which you have provided i think this will work

              -Xmx2048m -Xms2048m -XX:MaxNewSize=256m -XX:NewSize=256m -XX:MaxPermSize=128m -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=60 -XX:LargePageSizeInBytes=5m -XX:ParallelGCThreads=20   -XX:+StringCache -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:ThreadStackSize=1024 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

               

              06> Try to go through this articel "http://community.jboss.org/docs/DOC-15623?uniqueTitle=false" we have tried to explain it.

              • 4. Re: Variation in Max Heap size
                peterj

                See my response to Ruchir at http://community.jboss.org/message/553989#553989

                 

                Moral of the story: never accept anyones suggested settings as the perfect setting for your app - always test and tune. If you are not sure what I mean, then tell me why sun.rmi.dgc.client.gcInterval is set top 60 minutes (ask me about it, I know why it is set to this value and why that value is incorrect in production)

                • 5. Re: Variation in Max Heap size
                  ruchirc

                  You are correct Peter , performance of JVM in different for diff applications, diff platforms, diff capacity..., this is the reason I asked the platform details and other details. You will agree that we are giving pointer to the community to help them solve their problem, as a suggestion.I feel the community is professional enough to test it before placing in prod env.

                   

                  Lets keep this sprit going we will help the community more



                  Thanks,

                  Ruchir Choudhry

                  • 6. Re: Variation in Max Heap size
                    rohit.macherla

                    Hi Ruchir,

                     

                    Thanks for the response. Please find my response for the questions :

                     

                    01> What type of platform you are using

                    Rohit: We are using HP Unix B11.23_LR


                    02> What is the total RAM of the Machine

                    03> how many CPUs you have on the box and what type of CPUs you have

                    Rohit: 4 core CPU, 24GB RAM. Planning to upgrade to 8 core 36GB soon, owing to planned increase in the load.

                     

                    04> please avoid using "-server" as a input parameter to the JVM, JDK1.5 onwards dose it automatically for you, it checks the os to get this value

                    Rohit: Okay. Will test it.

                     

                    05> Based on the details which you have provided i think this will work

                    -Xmx2048m -Xms2048m -XX:MaxNewSize=256m -XX:NewSize=256m -XX:MaxPermSize=128m -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=60 -XX:LargePageSizeInBytes=5m -XX:ParallelGCThreads=20   -XX:+StringCache -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:ThreadStackSize=1024 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

                    06> Try to go through this articel "http://community.jboss.org/docs/DOC-15623?uniqueTitle=false" we have tried to explain it.

                    Rohit: I am half way through the document. We'll make the changes suggested in  amock environment for testing. However, we are more intent on why the heap size is changing.

                     

                     

                    As a general follow up to the chain, after some hiccups with our support with vendor, we were asked to submit a more recent version of the event and we are in the process of finding a relationship between the max heap variation and CPU utilization. Recently discovered a better way to find the heap statistics using twiddle. I'll keep the results posted after our testing.

                     

                    Hi Peter,

                    As far as I know, having remote objects garbage collected will result in a full GC and if an application does not have much involvement for remote objects, the interval of collection can be set to a higher value. I have no idea how to measure whether the heap I have uses remote objects. Currently, we use webservices and all our apps reside on the same JVM. How the external systems access our webservices, whether it creates remote objects or not is still a black area for me. Since I explicitly do not use RMI, I have enabled a one hour DGC.

                    I will greatly appreciate if you can provide some inputs on the same.

                     

                     

                    Thanks.

                    Rohit M.

                    • 7. Re: Variation in Max Heap size
                      peterj

                      An appplication server will always have remote objects, whether you use them directly or not. By default, the RMI code will perform a major collection every minute. The settings in the run scripts change this to every hour. Why every hour? Why not every 30 minutes, or every 24 hours or every week? Where did the recommendation of one hour come from, and what is it based on?

                       

                      At the very first JBoss World in 2005 I presented a talk on JBoss AS performance tuning, using the SpecJAppServer benchmark to do the analysis. A typical benchmark run included starting the app server, trying a web page or two to make certain things were running correctly, running the benchmark, and then shutting the app server down. The benchmark runs for about 20 minutes. We didn't want the RMI collector to fire off a major collection during the benchmark so we changed the delay to be long enough that it would not interfere with the benchmark - 60 minutes. After that talk, the 60 minute RMI setting appreared in the run scripts.

                       

                      The 60 minutes suited out benchmark run - it practically guaranteed that there would be no RMI collection during the time the app server was running. So how long do you typically run the app server before restarting it? A week? A month? Longer? Well, that is what you should base the RMI GC settings on, not an arbitrary recommendation of 60 minutes.

                       

                      Moral of the story: find out why someone recommends a certain setting, then adjust the setting based on your environment and circumstances.

                       

                      Oh, and by the way, the RMI thread that handles the GC uses System.gc(), so setting DisableExplicitGC will prevent the RMI GC from taking place. However, it is still a good idea to set sun.rmi.dgc.client.gcInterval and sun.rmi.dgc.server.gcInterval to a high value to prevent the RMI thread (the one that sleeps for the time interval and then calls Syste,.gc()) from running.

                      • 8. Re: Variation in Max Heap size
                        rohit.macherla

                        Thanks for the information Peter.

                         

                        Cheers,

                        Rohit M

                        • 9. Re: Variation in Max Heap size
                          ruchirc

                          Rothi,


                          I would suggest not to use too big heap size, its very difficult to debug the jvm, if anything goes wrong,  I see your capacity is more then enough, I would suggest split the box into 4 instance using additional IPs, and use software+ hardware base clustering, you can also use apache as a front end to do a mesh cluster.


                          This way your single instance footprint will be smaller and manageable.


                          you can try various JVM settings which is best for your env, some of the examples are there in my white paper, you can also try some of the suggestions given by Peter J, they are valuable inputs which is fine tune based on years of pensive work.


                          Post the problems which you face, we will be happy enough to help you out.


                          Best,

                          Ruchir Choudhry