12 Replies Latest reply on May 29, 2013 4:38 AM by pathduck

    JBoss 7.1: Best way to monitor GC Overhead?

    pathduck

      We're running 7.1.1 Final in production and have run into some OOM situations lately for some of our services.

       

      One problem is, there is no general rule for sizing of the heap (different apps, different memory reqs), so setting a baseline for heapsize to monitor is an almost impossible task. I wish it would be possible to monitor heap used in percentage of max. heap

       

      Anyway... a good indication that something is going wrong is Java GC Overhead approaching 100% so the process is spending all its cycles on GC which means the heap is probably full. For Tomcat it's possible to monitor "Collection Time per Minute" as well as "Process CPU Time per Minute" which allows to set baselines for monitoring.

       

      For JBoss 7 I am not able to find any such measurement, and none available either. There is System Load Average but that is not relevant for only Jboss. There is plenty of heap usage metrics but none for GC Collections.

       

      So any ideas how to monitor and create alerts for java GC Overhead?

       

      -Stian

        • 1. Re: JBoss 7.1: Best way to monitor GC Overhead?
          tsegismont

          There is a "Garbage Collector Resource" defined in AS7 plugin:

           

                <service name="Garbage Collector Resource"
                         discovery="SubsystemDiscovery"
                         class="BaseComponent"
                         description="The management interface for one of the garbage collectors in the Java virtual machine.">
          
          
                  <plugin-configuration>
                    <c:simple-property name="path" readOnly="true" default="name"/>
                    <c:simple-property name="includeRuntime" readOnly="true" default="true"/>
                  </plugin-configuration>
          
          
                  <metric property="collection-count" description="The total number of collections that have occurred."/>
                  <metric property="collection-time" description="The approximate accumulated collection elapsed time in milliseconds."/>
                  <metric property="name" dataType="trait" description="The name representing this garbage collector"/>
                  <metric property="valid" dataType="trait" description="Whether this this memory manager is valid in the Java virtual machine."/>
          
          
                  <resource-configuration>
                    <c:list-property name="memory-pool-names" required="true" readOnly="true" description="The name of memory pools that this garbage collector manages.">
                      <c:simple-property name="memory-pool-names" readOnly="true"/>
                    </c:list-property>
                  </resource-configuration>
                </service>
              </service>
          

           

          Unfortunately the "collection-count" and "collection-time" metric definitions don't have the dataType="trendsUp" attribute. If they did, per-minute metrics would be automatically computed.

           

          Would that fit your need?

          1 of 1 people found this helpful
          • 2. Re: JBoss 7.1: Best way to monitor GC Overhead?
            pathduck

            Thanks a lot for your reply Thomas

             

            Yes definitely this would help a lot - having per-minute metrics would allow creation of a monitor an time spent on GC per minute. So if it uses more than 58000ms/min on GC would mean a overhead of 96% which would be critical for performance.

             

            Do you know where a "Process CPU time/min" metric would be defined?

            • 3. Re: JBoss 7.1: Best way to monitor GC Overhead?
              tsegismont

              Hi Stian,

               

              I have created a Bugzilla entry and did the change in master branch:

              https://bugzilla.redhat.com/show_bug.cgi?id=962379

               

              As for the "Process CPU time/min" metric, Tomcat plugin inherits this one from JMX plugin. AS7 plugin does not extend it.

               

              As a workaround, you can import your AS7 server as a JMX server: Inventory > Child resources > Import JMX server

               

              Sample config for a standalone server:

               

              Type: JSR160

              Connector address: service:jmx:remoting-jmx://127.0.0.1:9999

              Principal: ManagementRealmUser

              Credentials: ManagementRealmPassword

              Additional Class Path Entries: /path/to/jdk1.7.0_17/lib/jconsole.jar,/path/to/jdk1.7.0_17/lib/tools.jar,/path/to/jboss-as-7.1.1.Final/modules/org/jboss/remoting3/remoting-jmx/main/remoting-jmx-1.0.2.Final.jar,/path/to/jboss-as-7.1.1.Final/modules/org/jboss/remoting3/main/jboss-remoting-3.2.3.GA.jar,/path/to/jboss-as-7.1.1.Final/modules/org/jboss/logging/main/jboss-logging-3.1.0.GA.jar,/path/to/jboss-as-7.1.1.Final/modules/org/jboss/xnio/main/xnio-api-3.0.3.GA.jar,/path/to/jboss-as-7.1.1.Final/modules/org/jboss/xnio/nio/main/xnio-nio-3.0.3.GA.jar,/path/to/jboss-as-7.1.1.Final/modules/org/jboss/sasl/main/jboss-sasl-1.0.0.Final.jar,/path/to/jboss-as-7.1.1.Final/modules/org/jboss/marshalling/main/jboss-marshalling-1.3.11.GA.jar,/path/to/jboss-as-7.1.1.Final/modules/org/jboss/marshalling/river/main/jboss-marshalling-river-1.3.11.GA.jar


              Cheers,

              Thomas

              • 4. Re: JBoss 7.1: Best way to monitor GC Overhead?
                pathduck

                Thanks Thomas, that's really helpful

                 

                Too bad the Process CPU Time can't be calculated the same way. We have way too many servers to mess around with having to connect to them using JMX as well I'm afraid.

                 

                Anyway the GC/minute is really helpful and is a great early indicator the JVM is having trouble. Can it be expected in 4.8?

                 

                Is there any possibility more of the JMX standard metrics can be extended through the plugin in the future?

                 

                I notice there is a "jmx" child resource in the AS Standalone Server, but there seems to be no metrics apart from Availability.

                 

                cheers,

                Stian

                • 5. Re: JBoss 7.1: Best way to monitor GC Overhead?
                  tsegismont

                  You're welcome. The change is in master branch and it will be included in RHQ 4.8

                   

                  The JMX child resource you see in AS standalone server is in fact the management resource for the AS7 JMX subsytem.

                   

                  There is currently no plan to embed JMX plugin resources in AS7 plugin. The best approach IMO would be to ask the Wildfly team to add "ProcessCpuTime" as a runtime attribute of the "platform-mbean/operating-system" node in the management tree ("platform-mbean/operating-system" is the equivalent in AS7 management tree of the "java.lang:type=OperatingSystem" MBean).

                   

                  For the record, this what you get when you query the JBoss CLI:

                   

                  [standalone@127.0.0.1:6999 type=operating-system] pwd
                  /core-service=platform-mbean/type=operating-system
                  
                  [standalone@127.0.0.1:6999 type=operating-system] :read-resource(include-runtime=true, recursive=true) 
                  {
                      "outcome" => "success",
                      "result" => {
                          "name" => "Linux",
                          "arch" => "amd64",
                          "version" => "3.8.11-100.fc17.x86_64",
                          "available-processors" => 4,
                          "system-load-average" => 0.43,
                          "object-name" => "java.lang:type=OperatingSystem"
                      }
                  }
                  

                   

                  Thomas

                  • 6. Re: JBoss 7.1: Best way to monitor GC Overhead?
                    pathduck

                    Thank you Thomas, you guys are fast

                     

                    In the 4.8 Snapshot I received there is now a Schedule for "Collection Count/Time per Minute" under platform mbean > garbage-collector > Garbage Collector Resources.

                     

                    The Description is not completely correct but as I understand it, this is inherited so can't be changed.

                     

                    The interval is 40 minutes which is much too seldom to use for real monitoring so it needs to be changed to maybe 10 minutes.

                    I can change it for a lot of servers using groups I guess - but should it not also be available under the Metric Collection Templates > Servers > JBossAS7 Standalone ? That way it would be possible to set for all resources of type "Garbage Collector".

                     

                    Am I looking at the wrong place?

                     

                    -Stian

                    • 7. Re: JBoss 7.1: Best way to monitor GC Overhead?
                      pilhuhn

                      If you only updated the plugin but did not do a refresh the UI (shift-f5 ) then the UI may have the old settings in the templates still chached. Try to shift-f5 the ui.

                      • 8. Re: JBoss 7.1: Best way to monitor GC Overhead?
                        pathduck

                        I tried to refresh the cache and also restart the RHQ server and the browser, but still can't find it.

                         

                        In fact, there is no "platform mbean" or Garbage Collector under JBoss AS7 Standalone at all.

                        In fact the only templates for most of the things there is just "Availability".

                         

                        I think I am doing this wrong somehow, not looking at the correct place?

                         

                        -Stian

                        • 9. Re: JBoss 7.1: Best way to monitor GC Overhead?
                          tsegismont

                          These metrics should appear in the "Metric Collection Template" page. Looks like a bug. Created a new Bugzilla entry: https://bugzilla.redhat.com/show_bug.cgi?id=962845

                           

                          Cheers

                          Thomas

                          • 10. Re: JBoss 7.1: Best way to monitor GC Overhead?
                            pathduck

                            Hey,

                            Had to dig this one up again since I was looking to enable some metrics and could not find any in the Templates and started digging around Google.

                             

                            So - Is this related to: https://issues.jboss.org/browse/AS7-4514 ?

                             

                            And where excactly should the "Platform-Mean" collection templates be located under JBoss AS7 Standalone?

                             

                            thanks,

                            Stian

                            • 11. Re: JBoss 7.1: Best way to monitor GC Overhead?
                              tsegismont

                              It's not related to https://issues.jboss.org/browse/WFLY-857. These are new metrics we would like to add.

                               

                              All existing metrics should appear in the "Metric Collection Template" page. There was a bug and Mazz has fixed it in master. Have a look at the details of https://bugzilla.redhat.com/show_bug.cgi?id=962841 (upstream bug of  https://bugzilla.redhat.com/show_bug.cgi?id=962845).

                              • 12. Re: JBoss 7.1: Best way to monitor GC Overhead?
                                pathduck

                                Ok, so it will be there in 4.8 I guess

                                 

                                A workaround would probably be to just use a dynagroup of Jboss AS7 instances and set the metrics to be collected there...

                                 

                                Thanks,

                                Stian