10 Replies Latest reply on May 14, 2013 6:59 PM by andy.miller

    Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?

    toca

      In JBoss-5.1.0.GA I used a JMX client (which in turn used the twiddle.bat shipped with JBoss) to return values for "currentThreadsBusy", "currentThreadCount" and "maxThreads" from  jboss.web:type=ThreadPool,name=http-0.0.0.0-8080".

       

      I've setup a similar twiddle.bat within my JBoss-as-7.1.1.Final instance, but cannot find all three of these values in the current MBeans exposed.

       

      Does anyone know where I can access this information? Or is this info no longer exposed in JBoss AS 7.1?

       

      Thanks!

       

      UPDATE: I found the corresponding value for "maxThreads" via the attribute "maxConnections" in the ObjectName "jboss.as:subsystem=web,connector=http".

      (This setting can be changed by adding "max-connections=NNN" to the connector element for the "urn:jboss:domain:web:1.1" subsystem in standalone.xml.)

        • 1. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
          andy.miller

          I don't know about JMX, as its not really our primary management method/protocol anymore.  You can get good information on the web threads through the CLI, which uses our domain model for management.  There is a caveat to all this.  The JBossWeb internal thread pool, does not expose any thread metrics.  In order to get thread metrics you have to use an external executor, and JBoss Threads.  You do this, with the following type of configuration.  This is just an example, and there are quite a few thread pool types to choose from:

           

          <subsystem xmlns="urn:jboss:domain:threads:1.1">

                      <unbounded-queue-thread-pool name="JBossWeb">

                          <max-threads count="60"/>

                          <keepalive-time time="75" unit="minutes"/>

                      </unbounded-queue-thread-pool>

                  </subsystem>

           

          ...

           

          <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="true">

                      <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" enable-lookups="false" executor="JBossWeb" max-connections="3260"/>

                      <virtual-server name="default-host" enable-welcome-root="true">

                          <alias name="jbosstesting.miller.org"/>

                      </virtual-server>

                  </subsystem>

           

          I bolded the key areas that tie together.  Once you have a configuration like this, which I would recommend, because you get a lot more flexibility in how threads are managed (the internal thread pool uses max-connections as the maxThreads parameter, so if you have a large number of connections you need to support you are going to get a large thread pool max, which is not typically what you want or need).

           

          In the CLI, you can go to the following address (for this example):

           

          /subsystem=threads/unbounded-queue-thread-pool=JBossWeb

           

          This is specific to my example.  With your configuration, you would replace the thread pool type, with whatever you configured, and the name would be what name you gave the executor in the connector configuration.

           

          Then you can do a :read-resource:(include-runtime=true,recursive=false), and you will get the following type of response:

           

          [standalone@jbosstesting.miller.org:9999 unbounded-queue-thread-pool=JBossWeb] :read-resource(include-runtime=true,recursive=false)

          {

              "outcome" => "success",

              "result" => {

                  "active-count" => 0,

                  "completed-task-count" => 0L,

                  "current-thread-count" => 0,

                  "keepalive-time" => {

                      "time" => 75L,

                      "unit" => "MINUTES"

                  },

                  "largest-thread-count" => 0,

                  "max-threads" => 60,

                  "name" => "JBossWeb",

                  "rejected-count" => 0,

                  "task-count" => 0L,

                  "thread-factory" => undefined

              }

          }

           

          The runtime attributes are, in this example, are active-count, completed-task-count, current-thread-count, largest-thread-count, rejected-count and task-count.

           

          The other attributes are from the configuration, and will not change.  I will warn you, that depending on which thread pool you configure, some of the runtime attributes have different names.

           

          This can all be scripted, and can be done remotely as well.  I believe there are some examples of how you can do this stuff via various scripting languages elsewhere.  Personally, I have only done this through a Java client, so I wouldn't be much help in other languages.

           

          For the other thread pools, here are the runtime attributes:

           

          blocking-bounded-queue-thread-pool ->  current-thread-count, largest-thread-count, rejected-count

          blocking-queueless-thread-pool -> current-thread-count, largest-thread-count, rejected-count

          bounded-queue-thread-pool -> current-thread-count, largest-thread-count, rejected-count

          queueless-thread-pool -> current-thread-count, largest-thread-count, rejected-count

          scheduled-thread-pool -> active-count, current-thread-count, completed-task-count, largest-thread-count, task-count

          unbounded-queue-thread-pool -> active-count, current-thread-count, completed-task-count, largest-thread-count, rejected-count, task-count

           

          If you go down this path, you will have much better ability to configure the threads properly, and you will get the runtime statistics you need.  You will also find there is a rich set of information exposed through the management interface, with all the configuration and runtime attributes for all the various subsystems.  Plus, through the management interface, you can change configuration, not just read information out.

          • 2. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
            fsl

            Thanks a lot. While I'd prefer JMX because many monitoring tools support it out-of-the-box, I can live with a custom script based on your sugestion. And using an executor is the recomended way from the tomcat community.

             

            It seems running a production instance of AS 7.x / EAP 6.x would require substancial customization of the default configs. While this is expected and I am used to do this on AS/EAP 5.x the changes are not obvious from either the AS 7.x configuration files, web or cli management interfaces, like they were for previous releases.

             

            I found no hints about this on the docs for AS and EAP. Did I miss something, can you point me to a best practices document about configuring EAP for production?

            • 3. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
              andy.miller

              I'm not sure about the docs, but you should take a look at the EAP docs, if you have a subscription.  With that said, the configuration is much simplier than in the past, and everything is in one file.  Also, everything has a defined schema, so you can look at the schema files to see what all the options are.  Of course, that's not going to tell you the appropriateness of any of the options based on your particulars.

               

              The schemas are all here:

               

              https://github.com/wildfly/wildfly/tree/master/build/src/main/resources/docs/schema

               

              Just make sure you look at the correct version for anyone you may be looking at.

               

              I did a performance tuning presentation for EAP 6 at JBossWorld last year too, and that has a lot of configuration information in it.  I think you can still find that on-line at:

               

              http://rhsummit.files.wordpress.com/2012/03/miller_accelerate_your_jboss.pdf

               

              That may be of help to you.  All configuration changes can be done through the CLI, or by editing the configuration files directly.  Also, the new web console can do quite a bit, but I don't believe it can do every kind of change, just yet.

              • 4. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
                fsl

                I said I did look the EAP docs, not just the AS ones :-) Sure the'll improve over time, but I feel a general lack of "best practices". :-(

                 

                While I agree the new configuration files are much simpler to work for and the cli is much better than plain MBeans, the old configuration files had the advange that most usual alternatives were present as comments, and most of the time each parameter was documented with comments. Maybe the Red Hat EAP team could work on more configuration examples and some kind of "javadocs" for the management API, which could be exposed by the cli. I think it got harder to find out which setting to change and how it relates to other settings.

                 

                I'm also used to working with the schemas, it's very nice to be able to use an (XML) editor with autocomplete and syntax warnings for configuration files.

                 

                Thanks for the presentation link.

                • 5. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
                  toca

                  Thanks for the quick and thorough reply Andy, much appreciated. I've setup an executor as in your example, and queried the information via the JBoss CLI. I'm sure I can use this information to fulfill my requirements.

                   

                  As a quick follow-up:

                  1) what will JBoss do if I don't define an executor (i.e does it use any of those you've listed by default? - and if so, what are the default settings?)

                  2) what are the defaults if I define an executor without setting max-connections? (I note this is an optional element in the schema.)

                  3) When I use the CLI in non-interactive mode the comma is used to separate commands, so the comma in the command "/subsystem=threads/unbounded-queue-thread-pool=JBossWeb/:read-resource(recursive=false,include-runtime=true)" is interpreted as a command separator...is there a way to escape the comma character using non-interactive mode, or should I resort to putting my command in a file and referencing that on the command line?

                   

                  Thanks for the link to the schemas and the presentation too by the way. If only I had a few months to look into JBoss performance and tuning :-)

                  • 6. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
                    andy.miller

                    To Ca wrote:

                     

                    Thanks for the quick and thorough reply Andy, much appreciated. I've setup an executor as in your example, and queried the information via the JBoss CLI. I'm sure I can use this information to fulfill my requirements.

                     

                    As a quick follow-up:

                    1) what will JBoss do if I don't define an executor (i.e does it use any of those you've listed by default? - and if so, what are the default settings?)

                    2) what are the defaults if I define an executor without setting max-connections? (I note this is an optional element in the schema.)

                    3) When I use the CLI in non-interactive mode the comma is used to separate commands, so the comma in the command "/subsystem=threads/unbounded-queue-thread-pool=JBossWeb/:read-resource(recursive=false,include-runtime=true)" is interpreted as a command separator...is there a way to escape the comma character using non-interactive mode, or should I resort to putting my command in a file and referencing that on the command line?

                     

                    Thanks for the link to the schemas and the presentation too by the way. If only I had a few months to look into JBoss performance and tuning :-)

                    1)  If you do not define an executor, it will use the internal thread pool to JBossWeb/Tomcat.

                    2)  The defaults, if you don't set max-connections are computed based on the number of cores in the system.  I forget the exact calculation that is used.  I would probably have to ask the JBossWeb team.  Now, that is if you are not using the native connector.  If you use the native connector the default is 512 for the thread pool.

                    3) The following command worked fine for me:  /subsystem=threads/unbounded-queue-thread-pool=JBossWeb/:read-resource(include-runtime=true,recursive=false)   I think you put recursive=false first, vs. include-runtime first in the :read-resource command, and it has to be the other way around.

                    • 7. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
                      andy.miller

                      My comment on the order of the attributes on the :read-resource command doesn't matter, I just successfully did it the other way around.  I'm not sure what is happening there.

                      • 8. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
                        andy.miller

                        I just did the following at the command-line, and it worked fine, so perhaps this will help you compare to what you are doing, and see the difference:

                         

                        date;./jboss-cli.sh --controller=jbosstesting.miller.org:9999 --connect --command="/subsystem=threads/unbounded-queue-thread-pool=JBossWeb/:read-resource(recursive=false,include-runtime=true)";

                         

                        Hopefully, this will reveal the difference and why yours is not working.

                        • 9. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
                          toca

                          Once again, a very thorough response, gratefully received!

                           

                          My 'mistake' was not to use the "--command=" prefix:

                           

                          i.e my command line was:

                          date;./jboss-cli.sh --controller=jbosstesting.miller.org:9999 --connect /subsystem=threads/unbounded-queue-thread-pool=JBossWeb/:read-resource(recursive=false,include-runtime=true)

                           

                          Using the command line as you've shown now works for me. Thanks for all your help Andy.

                          • 10. Re: Does JBoss AS 7.1 expose MBeans showing currentThreadsBusy, currentThreadCount and maxThreads?
                            andy.miller

                            Okay, great.  Glad that you were able to figure it out.