9 Replies Latest reply on May 24, 2012 10:24 AM by peterj

    High CPU Utilization

    walanj.amol

      Hi ,

       

      We have started our application with following settings.

      $JAVA_OPTS -Xmx4096m -Xms4096m -Xmn2g -Xss128k -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:MaxPermSize=512m -XX:PermSize=256m

       

      We are sending the direct http requst to our application.

      It causing the CPU Utilization in the range of 80-85 %.

       

      When we tried jst returning back from the Method with Null response Our CPU Utilization is in the range of 45-50%

       

      We are using the axis2 webservice in it.

       

       

      Hardware Configuration
      Application Node No of CPU8
      CPU TypeIntel(R) Xeon(R) CPU E5530 @ 2.40GHz, cache size 8192 KB
      Memory8 GB
      Network Card100 mbps Ethernet
      Database serverNo of CPU8
      CPU TypeIntel(R) Xeon(R) CPU E5620 @ 2.40GHz, cache size 12228 KB
      Memory160 GB
      Network Card100 mbps Ethernet

       

      So currently we are struggling to reduce the CPU usage.

        • 1. Re: High CPU Utilization
          wdfink

          What threads of the JBoss-Java process consume the CPU?

          Or is it during runtime (not the first call) and the GC is active?

          • 2. Re: High CPU Utilization
            walanj.amol

            Hi , Thanks for quick reply.

            Its during the runtime when we fired the bulk orders. (Client with 20 threads)

             

            What I had observed from jconsole is , the tomcat threads (http-0.0.0.0-8080-*) are taking more time to complete.

             

             

             

            I had tried to find the top 5 threads consuming the CPU during this run using the shell script.

             

            The Partial O/P is as follows :

             

              PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND                                                    
            11935 sis   15   0 6342m 3.7g  11m S 29.5 47.0   0:10.37 java                                                       
            11681 sis   15   0 6342m 3.7g  11m R 25.6 47.0   0:27.65 java                                                       
            11662 sis   15   0 6342m 3.7g  11m S 23.6 47.0   0:31.89 java                                                       
            11682 sis   15   0 6342m 3.7g  11m S 21.6 47.0   0:31.34 java                                                       
            11663 sis   15   0 6342m 3.7g  11m S 21.6 47.0   0:33.84 java

             

            Pasting here the thread Dump for one of the PID.The others dump also shows the same dump.

             

             

            ========> Java LWP: 11681 - Native Thread ID=2da1

            "http-0.0.0.0-8080-15" daemon prio=10 tid=0x00002aaac0877000 nid=0x2da1 waiting on condition [0x0000000042e7f000]

               java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x0000000746df1920> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)

                    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)

                    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)

                    at org.jboss.security.plugins.JBossAuthorizationManager.internalAuthorization(JBossAuthorizationManager.java:470)

                    at org.jboss.security.plugins.JBossAuthorizationManager.authorize(JBossAuthorizationManager.java:125)

                    at org.jboss.security.plugins.javaee.WebAuthorizationHelper.hasUserDataPermission(WebAuthorizationHelper.java:201)

                    at org.jboss.web.tomcat.security.JBossWebRealm.hasUserDataPermission(JBossWebRealm.java:643)

                    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)

                    at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)

                    at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

                    at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

                    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

                    at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

                    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

                    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

                    at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:905)

                    at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:592)

                    at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2036)

                    at java.lang.Thread.run(Thread.java:662)

             

             

            Thanks,

            Amol

            • 3. Re: High CPU Utilization
              walanj.amol

              Hi had found one more information related to this.
              The CPU Utilization is depend on the maxThreads configured in server.xml for http connector.

              I can see the almost linear increase in CPU Utilzation and TPS with the maxThread.

              When maxThread reaches approx. 75% of the threads from client its shows the CPU utilization in the range of 80-85%

              When I had configured the maxThread=5 , I can see the CPU Utilization in the range of 45-50 % , with drop in TPS almost 40-45%.

              When I had configured the maxThread=8 , I can see the CPU Utilization in the range of 70-75 %, with drop in TPS almost 30-35%.

               

              Also I tried to use the executor in server.xml.
              But there I got the same result as that of initial configuration.

               

              Thanks,

              Amol

              • 4. Re: High CPU Utilization
                wdfink

                I don't catch it, I suppose I did not understand your scenario.

                Do you deploy a WAR application and fire bulk operations with 20 threads parallel.

                The server has maxThreads=5 (8), which will put most of the client threads in a wait state.

                When do you reach the CPU utilization and what TPS mean?

                 

                And what do you want to achieve? More througput or less CPU?

                • 5. Re: High CPU Utilization
                  peterj

                  Do you by any chance work with Rajesh Gupta? See: https://community.jboss.org/message/732025#732025

                   

                  Alos, the more load you put on the system the more threads that will be used to handle the load. This explains the CPU utilization increase under higher load (or when the number of threads increases). Usually there is a "sweet spot" for the number of threads as compared the to number of CPUs you have. Then there is also processor affinity which can play an importanmt role - somtimes the JVM behaves better if it is limited to 2 or 4 CPU even if you have 8 available.

                  • 6. Re: High CPU Utilization
                    walanj.amol

                    My goal is to gain high TPS with avg. response time below 25ms and low CPU Utilization.
                    My client is a normal HttpClient written in Java.

                    Using Thread Pool Executor I manged my client side threads.

                     

                    Peter ,

                     

                    I didn't worked with Rajesh. The issue you mentioned also looks similar to mine.

                    But here you mentioned by increasing the threads the more load on CPU.

                    what I had found from my testing is there is 1-2 % increase when I increase almost 10 threads from the client.

                    But the CPU is largely depend on the maxThreads of the http connector configuration in server.xml under deploy/jbossweb.sar directory.

                     

                    I had calculated the time taken to complete 1 Query.

                    It's in the range of 5-15ms

                     

                    So What I can see that the max CPU is Utilized in Handling connections.

                    So I want to optimize this connections handling so that to reduce CPU with high TPS.

                     

                    Thanks,

                    Amol

                    • 7. Re: High CPU Utilization
                      peterj

                      To reduce CPU utilization, reduce the load! I don't see how you can expect the CPU utilization to go down or remain steady when you add load. (I assume that when you say to added 10 threads to the client that you are using a load testing tool to simulate load and that those ten threads added additonal load on the system.) In all the tests I have done we have constantly increase then load in steps (such as added 10 more simulated users to the load testing tool) and then see what happens at each phase along the way. We usually increase the load until adding more load doesn't yield any more throughput, or even when throughput goes down. That's when we know we have a bottleneck and we hunt it down (and hunting it down takes performance experrts that look at everything from operating system performance data, to database performance data, to JVM performance data, to applicaiton experts that look for hot spots in the application). But trying to keep CPU utilization low while increasing load, and there is no bottleneck yet, is really a waste of time. And at times when the load gets really high and we reach a bottleneck, if it is really just the CPU (that is, not other tuning will work), then we know we have reached the limit on a single machine and will roll in another machine.

                      • 8. Re: High CPU Utilization
                        walanj.amol

                        Hi Peter ,

                        Thanks for reply

                         

                        I agreed with your comment.
                        But our issue is we are getting high CPU for the minimal load itself.
                        I am getting 78-82 % CPU with 12 threads and 80-85 % with 20 threads.
                        And we supposed to have that much load (Concurrent users) in the production system.

                        • 9. Re: High CPU Utilization
                          peterj

                          Sounds like you need to profile your code and find out where it is spending its time.