Hi , Thanks for quick reply.
Its during the runtime when we fired the bulk orders. (Client with 20 threads)
What I had observed from jconsole is , the tomcat threads (http-0.0.0.0-8080-*) are taking more time to complete.
I had tried to find the top 5 threads consuming the CPU during this run using the shell script.
The Partial O/P is as follows :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11935 sis 15 0 6342m 3.7g 11m S 29.5 47.0 0:10.37 java 11681 sis 15 0 6342m 3.7g 11m R 25.6 47.0 0:27.65 java 11662 sis 15 0 6342m 3.7g 11m S 23.6 47.0 0:31.89 java 11682 sis 15 0 6342m 3.7g 11m S 21.6 47.0 0:31.34 java 11663 sis 15 0 6342m 3.7g 11m S 21.6 47.0 0:33.84 java
Pasting here the thread Dump for one of the PID.The others dump also shows the same dump.
========> Java LWP: 11681 - Native Thread ID=2da1
"http-0.0.0.0-8080-15" daemon prio=10 tid=0x00002aaac0877000 nid=0x2da1 waiting on condition [0x0000000042e7f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000746df1920> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Hi had found one more information related to this.
The CPU Utilization is depend on the maxThreads configured in server.xml for http connector.
I can see the almost linear increase in CPU Utilzation and TPS with the maxThread.
When maxThread reaches approx. 75% of the threads from client its shows the CPU utilization in the range of 80-85%
When I had configured the maxThread=5 , I can see the CPU Utilization in the range of 45-50 % , with drop in TPS almost 40-45%.
When I had configured the maxThread=8 , I can see the CPU Utilization in the range of 70-75 %, with drop in TPS almost 30-35%.
Also I tried to use the executor in server.xml.
But there I got the same result as that of initial configuration.
I don't catch it, I suppose I did not understand your scenario.
Do you deploy a WAR application and fire bulk operations with 20 threads parallel.
The server has maxThreads=5 (8), which will put most of the client threads in a wait state.
When do you reach the CPU utilization and what TPS mean?
And what do you want to achieve? More througput or less CPU?
Do you by any chance work with Rajesh Gupta? See: https://community.jboss.org/message/732025#732025
Alos, the more load you put on the system the more threads that will be used to handle the load. This explains the CPU utilization increase under higher load (or when the number of threads increases). Usually there is a "sweet spot" for the number of threads as compared the to number of CPUs you have. Then there is also processor affinity which can play an importanmt role - somtimes the JVM behaves better if it is limited to 2 or 4 CPU even if you have 8 available.
My goal is to gain high TPS with avg. response time below 25ms and low CPU Utilization.
My client is a normal HttpClient written in Java.
Using Thread Pool Executor I manged my client side threads.
I didn't worked with Rajesh. The issue you mentioned also looks similar to mine.
But here you mentioned by increasing the threads the more load on CPU.
what I had found from my testing is there is 1-2 % increase when I increase almost 10 threads from the client.
But the CPU is largely depend on the maxThreads of the http connector configuration in server.xml under deploy/jbossweb.sar directory.
I had calculated the time taken to complete 1 Query.
It's in the range of 5-15ms
So What I can see that the max CPU is Utilized in Handling connections.
So I want to optimize this connections handling so that to reduce CPU with high TPS.
To reduce CPU utilization, reduce the load! I don't see how you can expect the CPU utilization to go down or remain steady when you add load. (I assume that when you say to added 10 threads to the client that you are using a load testing tool to simulate load and that those ten threads added additonal load on the system.) In all the tests I have done we have constantly increase then load in steps (such as added 10 more simulated users to the load testing tool) and then see what happens at each phase along the way. We usually increase the load until adding more load doesn't yield any more throughput, or even when throughput goes down. That's when we know we have a bottleneck and we hunt it down (and hunting it down takes performance experrts that look at everything from operating system performance data, to database performance data, to JVM performance data, to applicaiton experts that look for hot spots in the application). But trying to keep CPU utilization low while increasing load, and there is no bottleneck yet, is really a waste of time. And at times when the load gets really high and we reach a bottleneck, if it is really just the CPU (that is, not other tuning will work), then we know we have reached the limit on a single machine and will roll in another machine.