We are using jboss5.1.0 GA with axis2 web services exposed to client.
we have observed that there is high CPU utilization when ever a request is served. to trace the cpu time we have removed complete code from our method and just returning an empty string for that perticular web service call. here also the CPU utilization is around 45 to 50 %. From application side not doing antything just returning an empty string as a response to the web service client call. where this CPU time is getting used?. we are using 'default' configuration for jboss. how to reduce the CPU utilization ? can anyone help me out here in reducing the CPU time
When a request comes in, and a thread is allocated to service that request, that thread will use up 100% of a core while it is running (unles it has to mwait on somethign such as a database request). Of course, it should be running only briefly (in the 10s of milliseconds). High CPU utilization for a brief time is not a problem. Why do you think it would be? Have you tried running a load test? That is the only way to judge the performance of your system.
By the way, you might want to compare a empty servlet request with an empty axis2 request to see what overhead axis2 is introductiong. I would set up load test for both cases. (I did an empty servlet test several years back and JBoss AS easily served way over 10K requests per second.)
Thank you for your reply.
I did few load tests to understand the CPU utilization.
In our application we are using Mysql5.5 enterprise edition having 40 million records. External client will only request one record details at a time.
1) I performed a load test with 1 million requests(read), i observed that the CPU is around 81 to 85 % throughput is 1200 per/sec with a response time of 15 milli seconds
2) Tested with empty Axis2 servlet. now CPU utilization is around 45 to 50 % (from this we can say our app is taking 30%-35 % of CPU). I just wanted to know is there any way to reduce CPU utilization here (from jboss perspective)? beacuse reducing CPU is our goal.
You should still try testing with an emopty servlet (no axis2). In fact, don't even deploy your app, create a second app with the empty servlet. That will tell you what overhead axis2 is introducing.
Usually the only way to reduce CPU utilization are:
a) Run on a bigger system (more cores)
b) Run on multiple systems and split your workload among them.
What have you done to tune the JVM? Specifically heap tuning? Have you monitored GC activity?
We tried different parametrs of GC and tested them too finally we are with the following GC params(as it is giving good TPS with minimal pause time of app).
JAVA_OPTS="$JAVA_OPTS -Xmx4096m -Xms4096m -Xmn2g -Xss128k -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:MaxPermSize=512m -XX:PermSize=256m
But we didnt find any drop/increase in CPU utilization with all the test we performed with different combination and different type of GC collector.
A couple of comments.
-Xmn2g: I would not let the young generation get up to half the heap size, you too often end up with too many full collection. I'd limit it to 1gb (I usually recommend 1/3 to 1/4 the heap size as a starting point for tuning). This is especially important when using the CMS collector because while the CMS collector is running you will loose the use of one of your cores because it will be busy running the collector.
-XX:ParallelGCThreads=8: I hope that you have at least 8 cores. Never set this to more cores than you have. The JVM by default sets this to the number of cores (for up to 8 cores, but after that it does 6 threads per 8 cores IIRC.)
You still haven't figured out the overheap introduced by Axis2 (not sure why you keep on ignoring this, especially since it is critical to resolving the issue you keep on brining up)