4 Replies Latest reply on May 31, 2004 8:40 PM by treespace

    200% performance increase with remote debugging on.

    pclear

      Hi.

      We've been running performance testing on our web application on one of our production servers lately and have encountered an interesting performance item. It is probably not related to JBoss, but I thought this community may have some interest and possibly some insight into this issue.

      We have an application which uses a large number of Stateless Session Beans and no Entity Beans with a Tomcat web frontend. We've been testing it with the Radview Webload product. We ran into a problem with our testing where the JMS implementation would stall (a known issue fixed in 3.2.3) under heavy load so in order to determine where the problem was, we setup the jvm so we could connect with a remote debugger. We fixed the jms problem but left the server in debug mode and continued our performance tests. We would generally get to about 100 users before we maxed out the CPU.

      We then noticed that the debug info was still in the start script so we removed it. We could now only get to about 30 users before hitting 100% CPU. This was very repeatable, and finally we took everything off the command line and it turned out the "-debug" flag is all we needed to make the performance reach it's previous levels.

      Has anybody seen this before? Any ideas why having the jvm in debug mode would increase performance?

      system specs:

      os: SunOS xxx.xxx.xxx.xxx 5.8 Generic_108528-29 sun4u sparc SUNW,Sun-Fire-880

      jvm: Java HotSpot(TM) Client VM (build 1.4.1_01-b01, mixed mode)

      Pete.


        • 1. Re: 200% performance increase with remote debugging on.

          If you have a constant 100% CPU usage there is a serious problem. Consider that means each of the CPUs is running a thread that is not blocked all the time. When you run in debug mode it lowers your performance by introducing blocked IO calls (I am assuming it's not using NIO) but it decreases your CPU usage. Increased CPU usage can mean better overall performance through more efficient IO but not if the thing is pegged at 100%.

          Cupla questions. How many processors on your 880? How does the load testing tool determine the CPU usage ? since it is running on a client it must communicate with the server to get those stats and mayby what is being reported is suspect.

          • 2. Re: 200% performance increase with remote debugging on.
            pclear

            Hi Treespace.

            Thanks for the response.

            The CPU is not at a constant 100% usage - I only used this as a term indicating that the box itself was at it's full capacity. Not only did we watch th e CPU but all the other OS load paramerters also indicated full capacity. It is a 4 CPU box and the "Load" value reported via rstatd was also very high (16-20) when debugging was turned off and we hit 30 "users".

            Webload uses rstatd to determine cpu usage/load and we often manually monitor these (using top, mpstat, etc...) and what it reports is always consistent with what we see manually.

            Another indication that the server is "working better" when the debug flag is turned on is that we can pass more of the load onto the database server which indicates that we are getting more transactions per minute. With debugging off, we could only load the DB server (12 CPU sun box) to about 30% cpu and a load value of 1 - 2 no matter how many "users" we had hitting the jboss server - obviously not heavily loaded. When we switch to debug on, we're able to get the database box up to about 75% cpu usage, with a load value of 8-10 which is a much heavier load.

            You're comment about blocked IO is interesting. The application is a massive re-implementation of an old mainframe cobol system. When it was coded there was very little thought given to performance and if we run it through a profiler, it is obvious that it spends most of it's time doing IO. For example, a single request from a user can result in over 30 database reads and 10 database updates in about 3 different transactions as well as a JMS message. Maybe somehow the different IO causes the db IO to be more efficient?

            pete.

            • 4. Re: 200% performance increase with remote debugging on.

              With all of that IO going on even one CPU would be snoozing most of the time unless there are several threads polling something-or-other. Might be worth your while to pinpoint those bottlenecks and come back and revisit the wacky debug behavior later. If there was anyway to circumvent JMS and queue in the application that could also be very telling. Good luck with your efforts.