1 2 Previous Next 16 Replies Latest reply on Mar 16, 2012 4:36 PM by nickynarak

    5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%

    eminil

      We have been running our app on JBoss 4.0.RC2 using EJB.

       

      We have now migrated/upgraded to JBoss 5.1.0.GA and use EJB3 with annotations.

       

      The problem is our whole system seems to be running *considerably* slower in the new JBoss. We've added timestamps in our code to messaure certain methods and it seems almost everything is running from 2-3 times slower in the VM.

       

      Anyone have any idea what could be the cause of this? Is it just that 5.1.0.GA stinks? Or is it that we need to change some settings or similar?

       

      This slowness is not acceptable for our app.

       

      Any help is very much appreciated!

        • 1. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
          jaikiran

          There have been some performance fixes done around EJB3, after 5.1.0 was released. Apply the latest version of EJB3 plugin http://www.jboss.org/ejb3/ejb3plugin.html against AS 5.1.0 and see if it makes a difference.

          • 2. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
            eminil

            We tried running jboss-ejb3-plugin-1.0.19-installer.jar

             

            I did not change anything. JBoss 5.1.0.GA is still running around 200-300% slower than 4.0.3RC2.

            • 3. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
              jaikiran

              Can you explain in a bit more detail on the usecase you are trying, with some sample code?

              • 4. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                eminil

                To demonstrate this we simply added a test method at the top of one of our EJBs.

                 

                public void testJBossSpeed(org.apache.log4j.Logger log){
                       
                        String s = "";
                       
                        for(int i=0; i < 1000000; i++){
                            s += i;
                            if (s.length()> 1000){
                                s = "";
                            }
                        }
                    }

                 

                (We used System.currentTimeMillis() to fetch time before the method starts and when it stops to figure out the time it took to run).


                JBoss 4.0.3RC2 = approx 2500ms

                 

                JBoss 5.1.0.GA = approx 3500ms

                 

                This is a fairly simple test though. When lots of code gets involved the slowness of 5.1.0 increases a lot to somewhere around 200-300%.

                 

                This is weird... it seems the whole VM is running slower? Is it some configuration issue we have messed up or forgot to turn on/off?

                • 5. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                  jaikiran

                  Are you using "remote" interfaces for invoking on the bean? Although i don't think that would make a difference in this specific example. Also the code you posted has nothing specific to EJB within that method. Can you separate out the time it takes to run the code in that method and the time it takes to return back to the client? And then compare this with the two instances?

                  • 6. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                    eminil

                    Hmm, i don't think the total time is relevant. And it is probably not what is causing the slowness.

                     

                    It is code running INSIDE the methods of the EJB that somehow seem to be running slower. The whole JVM started with JBoss 5.1.0.GA seems lagged...

                     

                    The loop described above shouldn't take 1000ms longer to run. Something is very strange there. The EJB seems to be lagged down by something.

                     

                    Another thing we've noticed is that with 5.1.0 it seems that the server CPU goes up to 100% everytime we call the methods on the EJBs. In the older JBoss 4 this doesn't seem to happen as often...

                     

                     

                    Some more info on the JBoss 5.1.0.GA setup:

                     

                    How we fetch the bean on the client (a servlet in this case):

                    Context ctx = new InitialContext(jndiProperties);           
                    RapportEJBRemote ejb = (RapportEJBRemote) ctx.lookup("RapportEJB");
                    return ejb;

                     

                    The remote class is defined as this:

                    @WebService
                    @SOAPBinding(style = SOAPBinding.Style.RPC)
                    public interface RapportEJBRemote extends Remote

                     

                    And the bean class:

                    @Stateless
                    @WebService(endpointInterface = "se.stuff.RapportEJBRemote")
                    @Remote(RapportEJBRemote.class)
                    @WebContext
                    (
                      contextRoot="/stuff-jboss-app-server",
                      //urlPattern="/*",
                      //authMethod=AuthMethod.BASIC,
                      transportGuarantee="CONFIDENTIAL",
                      secureWSDLAccess=false
                    )
                    @RemoteBinding(clientBindUrl="sslsocket://0.0.0.0:3843", jndiBinding="RapportEJB")
                    public class RapportEJB implements RapportEJBRemote

                     


                    One difference is that in 5.1.0 we're using SSL and https for the beans. In 4.0.3 we did not. But that shouldn't affect the time for code to run in the VM right? Like the simple method i posted above. It should only affect the total time between client-server right? As far as i can understand this shouldnt be the cause of our problem with slow java code executing in the jboss.

                    • 7. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                      peterj

                      Beware of microbenchmarks, like the one you have written. There are way too many factors that can throw the results off, including which garbage collector you are using, the heap size, which JDK you are using, and the list goes on.

                       

                      The only valid way to determine performance difference between the two app servers is to make sure you configure them the same, use the same JDK with the same JVM options, deploy the exact same app to them, and then use a load testing tool such as JMeter and run a test that takes at least 15 minutes to run. Then look at the response times.

                      • 8. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                        eminil

                        peterj wrote:

                         

                        Beware of microbenchmarks, like the one you have written. There are way too many factors that can throw the results off, including which garbage collector you are using, the heap size, which JDK you are using, and the list goes on.

                         

                        The only valid way to determine performance difference between the two app servers is to make sure you configure them the same, use the same JDK with the same JVM options, deploy the exact same app to them, and then use a load testing tool such as JMeter and run a test that takes at least 15 minutes to run. Then look at the response times.

                        Well, we have the same JDK installed.

                        It is running in the exact same computer.

                        Deploying the exact same app. Except the beans are writting using EJB3 annotations in JBoss 5.1.0. But that shouldn't affect the speed of simple string handling methods right?

                         

                        The simple methods to build an xml string in the EJB is taking up to 200-300% longer time.

                         

                        We need a single call to the method to be quick. It is not an option to have it that slow.

                         

                        JBoss 5.1.0 is slowing down all java methods that is running inside it somehow, and we need to figure out how to make that stop.

                        • 9. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                          peterj

                          What timings do you get if you move the currentTimeMillis to inside the testJBossSpeed method (I'm trying to eliminate any possible AOP processing)?

                           

                          The testJBossSpeedis creating 1 million String objects, all of which end up being garbage. The only things I can think of regarding JBoss AS doing anything here are: a) some AOP config is wrapped around the String class, b) something is hooked into garbage collection.

                           

                          Oh, there is one more possibility - heap usage. If 5.1.0 uses significantly more heap than 4.0.3, and both app servers are set to that same heap size, then 5.1.0 will encounter more garbage collections. You can run with -verbose:gc to determine if this is the case.

                          • 10. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                            eminil

                            peterj wrote:

                             

                            What timings do you get if you move the currentTimeMillis to inside the testJBossSpeed method (I'm trying to eliminate any possible AOP processing)?

                             

                            The testJBossSpeedis creating 1 million String objects, all of which end up being garbage. The only things I can think of regarding JBoss AS doing anything here are: a) some AOP config is wrapped around the String class, b) something is hooked into garbage collection.

                             

                            Oh, there is one more possibility - heap usage. If 5.1.0 uses significantly more heap than 4.0.3, and both app servers are set to that same heap size, then 5.1.0 will encounter more garbage collections. You can run with -verbose:gc to determine if this is the case.

                            The same timings apply even if we put everything inside the method.

                             

                            Will try the verbose option and see what happens...

                            • 11. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                              eminil

                              I tried running with -verbose:gc

                               

                              During the method it calls gc a few times, but the time gc spends with it is like 0.008 seconds per call. And it does it 5 times approx.

                               

                              Our whole EJB method spends 14 seconds in 5.1.0 GA and around 5 seconds in 4.0.3 so i don't think GC is a huge factor of it...

                               

                              (We even tried reinstalling 5.1.0 fresh to test with, incase we had broken some config files. But it gives the same slowness.)

                              • 12. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                                eminil

                                We also tried turning off all SSL and https. SO the bean looks like:

                                 

                                @Stateless
                                @WebService(endpointInterface = "se.stuff.RapportEJBRemote")
                                @Remote(RapportEJBRemote.class)
                                @WebContext
                                (
                                  contextRoot="/stuff-jboss-app-server",
                                  secureWSDLAccess=false
                                )
                                @RemoteBinding(jndiBinding="RapportEJB")
                                public class RapportEJB implements RapportEJBRemote

                                 

                                It still gives the exact same response times.

                                • 13. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                                  peterj
                                  The only thing I can think of is that something is messing with String allocation. I'm not sure how to debug this to find the root cause; let me think on it.
                                  • 14. Re: 5.1.0.GA considerably slower than 4.0.3RC2! Up to 200%
                                    eminil

                                    We used JProfiler with both instances of JBoss and it seems our 5.1.0 was spending a lot of time in the mssql driver classes. So we went into our mssql-ds.xml and changed the datasource from <no-tx-datasource> to <local-tx-datasource>. This fixed all the slowness we had with 5.1.0.

                                     

                                    My guess is that somehow this configuration lagged down JBoss with background processes or whatever so that simple method calls like the one i posted above also delayed because the CPU was busy.

                                     

                                    Hopefully we can use the local-tx-datasource without problems (I'm not entirely sure why we used a no-tx in 4.0.3, but think it had to do with database connections never being released with the local setting).

                                    1 2 Previous Next