1 Reply Latest reply on Aug 7, 2013 3:21 AM by stylpe

    ClassLoader bottleneck in 5.1.0.GA

    stylpe

      So it looks like our app is affected by JBCL-185 and/or JBCL-100. We were running visualvm during load testing, and discovered that EJB proxy instances had a very high self-time. This is a SLSB with a @Local interface, so it should be pretty fast.

       

      We did a thread dump, and it turns out most of the servlet threads were blocked waiting for a class loader. Some examples:

       

      Several are like this:

      Thread: ajp-0.0.0.0-8009-11 : priority:5, demon:true, threadId:198, threadState:BLOCKED

      - waiting on <0x22d22fe3> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:291)

      java.lang.ClassLoader.loadClass(ClassLoader.java:295)

      - locked <0x2c25bf62> (a org.jboss.ejb3.common.classloader.PrimitiveAwareClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:247)

      org.jboss.ejb3.common.lang.SerializableMethod.getClassFromName(SerializableMethod.java:307)

      org.jboss.ejb3.common.lang.SerializableMethod.getClassType(SerializableMethod.java:282)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:233)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:220)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.handleInvocationDirectly(SessionProxyInvocationHandlerBase.java:240)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:182)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)

      $Proxy815.getAdaptedItem(Unknown Source)

      [servlet method here]

       

      The thread owning 0x22d22fe3:

       

      Thread: ajp-0.0.0.0-8009-18 : priority:5, demon:true, threadId:205, threadState:BLOCKED

      - waiting on <0x1b4ddacc> (a org.jboss.classloader.spi.base.BaseClassLoader)

      java.lang.Class.forName0(Native Method)

      java.lang.Class.forName(Class.java:247)

      org.jboss.classloader.plugins.loader.ClassLoaderToLoaderAdapter.loadClass(ClassLoaderToLoaderAdapter.java:172)

      org.jboss.classloader.spi.ClassLoaderDomain.loadClassFromParent(ClassLoaderDomain.java:352)

      org.jboss.classloader.spi.ClassLoaderDomain.loadClassAfter(ClassLoaderDomain.java:323)

      org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:275)

      org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)

      org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)

      org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)

      - locked <0x357d9553> (a org.jboss.classloader.spi.base.BaseClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:295)

      - locked <0x22d22fe3> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:295)

      - locked <0x7866dc25> (a org.jboss.ejb3.common.classloader.PrimitiveAwareClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:247)

      org.jboss.ejb3.common.lang.SerializableMethod.getClassFromName(SerializableMethod.java:307)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:239)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:220)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:182)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)

      $Proxy815.getAdaptedItem(Unknown Source)

      [servlet method here]

       

      The thread owning 0x1b4ddacc:

       

      Thread: ajp-0.0.0.0-8009-5 : priority:5, demon:true, threadId:192, threadState:RUNNABLE

      java.util.Arrays.copyOfRange(Arrays.java:3209)

      java.lang.String.(String.java:215)

      java.lang.StringBuilder.toString(StringBuilder.java:430)

      org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:448)

      - locked <0x1b4ddacc> (a org.jboss.classloader.spi.base.BaseClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:295)

      - locked <0x24c3e995> (a org.jboss.ejb3.common.classloader.PrimitiveAwareClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:247)

      org.jboss.ejb3.common.lang.SerializableMethod.getClassFromName(SerializableMethod.java:307)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:239)

      org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:135)

      org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:216)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:207)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)

      $Proxy815.getAdaptedItem(Unknown Source)

      [servlet method here]

       

       

      According to what I've read on the issue pages and in discussions here, the root cause is the classes aren't properly cached, and thus have to be loaded on every single session bean invocation.

       

      Is there a way to work around this issue? Upgrading JBoss is not an option, but if upgrading only the class loader component is possible, we could do that.

        • 1. Re: ClassLoader bottleneck in 5.1.0.GA
          stylpe

          Oh, two more things: This call is going between a war and an ejb-jar, both in the same ear, so it might be due to misconfiguration somewhere. And if you're sure an upgrade would fix this, it would be gret to know, even though we are not able to do this. According to JBCL-185, though, as far as I can tell that fix isn't in any released version yet.