14 Replies Latest reply on Aug 19, 2013 9:53 AM by mattdarwin

    Crazy performance issue with JBossWS

    jbonham100

      Hi all

       

      I have a bottom-up JAX-WS web service deployed in JBoss. I was running some performance tests at different volumes and I ran into this issue: exactly one hour after the first request, the performance drops dramatically and garbage collection goes off the roof. I took some thread dumps and most HTTP threads are stuck trying to open a class for JBossWSEntityResolver. Our code is not even reached. Has anyone seen anything like it?

       

      Here's the thread dump for one of the threads, they look pretty much the same:

       

      "http-0.0.0.0-8080-149" daemon prio=10 tid=0x0000002b4245b000 nid=0x4e84 runnable [0x000000004958e000]
         java.lang.Thread.State: RUNNABLE
              at java.util.zip.ZipFile.freeEntry(Native Method)
              at java.util.zip.ZipFile.access$800(ZipFile.java:29)
              at java.util.zip.ZipFile$2.nextElement(ZipFile.java:330)
              - locked <0x0000002a99c1e280> (a java.util.zip.ZipFile)
              at java.util.zip.ZipFile$2.nextElement(ZipFile.java:299)
              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.initEntries(ZipEntryContext.java:491)
              - locked <0x0000002a99c1ba90> (a org.jboss.virtual.plugins.context.zip.ZipEntryContext)
              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.ensureEntries(ZipEntryContext.java:619)
              - locked <0x0000002a99c1ba90> (a org.jboss.virtual.plugins.context.zip.ZipEntryContext)
              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.checkIfModified(ZipEntryContext.java:773)
              - locked <0x0000002a99c1ba90> (a org.jboss.virtual.plugins.context.zip.ZipEntryContext)
              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.getChild(ZipEntryContext.java:817)
              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.createChildHandler(ZipEntryHandler.java:191)
              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.java:684)
              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
              at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
              at org.jboss.virtual.VirtualFile.findChild(VirtualFile.java:457)
              at org.jboss.virtual.plugins.vfs.VirtualFileURLConnection.resolveVirtualFile(VirtualFileURLConnection.java:106)
              at org.jboss.virtual.plugins.vfs.VirtualFileURLConnection.getVirtualFile(VirtualFileURLConnection.java:118)
              - locked <0x0000002a99c19a98> (a org.jboss.virtual.plugins.vfs.VirtualFileURLConnection)
              at org.jboss.virtual.plugins.vfs.VirtualFileURLConnection.getInputStream(VirtualFileURLConnection.java:93)
              at java.net.URL.openStream(URL.java:1010)
              at java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:1195)
              at org.jboss.ws.core.utils.JBossWSEntityResolver$1.run(JBossWSEntityResolver.java:80)
              at org.jboss.ws.core.utils.JBossWSEntityResolver$1.run(JBossWSEntityResolver.java:78)
              at java.security.AccessController.doPrivileged(Native Method)
              at org.jboss.ws.core.utils.JBossWSEntityResolver.loadEntitiesMappingFromClasspath(JBossWSEntityResolver.java:76)
              at org.jboss.ws.core.utils.JBossWSEntityResolver.<init>(JBossWSEntityResolver.java:59)
              at org.jboss.ws.core.utils.JBossWSEntityResolver.<init>(JBossWSEntityResolver.java:51)
              at sun.reflect.GeneratedConstructorAccessor180.newInstance(Unknown Source)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
              at java.lang.Class.newInstance0(Class.java:355)
              at java.lang.Class.newInstance(Class.java:308)
              at org.jboss.wsf.common.DOMUtils$1.setEntityResolver(DOMUtils.java:121)
              at org.jboss.wsf.common.DOMUtils$1.initialValue(DOMUtils.java:101)
              at org.jboss.wsf.common.DOMUtils$1.initialValue(DOMUtils.java:110)
              at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:141)
              at java.lang.ThreadLocal.get(ThreadLocal.java:131)
              at org.jboss.wsf.common.DOMUtils.getDocumentBuilder(DOMUtils.java:149)
              at org.jboss.wsf.common.DOMUtils.getOwnerDocument(DOMUtils.java:690)
              at org.jboss.ws.core.soap.SOAPDocument.<init>(SOAPDocument.java:54)
              at org.jboss.ws.core.soap.SOAPPartImpl.<init>(SOAPPartImpl.java:75)
              at org.jboss.ws.core.soap.SOAPMessageImpl.<init>(SOAPMessageImpl.java:86)
              at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:215)
              at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:193)
              at org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:455)
              at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:295)
              at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:205)
              at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:131)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
              at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
              at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
              at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
              at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
              at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
              at java.lang.Thread.run(Thread.java:619)

       

      Thanks in advance

      Andre