5 Replies Latest reply: Oct 17, 2010 4:00 AM by Anuj Khanna RSS

cpu 100% after particulary request occurs

Alex Sun Newbie

i can not reproduce this case, but i have found more than 5 times at least.
when this case occurs, i analysis the javacore files, and monitor the top command's thread result on my Linux box, then i found below thread is eating cpu.


And my enviroment :


WAS6.1, Seam 2.0.2.SP1, JBoss PortletBridge 1.0.0.B3, Richfaces 3.2.1.GA




the thread details:


3XMTHREADINFO      "WebContainer : 12" (TID:0x0A591500, sys_thread_t:0x0A198E14, state:CW, native ID:0x00002AFF) prio=5
4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.fullGetFirstQueuedThread(AbstractQueuedSynchronizer.java:1285)
4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.getFirstQueuedThread(AbstractQueuedSynchronizer.java:1248)
4XESTACKTRACE          at java/util/concurrent/locks/ReentrantLock$FairSync.tryAcquire(ReentrantLock.java:223)
4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:777)
4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1102)
4XESTACKTRACE          at java/util/concurrent/locks/ReentrantLock.tryLock(ReentrantLock.java:431)
4XESTACKTRACE          at org/jboss/seam/core/ConversationEntry.lock(ConversationEntry.java:204)
4XESTACKTRACE          at org/jboss/seam/core/Manager.restoreAndLockConversation(Manager.java:490)
4XESTACKTRACE          at org/jboss/seam/core/Manager.restoreConversation(Manager.java:485)
4XESTACKTRACE          at org/jboss/seam/jsf/SeamPhaseListener.afterRestoreView(SeamPhaseListener.java:388)
4XESTACKTRACE          at org/jboss/seam/jsf/SeamPhaseListener.afterServletPhase(SeamPhaseListener.java:226(Compiled Code))
4XESTACKTRACE          at org/jboss/seam/jsf/SeamPhaseListener.afterPhase(SeamPhaseListener.java:192(Compiled Code))
4XESTACKTRACE          at org/jboss/seam/jsf/CmstarSeamPhaseListenerWrapper.afterPhase(CmstarSeamPhaseListenerWrapper.java:84(Compiled Code))
4XESTACKTRACE          at org/jboss/portletbridge/lifecycle/LifecyclePhase.execute(LifecyclePhase.java:85(Compiled Code))
4XESTACKTRACE          at org/jboss/portletbridge/lifecycle/RestoreViewPhase.execute(RestoreViewPhase.java:62)
4XESTACKTRACE          at org/jboss/portletbridge/lifecycle/PortletLifecycle.execute(PortletLifecycle.java:114)
4XESTACKTRACE          at javax/faces/webapp/FacesServlet.service(FacesServlet.java:244)
4XESTACKTRACE          at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1068(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1009(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:145(Compiled Code))
4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
4XESTACKTRACE          at com/wiscom/epstar/web/TimingFilter.doFilter(TimingFilter.java:34)
4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
4XESTACKTRACE          at org/jboss/seam/web/RedirectFilter.doFilter(RedirectFilter.java:45)
4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
4XESTACKTRACE          at org/jboss/seam/web/ExceptionFilter.doFilter(ExceptionFilter.java:64)
4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
4XESTACKTRACE          at org/jboss/seam/web/LoggingFilter.doFilter(LoggingFilter.java:58)
4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter.doFilter(SeamFilter.java:158)
4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:130(Compiled Code))
4XESTACKTRACE          at org/ajax4jsf/webapp/BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:154)
4XESTACKTRACE          at org/ajax4jsf/webapp/BaseFilter.handleRequest(BaseFilter.java:260)
4XESTACKTRACE          at org/ajax4jsf/webapp/BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:366)
4XESTACKTRACE          at org/ajax4jsf/webapp/BaseFilter.doFilter(BaseFilter.java:493)
4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:130(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterChain._doFilter(WebAppFilterChain.java:87(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:771(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:679(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:278(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/wswebcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:90(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:744(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/wswebcontainer/WebContainer.handleRequest(WebContainer.java:1455(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:115(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewInformation(HttpInboundLink.java:383(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:102(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165(Compiled Code))
4XESTACKTRACE          at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
4XESTACKTRACE          at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:143(Compiled Code))
4XESTACKTRACE          at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:136(Compiled Code))
4XESTACKTRACE          at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:195(Compiled Code))
4XESTACKTRACE          at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:743(Compiled Code))
4XESTACKTRACE          at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:873)
4XESTACKTRACE          at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1473(Compiled Code))



  • 1. Re: cpu 100% after particulary request occurs
    Alex Sun Newbie

    javacore file header info:



    NULL           ------------------------------------------------------------------------
    0SECTION       TITLE subcomponent dump routine
    NULL           ===============================
    1TISIGINFO     Dump Event "user" (00004000) received 
    1TIDATETIME    Date:                 2009/03/25 at 09:23:44
    1TIFILENAME    Javacore filename:    /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20090325.092344.10374.txt
    NULL           ------------------------------------------------------------------------
    0SECTION       GPINFO subcomponent dump routine
    NULL           ================================
    2XHOSLEVEL     OS Level         : Linux 2.6.9-55.ELsmp
    2XHCPUS        Processors -
    3XHCPUARCH       Architecture   : x86
    3XHNUMCPUS       How Many       : 2
    NULL           
    1XHERROR2      Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.
    NULL           
    NULL           ------------------------------------------------------------------------
    0SECTION       ENVINFO subcomponent dump routine
    NULL           =================================
    1CIJAVAVERSION J2RE 5.0 IBM J9 2.3 Linux x86-32 build j9vmxi3223-20060504
    1CIVMVERSION   VM build 20060501_06428_lHdSMR
    1CIJITVERSION  JIT enabled - 20060428_1800_r8
    1CIRUNNINGAS   Running as a standalone JVM
    1CICMDLINE     /opt/IBM/WebSphere/AppServer/java/bin/java -Declipse.security -Dwas.status.socket=28387 -Dosgi.install.area=/opt/IBM/WebSphere/AppServer -Dosgi.configuration.area=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/configuration -Dosgi.framework.extensions=com.ibm.cds -Xshareclasses:name=webspherev61_%g,groupAccess,nonFatal -Xscmx50M -Xbootclasspath/p:/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -classpath /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/j2ee.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -Xms50m -Xmx1024m -Dws.ext.dirs=/opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/opt/IBM/WebSphere/AppServer/derby -Dcom.ibm.itp.location=/opt/IBM/WebSphere/AppServer/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=/opt/IBM/WebSphere/AppServer -Dpython.cachedir=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01 -Djava.awt.headless=true;-Dibm.stream.nio=true -Djava.security.auth.login.config=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/wsjaas.conf -Djava.security.policy=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/server.policy com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config idstarCell02 idstarNode03 server1
    1CIJAVAHOMEDIR Java Home Dir:   /opt/IBM/WebSphere/AppServer/java/jre
    1CIJAVADLLDIR  Java DLL Dir:    /opt/IBM/WebSphere/AppServer/java/jre/bin
    1CISYSCP       Sys Classpath:   /opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/vm.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/core.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/charsets.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/graphics.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/security.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmpkcs.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmorb.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmcfw.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmorbapi.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmjcefw.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmjgssprovider.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmjsseprovider2.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmjaaslm.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmcertpathprovider.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/server.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/xml.jar;
    1CIUSERARGS    UserArgs:
    2CIUSERARG               -Xjcl:jclscar_23
    2CIUSERARG               -Dcom.ibm.oti.vm.bootstrap.library.path=/opt/IBM/WebSphere/AppServer/java/jre/bin
    2CIUSERARG               -Dsun.boot.library.path=/opt/IBM/WebSphere/AppServer/java/jre/bin
    2CIUSERARG               -Djava.library.path=/opt/IBM/WebSphere/AppServer/java/jre/bin:/opt/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/opt/IBM/WebSphere/AppServer/java/jre/bin:/opt/IBM/WebSphere/AppServer/bin:/usr/lib
    2CIUSERARG               -Djava.home=/opt/IBM/WebSphere/AppServer/java/jre
    2CIUSERARG               -Djava.ext.dirs=/opt/IBM/WebSphere/AppServer/java/jre/lib/ext
    2CIUSERARG               -Duser.dir=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01
    2CIUSERARG               _j2se_j9
    2CIUSERARG               vfprintf 0x0804B664
    2CIUSERARG               -Declipse.security
    2CIUSERARG               -Dwas.status.socket=28387
    2CIUSERARG               -Dosgi.install.area=/opt/IBM/WebSphere/AppServer
    2CIUSERARG               -Dosgi.configuration.area=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/configuration
    2CIUSERARG               -Dosgi.framework.extensions=com.ibm.cds
    2CIUSERARG               -Xshareclasses:name=webspherev61_%g,groupAccess,nonFatal
    2CIUSERARG               -Xscmx50M
    2CIUSERARG               -Xbootclasspath/p:/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar
    2CIUSERARG               -Dibm.websphere.internalClassAccessMode=allow
    2CIUSERARG               -Xms50m
    2CIUSERARG               -Xmx1024m
    2CIUSERARG               -Dws.ext.dirs=/opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
    2CIUSERARG               -Dderby.system.home=/opt/IBM/WebSphere/AppServer/derby
    2CIUSERARG               -Dcom.ibm.itp.location=/opt/IBM/WebSphere/AppServer/bin
    2CIUSERARG               -Djava.util.logging.configureByServer=true
    2CIUSERARG               -Duser.install.root=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01
    2CIUSERARG               -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder
    2CIUSERARG               -Dwas.install.root=/opt/IBM/WebSphere/AppServer
    2CIUSERARG               -Dpython.cachedir=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/temp/cachedir
    2CIUSERARG               -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager
    2CIUSERARG               -Dserver.root=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01
    2CIUSERARG               -Djava.awt.headless=true;-Dibm.stream.nio=true
    2CIUSERARG               -Djava.security.auth.login.config=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/wsjaas.conf
    2CIUSERARG               -Djava.security.policy=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/server.policy
    2CIUSERARG               -Dinvokedviajava
    2CIUSERARG               -Djava.class.path=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/j2ee.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
    2CIUSERARG               vfprintf
    2CIUSERARG               _port_library 0xF7FFCCA0
    2CIUSERARG               -Xdump



  • 2. Re: cpu 100% after particulary request occurs
    Alex Sun Newbie

    and,



    <core:manager concurrent-request-timeout="1000"
              conversation-timeout="600000" conversation-id-parameter="cid" />



  • 3. Re: cpu 100% after particulary request occurs
    Stuart Douglas Master

    This looks like an IBM jre bug that I found mentioned here. Maybe upgrade or use a different JVM.

  • 4. Re: cpu 100% after particulary request occurs
    Alex Sun Newbie

    It's a IBM JDK 1.5 bug.

  • 5. Re: cpu 100% after particulary request occurs
    Anuj Khanna Newbie

    Hi,


    We are also facing this issue and need to check if this is a JDK bug, which patch was downloaded and installed to fix this. From where can i download the patch.


    Also wanted to check if the scenario to reproduce has any similarity to yours


    If i load and page and while it is loading if i perform another action for example clicking a button or another seam link then CPU usage shoots up.


    Any help will be highly appreciated.