1 2 3 Previous Next 30 Replies Latest reply on Jul 2, 2014 12:06 AM by ramakoteswar

    Performance slow down on JBoss 7.1.1

    jeev

      Hi,

       

      We have a J2EE application managed by spring deployed on JBoss 7.1.1.  We are experiencing slowing down of JBoss gradually and finally we've to restart it once/twice in a week.

      And its deployed on standalone configuration on JBoss.

      Our application uses the following technologies

      Spring3.0, Richfaces 4.1.0 Final, JBoss 7.1.1.Final, Hibernate 3.6.

      We've analysed the memory, threads, http sessions, cpu load using java melody monitoring tool. We've checked on the DB side, our code and almost everything. But still this slowdown happens.

      It seems like during heavy load, it becomes slower, but even after the load is back to normal, the response time is still slow , until we restart the server.

       

      We've even put some logs from our side to track  the time taken by each and every layer like service, DAO etc. We've also put a servlet filter to track the total response time taken by each request.

      When the performance is slow, every log is normal, except that the time taken by a request-response cycle is very huge in our filter.

      For eg. Even if the time taken on the server side for a particular transaction is 2 sec(before JSF decoding/encoding), in our servlet filter its almost near to a minute. So is there any problem during the JSF decoding or rendering process??We never got out of memory errors or anything of that sort.

       

      Really we dont have a clue where is the problem???

      So can anyone throw us some light on this issue. Do we need to fine tune any of the JBoss parameters/settings in standalone.xml ? Because restarting the server every week is not a solution and the customer has already started fuming because of this...

       

      Thanks in advance

      Jeevendran

        • 1. Re: Performance slow down on JBoss 7.1.1
          nickarls

          I recall having a similar issue that was due to some regression in mojarra. Tried downgrading to e.g. 2.1.4?

          • 2. Re: Performance slow down on JBoss 7.1.1
            jeev

            Hi Nicklas,

            Actually we are using Mojarra 2.0.3 version only..

            Could you tell me which Mojarra version can I use which is quite stable without those issues??

             

            Thanks

            • 3. Re: Performance slow down on JBoss 7.1.1
              nickarls

              In our cases 2.1.4 has been working out. I don't think there is any "quite stable" version. As with any software it's "If it works, it ain't broken".

              • 4. Re: Performance slow down on JBoss 7.1.1
                reena_jacob

                Hi Jeeven,

                 

                Because of the slow performance of RichFaces 4.1.0 ,we switched over to PrimeFaces 3.4  using JSF 2.1.4.

                 

                Thanks

                • 5. Re: Performance slow down on JBoss 7.1.1
                  nickarls

                  I'm not sure if the change that resulted in the slowdown was a regression that has been fixed in a later version or if it's a "permanent feature"...

                  • 6. Re: Performance slow down on JBoss 7.1.1
                    lukascz

                    Hi Jeevendran,

                     

                    I'm experiencing exactly the same behavior. I spent a lot of time on tracking down the issue but without success. The server must be restarted every 3 days, otherwise the responsiveness goes down significantly. I was trying to find the bottleneck but it seems that there is plenty of resources: the problem is definitely not CPU, IO blocking, insufficient number of http threads. For each request I'm measuring time time JSF spends in render_response phase and it's increasing proportionally on all pages. So the bottleneck is not hibernate either. For example, after fresh restart, it takes 400 ms to render the page that requires some DB access and some other is rendered in 50 ms and doesn't need any DB access. After few days, it takes 4000 ms to render first page and 500 ms to render the other page. I tried using Mojarra 2.1.7 and 2.1.11 but the behavior is the same. I didn't try downgrading to 2.1.4

                     

                    I'm using both, Primefaces 3.4 and RichFaces 4.2. But primarily Primefaces.

                     

                    Please let us know if you have some news.

                     

                    Thanks,

                    Lukas

                    • 7. Re: Performance slow down on JBoss 7.1.1
                      nickarls
                      • 8. Re: Performance slow down on JBoss 7.1.1
                        ctomc

                        Hi,

                         

                        7.2 has many things in this area fixed. but original problem in mojarra still exists.

                         

                        with 7.2 you can easier replace jsf impl so you could use myfaces instead of mojarra.

                         

                        see https://community.jboss.org/wiki/DesignOfAS7Multi-JSFFeature & AS7-5826 for more info

                         

                         

                        --

                        tomaz

                        • 9. Re: Performance slow down on JBoss 7.1.1
                          lukascz

                          Unfortunately none of those issues are talking about peformance degradation that Jeevendran and I are experiencing. So I'm not sure if it's really Mojarra issue as it seems it wasn't reported. It's also very hard to reproduce, you need to wait some time. And sometimes it gets slow after 5 days, sometimes after few hours.

                          • 10. Re: Performance slow down on JBoss 7.1.1
                            jaikiran

                            When it does get slow, take regular thread dumps from the server http://community.jboss.org/wiki/ThreadDump and see what the server side is upto.

                            • 11. Re: Performance slow down on JBoss 7.1.1
                              lukascz

                              Actually I did that. Here are some threads that were being executed long time and were taking a lot of CPU (aroung 90%)

                               

                              java.lang.Thread.State: RUNNABLE
                                  at java
                              .lang.ThreadLocal.get(ThreadLocal.java:143)
                                  at org
                              .jboss.weld.context.cache.RequestScopedBeanCache.isActive(RequestScopedBeanCache.java:37)
                                  at org
                              .jboss.seam.faces.environment.SeamExternalContext$Proxy$_$$_WeldClientProxy.getInitParameter(SeamExternalContext$Proxy$_$$_WeldClientProxy.java)
                                  at javax
                              .faces.component.UIComponent.popComponentFromEL(UIComponent.java:1944)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2218)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.doPostAddProcessing(UIComponentBase.java:1883)
                                  at javax
                              .faces.component.UIComponentBase.setParent(UIComponentBase.java:400)
                                  at javax
                              .faces.component.UIComponentBase$ChildrenList.add(UIComponentBase.java:2635)
                                  at javax
                              .faces.component.UIComponentBase$ChildrenList.add(UIComponentBase.java:2607)
                                  at com
                              .sun.faces.facelets.tag.jsf.ComponentSupport.addComponent(ComponentSupport.java:574)
                                  at com
                              .sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.addComponentToView(ComponentTagHandlerDelegateImpl.java:311)
                                  at com
                              .sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:211)
                                  at javax
                              .faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:120)
                                  at com
                              .sun.faces.facelets.compiler.NamespaceHandler.apply(NamespaceHandler.java:93)
                                  at com
                              .sun.faces.facelets.compiler.EncodingHandler.apply(EncodingHandler.java:86)
                                  at com
                              .sun.faces.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:308)
                                  at com
                              .sun.faces.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:367)
                                  at com
                              .sun.faces.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:346)
                                  at com
                              .sun.faces.facelets.impl.DefaultFaceletContext.includeFacelet(DefaultFaceletContext.java:199)
                                  at com
                              .sun.faces.facelets.tag.ui.DecorateHandler.apply(DecorateHandler.java:145)
                                  at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                                  at javax
                              .faces.view.facelets.DelegatingMetaTagHandler.applyNextHandler(DelegatingMetaTagHandler.java:137)
                                  at org
                              .richfaces.view.facelets.html.BehaviorsAddingComponentHandlerWrapper.applyNextHandler(BehaviorsAddingComponentHandlerWrapper.java:53)
                                  at com
                              .sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:195)
                                  at javax
                              .faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:120)


                              java
                              .lang.Thread.State: RUNNABLE
                                  at java
                              .lang.Thread.currentThread(Native Method)
                                  at java
                              .lang.ThreadLocal.get(ThreadLocal.java:143)
                                  at org
                              .jboss.weld.bean.proxy.InterceptionDecorationContext.pop(InterceptionDecorationContext.java:43)
                                  at org
                              .jboss.weld.bean.proxy.InterceptionDecorationContext.endInterceptorContext(InterceptionDecorationContext.java:93)
                                  at org
                              .jboss.seam.faces.environment.SeamExternalContext$Proxy$_$$_WeldClientProxy.getInitParameter(SeamExternalContext$Proxy$_$$_WeldClientProxy.java)
                                  at javax
                              .faces.component.UIComponent.pushComponentToEL(UIComponent.java:1860)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2200)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                                  at javax
                              .faces.component.UIComponentBase.doPostAddProcessing(UIComponentBase.java:1883)
                                  at javax
                              .faces.component.UIComponentBase.setParent(UIComponentBase.java:400)
                                  at javax
                              .faces.component.UIComponentBase$ChildrenList.add(UIComponentBase.java:2635)
                                  at javax
                              .faces.component.UIComponentBase$ChildrenList.add(UIComponentBase.java:2607)
                                  at com
                              .sun.faces.facelets.tag.jsf.ComponentSupport.addComponent(ComponentSupport.java:574)
                                  at com
                              .sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.addComponentToView(ComponentTagHandlerDelegateImpl.java:311)
                                  at com
                              .sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:211)
                                  at javax
                              .faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:120)
                                  at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                                  at javax
                              .faces.view.facelets.DelegatingMetaTagHandler.applyNextHandler(DelegatingMetaTagHandler.java:137)
                                  at org
                              .richfaces.view.facelets.html.BehaviorsAddingComponentHandlerWrapper.applyNextHandler(BehaviorsAddingComponentHandlerWrapper.java:53)
                                  at com
                              .sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:195)
                                  at javax
                              .faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:120)
                                  at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                                  at com
                              .sun.faces.facelets.tag.ui.DefineHandler.applyDefinition(DefineHandler.java:107)
                                  at com
                              .sun.faces.facelets.tag.ui.CompositionHandler.apply(CompositionHandler.java:178)
                                  at com
                              .sun.faces.facelets.impl.DefaultFaceletContext$TemplateManager.apply(DefaultFaceletContext.java:395)
                                  at com
                              .sun.faces.facelets.impl.DefaultFaceletContext.includeDefinition(DefaultFaceletContext.java:366)
                                  at com
                              .sun.faces.facelets.tag.ui.InsertHandler.apply(InsertHandler.java:112)
                                  at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                                  at javax
                              .faces.view.facelets.DelegatingMetaTagHandler.applyNextHandler(DelegatingMetaTagHandler.java:137)
                                  at org
                              .richfaces.view.facelets.html.BehaviorsAddingComponentHandlerWrapper.applyNextHandler(BehaviorsAddingComponentHandlerWrapper.java:53)
                                  at com
                              .sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:195)
                                  at javax
                              .faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:120)
                                  at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                                  at com
                              .sun.faces.facelets.tag.jsf.core.ViewHandler.apply(ViewHandler.java:182)
                                  at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                                  at com
                              .sun.faces.facelets.compiler.NamespaceHandler.apply(NamespaceHandler.java:93)
                                  at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                                  at com
                              .sun.faces.facelets.compiler.EncodingHandler.apply(EncodingHandler.java:86)

                                 java
                              .lang.Thread.State: RUNNABLE
                              at java
                              .lang.ThreadLocal$ThreadLocalMap$Entry.<init>(ThreadLocal.java:276)
                              at java
                              .lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:443)
                              at java
                              .lang.ThreadLocal$ThreadLocalMap.access$100(ThreadLocal.java:261)
                              at java
                              .lang.ThreadLocal.setInitialValue(ThreadLocal.java:164)
                              at java
                              .lang.ThreadLocal.get(ThreadLocal.java:150)
                              at org
                              .jboss.weld.bean.proxy.InterceptionDecorationContext.push(InterceptionDecorationContext.java:58)
                              at org
                              .jboss.weld.bean.proxy.InterceptionDecorationContext.startInterceptorContext(InterceptionDecorationContext.java:97)
                              at org
                              .jboss.seam.faces.environment.SeamExternalContext$Proxy$_$$_WeldClientProxy.getInitParameter(SeamExternalContext$Proxy$_$$_WeldClientProxy.java)
                              at javax
                              .faces.component.UIComponent.pushComponentToEL(UIComponent.java:1860)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2200)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.publishAfterViewEvents(UIComponentBase.java:2206)
                              at javax
                              .faces.component.UIComponentBase.doPostAddProcessing(UIComponentBase.java:1883)
                              at javax
                              .faces.component.UIComponentBase.setParent(UIComponentBase.java:400)
                              at javax
                              .faces.component.UIComponentBase$ChildrenList.add(UIComponentBase.java:2635)
                              at javax
                              .faces.component.UIComponentBase$ChildrenList.add(UIComponentBase.java:2607)
                              at com
                              .sun.faces.facelets.tag.jsf.ComponentSupport.addComponent(ComponentSupport.java:574)
                              at com
                              .sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.addComponentToView(ComponentTagHandlerDelegateImpl.java:311)
                              at com
                              .sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:211)
                              at javax
                              .faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:120)
                              at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                              at com
                              .sun.faces.facelets.tag.jsf.core.ViewHandler.apply(ViewHandler.java:182)
                              at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                              at com
                              .sun.faces.facelets.compiler.NamespaceHandler.apply(NamespaceHandler.java:93)
                              at javax
                              .faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:98)
                              at com
                              .sun.faces.facelets.compiler.EncodingHandler.apply(EncodingHandler.java:86)
                              at com
                              .sun.faces.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:308)
                              at com
                              .sun.faces.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:367)

                              • 12. Re: Performance slow down on JBoss 7.1.1
                                jaikiran

                                Lukas Camra wrote:

                                 

                                Actually I did that. Here are some threads that were being executed long time and were taking a lot of CPU (aroung 90%)

                                 

                                That actually contradicts with your earlier post:

                                 

                                Lukas Camra wrote:

                                 

                                the problem is definitely not CPU

                                 

                                But anyway, thanks for that stacktrace. I or someone else might take a look at it and see if we can find some hints. In the meantime, if this happens again, please do keep taking the thread dumps and attach them so that we can compare those and try and find a pattern.

                                • 13. Re: Performance slow down on JBoss 7.1.1
                                  jaikiran

                                  By the way, I don't know if it's of much help but I see a similar bug report here http://java.net/jira/browse/GLASSFISH-18826

                                  • 14. Re: Performance slow down on JBoss 7.1.1
                                    lukascz

                                    Well, I have machine with 8 cores and JBoss never takes more than 320% when the performance gets slow. Moreover, when the system is slow (even with no load), even one request takes a lot of CPU. CPU is not bottleneck as it doesn't make sense it's taking so much of CPU.

                                    1 2 3 Previous Next