7 Replies Latest reply on Oct 11, 2011 11:50 AM by jimyliu

    performance logging in tomcat or mod_jk

    jimyliu

      Hi,

       

      We recently had JVM crash problems on tomcat server. But there is no hs_error or core dump file.  We tried to identify what URL request caused the JVM crash. But in the tomcat or mod_jk.log, it only shows the request entry when the request is finished. Is there any way I can see the request entry before when the request starts either in apache, mod_jk or tomcat local_access log ?

       

      Thanks,

       

      Jimy

        • 1. Re: performance logging in tomcat or mod_jk
          vcorrea

          Hi Jimy,

           

          I would sugest that you analyse it with an jvm introspect tool. Im using byteman ( jboss.org/byteman ). You can write simple script to analyse when a class is istantiated. So if you have suspicious classes you can check its status on runtime. Hope that it can help

          • 2. Re: performance logging in tomcat or mod_jk
            jimyliu

            Hi Victor,   Thanks for the suggestion. I will test with byteman to see if that can help my situation.

            • 3. Re: performance logging in tomcat or mod_jk
              rhusar

              Jimy, you can configure your mod_jk to log more and you will see what was happening with the request (the logging will be however too verbose so not a good fit for production for sure).

               

              If there are chances you can reproduce than surely +1 for Byteman if you want to instrument a class/method to do additional logging.

               

              BTW "But there is no hs_error" - did the JVM user have permission to write it there? Make sure you are not calling System.exit() ;-)

              • 4. Re: performance logging in tomcat or mod_jk
                vcorrea

                Did you activated this option on your vm args?

                 

                XX:+HeapDumpOnOutOfMemoryError

                ?

                • 5. Re: performance logging in tomcat or mod_jk
                  jimyliu

                  Yes, configure the mod_jk in debug mode is too heavy and not possible for the live system.   We did try to examine our code to make sure there is no System.exit().  

                   

                  Since there is frequently GC activities before it crash and the heap size looked ok, we did not set this option XX:+HeapDumpOnOutOfMemoryError , either.  But I will set it to see if that helps.

                   

                  Thanks ,

                   

                  Jimy

                  • 6. Re: performance logging in tomcat or mod_jk
                    rhusar

                    So this issue happens relatively often in production env but you cant reproduce it? Ah, these are always hard to find in live system. Most likely this looks like memory issue, so make sure your system and your JVM are not running out of memory.

                    • 7. Re: performance logging in tomcat or mod_jk
                      jimyliu

                      Sorry, I type incorrectly, there is NO frequently GC activities before it crash.  And yes, not reproducible either. :-(

                      Will definitely try the byteman if I can get the breaking point.