13 Replies Latest reply on Jan 4, 2013 9:54 AM by deepaksrivastavaz

    Clean shutdown/redeploy in Tomcat

    lovelyliatroim

      Hi Guys,
      I am trying to to shutdown cleanly and redeploy cleanly in tomcat but have failed to date. Reason I think has to do with the use of commons logging in both jgroups and jboss cache.

      Here is my stack trace when shutdown.sh is called.

      INFO: Stopping Coyote HTTP/1.1 on http-8080
      17.07.2008 10:50:14 org.apache.catalina.loader.WebappClassLoader loadClass
      INFO: Illegal access: this web application instance has been stopped already. Could not load java.io.PrintWriter. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
      java.lang.IllegalStateException
       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
       at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:154)
       at org.apache.log4j.Category.forcedLog(Category.java:388)
       at org.apache.log4j.Category.log(Category.java:853)
       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
       at java.lang.Thread.run(Thread.java:619)
      17.07.2008 10:50:14 org.apache.catalina.loader.WebappClassLoader loadClass
      INFO: Illegal access: this web application instance has been stopped already. Could not load org.apache.log4j.spi.VectorWriter. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
      java.lang.IllegalStateException
       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
       at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:154)
       at org.apache.log4j.Category.forcedLog(Category.java:388)
       at org.apache.log4j.Category.log(Category.java:853)
       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
       at java.lang.Thread.run(Thread.java:619)
      Exception in thread "EvictionTimer-0" java.lang.NullPointerException
       at org.apache.log4j.LogManager.getLogger(LogManager.java:188)
       at org.apache.log4j.Logger.getLogger(Logger.java:104)
       at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:283)
       at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
       at org.jboss.cache.eviction.BaseEvictionAlgorithm.process(BaseEvictionAlgorithm.java:107)
       at org.jboss.cache.eviction.EvictionTimerTask$1.run(EvictionTimerTask.java:114)
       at java.util.TimerThread.mainLoop(Timer.java:512)
       at java.util.TimerThread.run(Timer.java:462)
      Listening for transport dt_socket at address: 8000
      17.07.2008 10:50:27 org.apache.catalina.loader.WebappClassLoader loadClass
      INFO: Illegal access: this web application instance has been stopped already. Could not load org.apache.log4j.spi.VectorWriter. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
      java.lang.IllegalStateException
       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
       at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:154)
       at org.apache.log4j.Category.forcedLog(Category.java:388)
       at org.apache.log4j.Category.log(Category.java:853)
       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
       at org.jgroups.protocols.UDP.run(UDP.java:268)
       at java.lang.Thread.run(Thread.java:619)
      Exception in thread "EvictionTimer-1" java.lang.NullPointerException
       at org.apache.log4j.LogManager.getLogger(LogManager.java:188)
       at org.apache.log4j.Logger.getLogger(Logger.java:104)
       at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:283)
       at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
       at org.jboss.cache.eviction.BaseEvictionAlgorithm.process(BaseEvictionAlgorithm.java:107)
       at org.jboss.cache.eviction.EvictionTimerTask$1.run(EvictionTimerTask.java:114)
       at java.util.TimerThread.mainLoop(Timer.java:512)
       at java.util.TimerThread.run(Timer.java:462)
      


      Now I have looked around to see if any one else has this problem. I cant find a definitive answer that the problem is x,y,z but it looks like a class loader issue with regards logging.

      Here is a bug report in tomcat 5.x version with similar exception problems.
      https://issues.apache.org/bugzilla/show_bug.cgi?id=41504

      Im using the 6 version and still get it(Even though it was never marked as fixed).

      So the question is where is the problem, tomcats fault, commons logging,log4j or jboss cache/jgroups for not cleaning up when using commons logging? The logging libraries are packed in the war, have moved them out of the war into the common lib directory in tomcat but made no difference. I have tried commons 1.1 and 1.1.1. Using log4j 1.2.14 version.

      Any thoughts??

      Thanks,
      LL

      p.s When i say jboss/jgroups fault for not cleaning up, i have read that sometimes you should call
      LogFactory.release(Thread.currentThread().getContextClassLoader()) to release the resources it takes.


        • 1. Re: Clean shutdown/redeploy in Tomcat
          manik

          Hmm, I can't say I've seen this one before. I'm guessing moving cachhe and jgroups libraries outside the war fixes it though? (I know this may not be ideal ...)

          • 2. Re: Clean shutdown/redeploy in Tomcat
            lovelyliatroim

             


            Hmm, I can't say I've seen this one before. I'm guessing moving cachhe and jgroups libraries outside the war fixes it though? (I know this may not be ideal ...)


            Moving them into $TOMCAT_HOME/lib solves the problem but like you said its not ideal having it this way!!! Adds admin overhead which i would like to avoid!!



            • 3. Re: Clean shutdown/redeploy in Tomcat
              jreeman

              Hi,

              I have exactly the same error. Did you find a solution ?

              • 4. Re: Clean shutdown/redeploy in Tomcat
                jreeman
                • 5. Re: Clean shutdown/redeploy in Tomcat
                  lovelyliatroim

                  No I didnt, apart from moving libraries to the lib directory, this will stop the dump.

                  Havent looked at it since then to be honest and is coming soon back on the radar.

                  Let me know if you find a solution other than moving the jars to the lib directory.

                  Thanks,
                  LL

                  • 6. Re: Clean shutdown/redeploy in Tomcat
                    jreeman

                    hello,

                    it sounds like a log4j bug. I'm using 1.2.14 version and you ?

                    Perhaps it will be fixed in the very last version 1.2.15 ?

                    • 7. Re: Clean shutdown/redeploy in Tomcat
                      lovelyliatroim

                      Same version 1.2.14.


                      Perhaps it will be fixed in the very last version 1.2.15 ?

                      Only one way to find out..........
                      Stack trace is different, still illegal state exception, interesting though is the
                      "Caused by: java.lang.ClassNotFoundException: org.apache.log4j.spi.NOPLoggerRepository". Exists in the JAR but obviously looking for it at the wrong classloader level.



                      INFO: Illegal access: this web application instance has been stopped already. Could not load org.apache.log4j.spi.NOPLoggerRepository. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
                      java.lang.IllegalStateException
                       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
                       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
                       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
                       at org.apache.log4j.LogManager.getLoggerRepository(LogManager.java:175)
                       at org.apache.log4j.LogManager.getLogger(LogManager.java:199)
                       at org.apache.log4j.Logger.getLogger(Logger.java:105)
                       at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:283)
                       at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:171)
                       at org.jboss.cache.eviction.BaseEvictionAlgorithm.initialize(BaseEvictionAlgorithm.java:81)
                       at org.jboss.cache.eviction.BaseEvictionAlgorithm.process(BaseEvictionAlgorithm.java:106)
                       at org.jboss.cache.eviction.EvictionTimerTask.handleRegion(EvictionTimerTask.java:133)
                       at org.jboss.cache.eviction.EvictionTimerTask.processRegions(EvictionTimerTask.java:118)
                       at org.jboss.cache.eviction.EvictionTimerTask.access$000(EvictionTimerTask.java:28)
                       at org.jboss.cache.eviction.EvictionTimerTask$1.run(EvictionTimerTask.java:106)
                       at java.util.TimerThread.mainLoop(Timer.java:512)
                       at java.util.TimerThread.run(Timer.java:462)
                      Exception in thread "EvictionTimer-0" java.lang.NoClassDefFoundError: org/apache/log4j/spi/NOPLoggerRepository
                       at org.apache.log4j.LogManager.getLoggerRepository(LogManager.java:175)
                       at org.apache.log4j.LogManager.getLogger(LogManager.java:199)
                       at org.apache.log4j.Logger.getLogger(Logger.java:105)
                       at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:283)
                       at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:171)
                       at org.jboss.cache.eviction.BaseEvictionAlgorithm.initialize(BaseEvictionAlgorithm.java:81)
                       at org.jboss.cache.eviction.BaseEvictionAlgorithm.process(BaseEvictionAlgorithm.java:106)
                       at org.jboss.cache.eviction.EvictionTimerTask.handleRegion(EvictionTimerTask.java:133)
                       at org.jboss.cache.eviction.EvictionTimerTask.processRegions(EvictionTimerTask.java:118)
                       at org.jboss.cache.eviction.EvictionTimerTask.access$000(EvictionTimerTask.java:28)
                       at org.jboss.cache.eviction.EvictionTimerTask$1.run(EvictionTimerTask.java:106)
                       at java.util.TimerThread.mainLoop(Timer.java:512)
                       at java.util.TimerThread.run(Timer.java:462)
                      Caused by: java.lang.ClassNotFoundException: org.apache.log4j.spi.NOPLoggerRepository
                       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1358)
                       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
                       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
                       ... 13 more
                      log4j:ERROR Error occured while converting date.
                      java.lang.NullPointerException
                       at java.lang.System.arraycopy(Native Method)
                       at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:328)
                       at java.lang.StringBuffer.getChars(StringBuffer.java:201)
                       at org.apache.log4j.helpers.AbsoluteTimeDateFormat.format(AbsoluteTimeDateFormat.java:119)
                       at java.text.DateFormat.format(DateFormat.java:316)
                       at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:444)
                       at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
                       at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
                       at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
                       at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
                       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
                       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
                       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
                       at org.apache.log4j.Category.callAppenders(Category.java:206)
                       at org.apache.log4j.Category.forcedLog(Category.java:391)
                       at org.apache.log4j.Category.log(Category.java:856)
                       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                      26.09.2008 11:52:52 org.apache.catalina.loader.WebappClassLoader loadClass
                      INFO: Illegal access: this web application instance has been stopped already. Could not load java.io.LineNumberReader. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
                      java.lang.IllegalStateException
                       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
                       at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
                       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
                       at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:78)
                       at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:407)
                       at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:305)
                       at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
                       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
                       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
                       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
                       at org.apache.log4j.Category.callAppenders(Category.java:206)
                       at org.apache.log4j.Category.forcedLog(Category.java:391)
                       at org.apache.log4j.Category.log(Category.java:856)
                       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                      log4j:ERROR Error occured while converting date.
                      java.lang.NullPointerException
                       at java.lang.System.arraycopy(Native Method)
                       at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:328)
                       at java.lang.StringBuffer.getChars(StringBuffer.java:201)
                       at org.apache.log4j.helpers.AbsoluteTimeDateFormat.format(AbsoluteTimeDateFormat.java:119)
                       at java.text.DateFormat.format(DateFormat.java:316)
                       at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:444)
                       at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
                       at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
                       at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
                       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
                       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
                       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
                       at org.apache.log4j.Category.callAppenders(Category.java:206)
                       at org.apache.log4j.Category.forcedLog(Category.java:391)
                       at org.apache.log4j.Category.log(Category.java:856)
                       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                       [Timer,10.251.153.36:1897] ERROR protocols.PING - failed sending discovery request
                      java.lang.NullPointerException
                       at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                       at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                       at org.jgroups.protocols.TP.down(TP.java:1008)
                       at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:220)
                       at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:214)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:385)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                      log4j:ERROR Error occured while converting date.
                      java.lang.NullPointerException
                       at java.lang.System.arraycopy(Native Method)
                       at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:328)
                       at java.lang.StringBuffer.getChars(StringBuffer.java:201)
                       at org.apache.log4j.helpers.AbsoluteTimeDateFormat.format(AbsoluteTimeDateFormat.java:119)
                       at java.text.DateFormat.format(DateFormat.java:316)
                       at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:444)
                       at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
                       at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
                       at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
                       at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
                       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
                       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
                       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
                       at org.apache.log4j.Category.callAppenders(Category.java:206)
                       at org.apache.log4j.Category.forcedLog(Category.java:391)
                       at org.apache.log4j.Category.log(Category.java:856)
                       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                      log4j:ERROR Error occured while converting date.
                      java.lang.NullPointerException
                       at java.lang.System.arraycopy(Native Method)
                       at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:328)
                       at java.lang.StringBuffer.getChars(StringBuffer.java:201)
                       at org.apache.log4j.helpers.AbsoluteTimeDateFormat.format(AbsoluteTimeDateFormat.java:119)
                       at java.text.DateFormat.format(DateFormat.java:316)
                       at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:444)
                       at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
                       at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
                       at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
                       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
                       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
                       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
                       at org.apache.log4j.Category.callAppenders(Category.java:206)
                       at org.apache.log4j.Category.forcedLog(Category.java:391)
                       at org.apache.log4j.Category.log(Category.java:856)
                       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                       [Timer,10.251.153.36:1899] ERROR protocols.PING - failed sending discovery request
                      java.lang.NullPointerException
                       at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                       at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                       at org.jgroups.protocols.TP.down(TP.java:1008)
                       at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:220)
                       at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:214)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:385)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                      log4j:ERROR Error occured while converting date.
                      java.lang.NullPointerException
                       at java.lang.System.arraycopy(Native Method)
                       at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:328)
                       at java.lang.StringBuffer.getChars(StringBuffer.java:201)
                       at org.apache.log4j.helpers.AbsoluteTimeDateFormat.format(AbsoluteTimeDateFormat.java:119)
                       at java.text.DateFormat.format(DateFormat.java:316)
                       at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:444)
                       at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
                       at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
                       at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
                       at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
                       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
                       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
                       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
                       at org.apache.log4j.Category.callAppenders(Category.java:206)
                       at org.apache.log4j.Category.forcedLog(Category.java:391)
                       at org.apache.log4j.Category.log(Category.java:856)
                       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                      log4j:ERROR Error occured while converting date.
                      java.lang.NullPointerException
                       at java.lang.System.arraycopy(Native Method)
                       at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:328)
                       at java.lang.StringBuffer.getChars(StringBuffer.java:201)
                       at org.apache.log4j.helpers.AbsoluteTimeDateFormat.format(AbsoluteTimeDateFormat.java:119)
                       at java.text.DateFormat.format(DateFormat.java:316)
                       at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:444)
                       at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
                       at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
                       at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
                       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
                       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
                       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
                       at org.apache.log4j.Category.callAppenders(Category.java:206)
                       at org.apache.log4j.Category.forcedLog(Category.java:391)
                       at org.apache.log4j.Category.log(Category.java:856)
                       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                       [Timer,10.251.153.36:1897] ERROR protocols.PING - failed sending discovery request
                      java.lang.NullPointerException
                       at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                       at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                       at org.jgroups.protocols.TP.down(TP.java:1008)
                       at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:220)
                       at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:214)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:385)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                      log4j:ERROR Error occured while converting date.
                      java.lang.NullPointerException
                       at java.lang.System.arraycopy(Native Method)
                       at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:328)
                       at java.lang.StringBuffer.getChars(StringBuffer.java:201)
                       at org.apache.log4j.helpers.AbsoluteTimeDateFormat.format(AbsoluteTimeDateFormat.java:119)
                      
                       at java.text.DateFormat.format(DateFormat.java:316)
                       at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:444)
                       at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
                       at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
                       at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
                       at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
                       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
                       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
                       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
                       at org.apache.log4j.Category.callAppenders(Category.java:206)
                       at org.apache.log4j.Category.forcedLog(Category.java:391)
                       at org.apache.log4j.Category.log(Category.java:856)
                       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                      log4j:ERROR Error occured while converting date.
                      java.lang.NullPointerException
                       at java.lang.System.arraycopy(Native Method)
                       at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:328)
                       at java.lang.StringBuffer.getChars(StringBuffer.java:201)
                       at org.apache.log4j.helpers.AbsoluteTimeDateFormat.format(AbsoluteTimeDateFormat.java:119)
                       at java.text.DateFormat.format(DateFormat.java:316)
                       at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:444)
                       at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
                       at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
                       at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
                       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
                       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
                       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
                       at org.apache.log4j.Category.callAppenders(Category.java:206)
                       at org.apache.log4j.Category.forcedLog(Category.java:391)
                       at org.apache.log4j.Category.log(Category.java:856)
                       at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:251)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:389)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)
                       [Timer,10.251.153.36:1899] ERROR protocols.PING - failed sending discovery request
                      java.lang.NullPointerException
                       at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                       at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                       at org.jgroups.protocols.TP.down(TP.java:1008)
                       at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:220)
                       at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:214)
                       at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:385)
                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                       at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:619)


                      • 8. Re: Clean shutdown/redeploy in Tomcat
                        lovelyliatroim

                        Moved just the log4j lib into the lib folder but still hangs

                        Stack trace is different

                        12:03:15,719 [Timer,10.251.153.36:1936] ERROR protocols.PING - failed sending discovery request
                        java.lang.NullPointerException
                         at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                         at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                         at org.jgroups.protocols.TP.down(TP.java:1008)
                         at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:220)
                         at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:214)
                         at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:385)
                         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                         at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                         at java.lang.Thread.run(Thread.java:619)
                        12:03:17,109 [Timer,null] ERROR util.TimeScheduler - failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@fd92e7
                        java.lang.NullPointerException
                         at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                         at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                         at org.jgroups.protocols.pbcast.STABLE$StableTask.run(STABLE.java:807)
                         at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:196)
                         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                         at java.lang.Thread.run(Thread.java:619)
                        12:03:17,219 [Timer,10.251.153.36:1936] ERROR protocols.PING - failed sending discovery request
                        java.lang.NullPointerException
                         at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                         at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                         at org.jgroups.protocols.TP.down(TP.java:1008)
                         at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:220)
                         at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:214)
                         at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:385)
                         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                         at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                         at java.lang.Thread.run(Thread.java:619)
                        12:03:32,843 [Timer,null] ERROR protocols.PING - failed sending discovery request
                        java.lang.NullPointerException
                         at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                         at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                         at org.jgroups.protocols.TP.down(TP.java:1008)
                         at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:220)
                         at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:214)
                         at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:385)
                         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                         at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                         at java.lang.Thread.run(Thread.java:619)
                        12:03:34,343 [Timer,null] ERROR protocols.PING - failed sending discovery request
                        java.lang.NullPointerException
                         at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                         at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                         at org.jgroups.protocols.TP.down(TP.java:1008)
                         at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:220)
                         at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:214)
                         at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:385)
                         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                         at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                         at java.lang.Thread.run(Thread.java:619)
                        12:03:42,562 [Timer,null] ERROR util.TimeScheduler - failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@fd92e7
                        java.lang.NullPointerException
                         at org.apache.log4j.Category.isEnabledFor(Category.java:749)
                         at org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:327)
                         at org.jgroups.protocols.pbcast.STABLE$StableTask.run(STABLE.java:807)
                         at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:196)
                         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                         at java.lang.Thread.run(Thread.java:619)
                        
                        


                        Looks like you would have to move all cache related jars up to the lib level!! :(



                        • 9. Re: Clean shutdown/redeploy in Tomcat
                          jreeman

                          yes, a request feature has been added, see the detail here https://jira.jboss.org/jira/browse/JBCACHE-1416.

                          • 10. Re: Clean shutdown/redeploy in Tomcat
                            jreeman

                            I tried to upgrade the jboss cache to 3.0.0.CR1 version to see if there are always the class loader problems but when starting by cache I have a lot of exception :

                            org.jboss.cache.CacheException: Unable to invoke method public void org.jboss.cache.RegionManagerImpl.start() on object
                            at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:148)
                            at org.jboss.cache.factories.ComponentRegistry$PrioritizedMethod.invoke(ComponentRegistry.java:973)
                            at org.jboss.cache.factories.ComponentRegistry.internalStart(ComponentRegistry.java:749)
                            at org.jboss.cache.factories.ComponentRegistry.start(ComponentRegistry.java:613)
                            at org.jboss.cache.invocation.CacheInvocationDelegate.start(CacheInvocationDelegate.java:355)
                            at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:122)
                            at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:94)
                            at fr.arkhineo.ntp.controller.logic.cache.ControllerCache.start(ControllerCache.java:49)
                            at fr.arkhineo.ntp.controller.ApplicationLifeCycle.initializeApplication(ApplicationLifeCycle.java:58)
                            at fr.arkhineo.ntp.controller.servlet.ControllerDispatchServlet.init(ControllerDispatchServlet.java:35)
                            at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1172)
                            at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:992)
                            at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4058)
                            at org.apache.catalina.core.StandardContext.start(StandardContext.java:4371)
                            at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
                            at org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
                            at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
                            at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
                            at org.apache.catalina.core.StandardService.start(StandardService.java:516)
                            at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
                            at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                            at java.lang.reflect.Method.invoke(Method.java:597)
                            at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
                            at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
                            Caused by: java.lang.reflect.InvocationTargetException
                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                            at java.lang.reflect.Method.invoke(Method.java:597)
                            at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:144)
                            ... 26 more
                            Caused by: org.jboss.cache.config.ConfigurationException: maxNodes must be must be configured to a value greater than or equal to 0
                            at org.jboss.cache.eviction.FIFOAlgorithmConfig.validate(FIFOAlgorithmConfig.java:65)
                            at org.jboss.cache.RegionManagerImpl.start(RegionManagerImpl.java:121)
                            ... 31 more
                            16 oct. 2008 15:06:28 org.apache.catalina.core.StandardContext loadOnStartup
                            GRAVE: La servlet a g�n�r� une exception "load()"
                            org.jboss.cache.config.ConfigurationException: maxNodes must be must be configured to a value greater than or equal to 0
                            at org.jboss.cache.eviction.FIFOAlgorithmConfig.validate(FIFOAlgorithmConfig.java:65)
                            at org.jboss.cache.RegionManagerImpl.start(RegionManagerImpl.java:121)
                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                            at java.lang.reflect.Method.invoke(Method.java:597)
                            at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:144)
                            at org.jboss.cache.factories.ComponentRegistry$PrioritizedMethod.invoke(ComponentRegistry.java:973)
                            at org.jboss.cache.factories.ComponentRegistry.internalStart(ComponentRegistry.java:749)
                            at org.jboss.cache.factories.ComponentRegistry.start(ComponentRegistry.java:613)
                            at org.jboss.cache.invocation.CacheInvocationDelegate.start(CacheInvocationDelegate.java:355)
                            at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:122)
                            at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:94)
                            at fr.arkhineo.ntp.controller.logic.cache.ControllerCache.start(ControllerCache.java:49)
                            at fr.arkhineo.ntp.controller.ApplicationLifeCycle.initializeApplication(ApplicationLifeCycle.java:58)
                            at fr.arkhineo.ntp.controller.servlet.ControllerDispatchServlet.init(ControllerDispatchServlet.java:35)
                            at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1172)
                            at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:992)
                            at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4058)
                            at org.apache.catalina.core.StandardContext.start(StandardContext.java:4371)
                            at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
                            at org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
                            at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
                            at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
                            at org.apache.catalina.core.StandardService.start(StandardService.java:516)
                            at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
                            at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                            at java.lang.reflect.Method.invoke(Method.java:597)
                            at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
                            at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
                            16 oct. 2008 15:06:28 org.apache.coyote.http11.Http11Protocol start
                            INFO: D�marrage de Coyote HTTP/1.1 sur http-8280
                            16 oct. 2008 15:06:28 org.apache.jk.common.ChannelSocket init
                            INFO: JK: ajp13 listening on /0.0.0.0:8209
                            16 oct. 2008 15:06:28 org.apache.jk.server.JkMain start
                            INFO: Jk running ID=0 time=0/155 config=null
                            16 oct. 2008 15:06:28 org.apache.catalina.startup.Catalina start
                            INFO: Server startup in 1964 ms
                            16 oct. 2008 15:06:28 org.apache.catalina.core.ApplicationContext log
                            GRAVE: StandardWrapper.Throwable
                            org.jboss.cache.CacheException: Unable to invoke method public void org.jboss.cache.RegionManagerImpl.start() on object
                            at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:148)
                            at org.jboss.cache.factories.ComponentRegistry$PrioritizedMethod.invoke(ComponentRegistry.java:973)
                            at org.jboss.cache.factories.ComponentRegistry.internalStart(ComponentRegistry.java:749)
                            at org.jboss.cache.factories.ComponentRegistry.start(ComponentRegistry.java:613)
                            at org.jboss.cache.invocation.CacheInvocationDelegate.start(CacheInvocationDelegate.java:355)
                            at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:122)
                            at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:94)
                            at fr.arkhineo.ntp.controller.logic.cache.ControllerCache.start(ControllerCache.java:49)
                            at fr.arkhineo.ntp.controller.ApplicationLifeCycle.initializeApplication(ApplicationLifeCycle.java:58)
                            at fr.arkhineo.ntp.controller.servlet.ControllerDispatchServlet.init(ControllerDispatchServlet.java:35)
                            at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1172)
                            at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:808)
                            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129)
                            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
                            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
                            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
                            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
                            at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
                            at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
                            at java.lang.Thread.run(Thread.java:619)
                            Caused by: java.lang.reflect.InvocationTargetException
                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                            at java.lang.reflect.Method.invoke(Method.java:597)
                            at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:144)
                            ... 21 more
                            Caused by: org.jboss.cache.config.ConfigurationException: maxNodes must be must be configured to a value greater than or equal to 0
                            at org.jboss.cache.eviction.FIFOAlgorithmConfig.validate(FIFOAlgorithmConfig.java:65)
                            at org.jboss.cache.RegionManagerImpl.start(RegionManagerImpl.java:121)
                            ... 26 more
                            16 oct. 2008 15:06:28 org.apache.catalina.core.StandardWrapperValve invoke
                            GRAVE: Exception lors de l'allocation pour la servlet dispatchServlet
                            org.jboss.cache.config.ConfigurationException: maxNodes must be must be configured to a value greater than or equal to 0
                            at org.jboss.cache.eviction.FIFOAlgorithmConfig.validate(FIFOAlgorithmConfig.java:65)
                            at org.jboss.cache.RegionManagerImpl.start(RegionManagerImpl.java:121)
                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                            at java.lang.reflect.Method.invoke(Method.java:597)
                            at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:144)
                            at org.jboss.cache.factories.ComponentRegistry$PrioritizedMethod.invoke(ComponentRegistry.java:973)
                            at org.jboss.cache.factories.ComponentRegistry.internalStart(ComponentRegistry.java:749)
                            at org.jboss.cache.factories.ComponentRegistry.start(ComponentRegistry.java:613)
                            at org.jboss.cache.invocation.CacheInvocationDelegate.start(CacheInvocationDelegate.java:355)
                            at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:122)
                            at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:94)
                            at fr.arkhineo.ntp.controller.logic.cache.ControllerCache.start(ControllerCache.java:49)
                            at fr.arkhineo.ntp.controller.ApplicationLifeCycle.initializeApplication(ApplicationLifeCycle.java:58)
                            at fr.arkhineo.ntp.controller.servlet.ControllerDispatchServlet.init(ControllerDispatchServlet.java:35)
                            at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1172)
                            at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:808)
                            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129)
                            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
                            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
                            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
                            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
                            at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
                            at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
                            at java.lang.Thread.run(Thread.java:619)


                            here is my configuration :
                            <?xml version="1.0" encoding="UTF-8"?>
                            
                            <server>
                             <mbean code="org.jboss.cache.jmx.CacheJmxWrapper"
                             name="jboss.cache:service=TreeCache">
                            
                             <depends>jboss:service=Naming</depends>
                             <attribute name="IsolationLevel">READ_COMMITTED</attribute>
                             <attribute name="CacheMode">REPL_SYNC</attribute>
                            
                             <!-- Policy configurations : FIFO Policy -->
                             <attribute name="EvictionPolicyConfig">
                             <config>
                             <attribute name="policyClass">org.jboss.cache.eviction.FIFOPolicy</attribute>
                             </config>
                             </attribute>
                            
                             <attribute name="ClusterName">Cluster-Of-Controllers</attribute>
                            
                             <!-- JGroups protocol stack properties -->
                             <attribute name="ClusterConfig">
                             <config>
                             <TCP start_port="7800" loopback="false" enable_diagnostics="false" />
                            
                             <!-- choose a mcast_addr that no other machine/processus use -->
                             <!-- ip_ttl = 0 for local cluster, ip_ttl=1 for multicast in the current network/sub-network > 1
                             and < 32 same organization or site -->
                             <MPING timeout="3000" bind_to_all_interfaces="false"
                             mcast_addr="225.0.0.1" mcast_port="7500" ip_ttl="0"
                             num_initial_members="2" />
                            
                             <!-- suspecting node failures only when a regular TCP connection to the node fails -->
                             <FD_SOCK />
                            
                             <!-- configuration to have a sequence for each message and detect/treat gap in this sequence -->
                             <pbcast.NAKACK gc_lag="50" max_xmit_size="8192" retransmit_timeout="600,1200,2400,4800" />
                            
                             <!-- the sender resends the message if it hasn't received an ACK after 100 ms the first time -->
                             <UNICAST timeout="600,1200,2400"/>
                            
                             <!-- time before two garbage collection -->
                             <pbcast.STABLE desired_avg_gossip="20000" />
                            
                             <!-- merge partitioned cluster -->
                             <MERGE2 max_interval="10000" min_interval="2000"/>
                            
                             <!-- managing nodes that joins -->
                             <pbcast.GMS join_retry_timeout="2000" join_timeout="5000" print_local_addr="false" shun="true" />
                            
                             <!-- transfer the state of the cache to the other node -->
                             <pbcast.STATE_TRANSFER />
                             </config>
                             </attribute>
                            
                             <attribute name="FetchStateOnStartup">false</attribute>
                             <attribute name="InitialStateRetrievalTimeout">5000</attribute>
                             <attribute name="LockAcquisitionTimeout">15000</attribute>
                             <attribute name="UseMarshalling">false</attribute>
                             <attribute name="CacheLoaderShared">true</attribute>
                             <attribute name="CacheLoaderPreload">/</attribute>
                             <attribute name="CacheLoaderPassivation">false</attribute>
                             <attribute name="CacheLoaderFetchPersistentState">false</attribute>
                             <attribute name="CacheLoaderFetchTransientState">false</attribute>
                             <attribute name="CacheLoaderAsynchronous">false</attribute>
                             </mbean>
                            </server>


                            Could you tell me where I can find documentation to upgrade my xml config file ?

                            Thx

                            • 11. Re: Clean shutdown/redeploy in Tomcat
                              manik

                              I am updating docs at the moment for inclusion in 3.0.0.CR2.

                              In the meanwhile, I recommend either looking at some of the sample cfgs shipped with CR1 as a guide, using the XSD schema shipped with CR1, or using the config conversion scripts (config2to3.sh or .bat) that is in the CR1 distribution.

                              • 12. Re: Clean shutdown/redeploy in Tomcat
                                lahirukarunatilake

                                I too have a similar issue in JBC 3.2.5 with Tomcat 6.0.13 and Hibernate 3. Tomcat does not shutdown gracefully when JBC is configured as the L2 cache for hibernate. Is there a plan for this issue?

                                • 13. Re: Clean shutdown/redeploy in Tomcat
                                  deepaksrivastavaz

                                  Refer https://community.jboss.org/thread/213493 for a possible solution.

                                   

                                  --Deepak