6 Replies Latest reply on Mar 29, 2011 1:35 AM by dasmurali

    Messages Stuck in 'Delivering State'.

    groovenarula

      Our system consists of one virtualized VMWare guest server running Red Hat Enterprise
      Server 64 bit, Java HotSpot(TM) 64-Bit Server VM
      and JBoss 5.1.0.GA. The machine runs a single instance of JBoss with Postgresql (running on a different server).

       

      The JBOSS instance is dedicated to processing JMS Messages for 3 tomcat based web application clients again running on different servers.

       

      We have made no other changes to JBOSS Messaging Postgresql Presistence manager's configuration other then disabling  'ChannelFactoryName'

       

      <!-- <depends optional-attribute-name="ChannelFactoryName">jboss.jgroups:service=ChannelFactory</depends> -->

       

      (referred the following : http://community.jboss.org/message/204473#204473 )

       

      The JMS clients are all Spring based web applications that consumer messages from one of the queues on the jboss server.

       

      We deployed this configuration to production this Monday. So far the server has been very stable. The only issue that we've run into is that there are (now) about 7 messages that are stuck in the 'Delivering' state since the last 3 days. So basically everytime we look at the queue's status using the jmx-console, it shows MessageCount is stuck at 7 and DeliveringCount is also stuck at 7. The tomcat servers have recevied several other messages since we've noticed these stuck messages. We tried to remove these messages using the 'removeAllMessages' operation, but we then saw an error in the jboss logs basically stating that messages that are in a DeliveringState cannot be removed.

       

      I would like to know if someone can please help me find a way to delete these messages. Also we have not been able to reproduce this behavior in our development / test enviroment. So we're wondering if this could have been caused by some intermittent communication failure (network not available etc) between the Jboss Servers and Tomcat Servers. Does anyone know of any such issues with the JBM version (I believe it's 1.4.3 GA) bundled with JBOSS AS 5.1.0.

       

      We do plan to upgrade to HornetQ in the near future (there is a critical bug in the current Final version of HornetQ that we're wait to be resolved in the next version before we migrate over the HornetQ).

       

      So if anyboby can shed some light on what could cause messages to end up in a DeliveryState and what steps I can take to remove these messages from the queue, it will be much appreciated.

       

      Thanks in advance.

       

      Gurvinder

        • 1. Messages Stuck in 'Delivering State'.
          groovenarula

          Can someone from the community please help ? We are now upto 29 messages and we keep getting more messages stuck in the queue. I would really appreciate it if someone can help me with how to diagnose the cuase and what I can do to 'release' these stuck messages.

           

          Thanks

          Gurvinder

          • 2. Messages Stuck in 'Delivering State'.
            welle

            Can it be that some of the threads that reads the messages are hanging? I would make a threaddump and browse through iu

             

            Not much of a help, though...

            • 3. Messages Stuck in 'Delivering State'.
              groovenarula

              Well a server restart flushed out the messages that were in delivering state. But I'm now wondering if this is something we are going to have to do on a regular basis....

              • 4. Messages Stuck in 'Delivering State'.
                welle

                The reason why a restart "solved it" is because you stop the deliver and the message "returns" to the queue/topic and then re-read after the restart by a thread that didn't hang.

                 

                You should not have to restart the server!

                • 5. Messages Stuck in 'Delivering State'.
                  ryanhos

                  Gurvinderpal,

                   

                  We're experiencing the "stuck in delivering" issue also, though on a clustered setup.  It occurs on queues consumed by MDBs and by JBoss ESB.  We don't yet have a solution, but I'm closing-in on one.  I'm wodering if it isn't a remoting timeout issue when the server is under high load (we're doing load testing) or during a stop-the-world GC.  This poster has had success increasing the remoting timeouts:  http://community.jboss.org/message/442276#442276

                   

                  The next time you notice this, check the consumer count on the queue.  Is it the expected value, or is it zero?  (I believe MDB-consumed queues should always have one consumer.)  What about the delivering count?  Does it match the number of stuck messages?  What happens if you stop and start the MDB consuming that queue via the jmx-console?  The delivering messages will disappear, but do the messages actually get consumed?  (have they perhaps already been consumed?  Can you prove this via the logs?)  The more evidence and information we gather about this, the closer we get to a solution.  However, most of JBoss' attention has been turned toward HornetQ (according to my limited powers of observation).  Even if you find the exact source of the problem, a solution may not arrive quickly or at all, unless you've paid-up your JBoss credits.

                  • 6. Messages Stuck in 'Delivering State'.
                    dasmurali

                    Hi,

                     

                    We are alos facing the same issue with JBoss Messaging (JBoss app server 5.1.0). After processing some messages, the Messages are "stuck" in the queue with both "MessageCount" and "DeliveryCount" as NON ZERO values and "Consumer Count" as ONE(we had one MDB deployed in the same server).

                     

                    We are using JBoss Messaging(1.4.3.GA) and JBoss remote(2.5.1) comes by default with JBoss 5.1.0.GA app server.

                    App server is running on 64 bit linux m/c with 2GB RAM.

                     

                    Please find the below thread dump at the time of "delivery stuck".

                     

                    <code>

                     

                    Thread: WorkManager(2)-13 : priority:5, demon:true, threadId:102, threadState:WAITING

                    sun.misc.Unsafe.park(Native Method)
                    java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
                    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
                    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
                    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
                    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
                    java.lang.Thread.run(Thread.java:595)

                    Thread: WorkManager(2)-14 : priority:5, demon:true, threadId:103, threadState:WAITING

                    sun.misc.Unsafe.park(Native Method)
                    java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
                    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
                    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
                    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
                    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
                    java.lang.Thread.run(Thread.java:595)

                    Thread: WorkManager(2)-15 : priority:5, demon:true, threadId:104, threadState:WAITING

                    sun.misc.Unsafe.park(Native Method)
                    java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
                    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
                    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
                    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
                    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
                    java.lang.Thread.run(Thread.java:595)

                    Thread: WorkManager(2)-16 : priority:5, demon:true, threadId:105, threadState:WAITING

                    sun.misc.Unsafe.park(Native Method)
                    java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
                    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
                    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
                    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
                    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
                    java.lang.Thread.run(Thread.java:595)

                    Thread: WorkManager(2)-17 : priority:5, demon:true, threadId:106, threadState:WAITING

                    sun.misc.Unsafe.park(Native Method)
                    java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
                    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
                    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
                    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
                    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
                    java.lang.Thread.run(Thread.java:595)

                    Thread: WorkManager(2)-18 : priority:5, demon:true, threadId:107, threadState:WAITING

                    sun.misc.Unsafe.park(Native Method)
                    java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
                    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
                    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
                    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
                    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
                    java.lang.Thread.run(Thread.java:595)

                    Thread: WorkManager(2)-19 : priority:5, demon:true, threadId:108, threadState:WAITING

                    sun.misc.Unsafe.park(Native Method)
                    java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
                    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
                    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
                    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
                    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
                    java.lang.Thread.run(Thread.java:595)

                     

                    </code>

                     

                     

                    I am thinking,  these are MDB (deployed in the same server) threads waiting to pick the messages. Not sure why they all are in "WAITING" state.

                     

                    The same application is working damn good in JBoss 4.0.4 server.

                     

                    Thanks and Regards

                    Murali Reddy