1 2 3 4 Previous Next 59 Replies Latest reply on Jun 20, 2011 1:58 AM by jamshed.katta Go to original post
      • 15. Re: Possible Deadlock Sending to JMS Queue
        hosier.david

        All the messages look like "This is a notification (9990)", where the number is incremented for each message to help me identify where things crap out. In my tests, I actually send out 3 messages when the client subscribes before I start sending the 10,000 messages. So if you get similar behavior, you may actually get a few more messages than 5818 since your test will not have those initial three messages. In case you want to send those in your test as well, the three messages that are sent at the beginning are:

         

        Welcome 1

        It's so nice to meet you

        Hopefully we can have lots of fun together

        • 16. Re: Possible Deadlock Sending to JMS Queue
          bill.burke

          What about your hornetq-configuration.xml?  What does that look like?

          • 17. Possible Deadlock Sending to JMS Queue
            hosier.david

            Here is a question for Clebert perhaps, and maybe Bill knows the answer. I have TRACE logging turned on for the org category. When the REST API does its acknowledgements, which I can see with the aforementioned log message, I see absolutely no logging at all from the HornetQ code. That seems a little surprising to me considering how much logging I am seeing when the messages are sent to the JMS Queue. Would you expect that I should be seeing some HornetQ logging when an acknowledgment is happening while the TRACE level is enabled?

            • 18. Re: Possible Deadlock Sending to JMS Queue
              hosier.david

              Here is my conifiguration. I have the blocking configuration enabled here with my paging configuration commented out. As a reminder, this is on JBoss AS 6.0.0.Final that I have updated to HornetQ 2.2.2.Final.

              • 19. Re: Possible Deadlock Sending to JMS Queue
                clebert.suconic

                Bill fixed some stuff that I believe was preventing the ACKs to be accepted. This would be some issue on the Rest code. You would need to update your rest API from trunk.

                • 20. Re: Possible Deadlock Sending to JMS Queue
                  hosier.david

                  Ok, great. I'll check that out. Thanks. I assume I just need to follow the directions on http://community.jboss.org/wiki/BuildingHornetQFromSource to get it all built and then just update my server.

                  • 21. Re: Possible Deadlock Sending to JMS Queue
                    clebert.suconic

                    We just moved to maven, so that WIKI is probably not up to date yet:

                     

                    http://community.jboss.org/thread/165671?tstart=0

                    • 22. Re: Possible Deadlock Sending to JMS Queue
                      hosier.david

                      I've built from trunk, removed the old hornetq-rest-2.2.2.Final.jar from my war, and put hornetq-rest-2.2.3-SNAPSHOT.jar in my war. The problem still remains where messages are not being acknowledged. When using the BLOCK setting, my test continues to stop at the same spot. When using PAGE, the page files never go away, but I do get all the messages delivered. I am aiming to deliver to a customer tomorrow (not your problem, I know), so my only real choice is to use the PAGE setting, but then I will start to fill up my customer's disk... I don't use subversion, so I don't really know how to diff between versions. I am using a Queue push subscription, so I am assuming the code in question is http://anonsvn.jboss.org/repos/hornetq/trunk/hornetq-rest/hornetq-rest/src/main/java/org/hornetq/rest/queue/push/PushConsumer.java. If I could see what was changed that you think should have fixed the problem, maybe I can poke around and try some things. The code seems correct at first glance however, but I've never used the HornetQ core API, so I don't know for sure.

                      • 23. Re: Possible Deadlock Sending to JMS Queue
                        clebert.suconic

                        Any chance you could create a code example showing the issue?

                        • 24. Re: Possible Deadlock Sending to JMS Queue
                          hosier.david

                          I've reproduced it with a very simple test, but before I put that info together, I want to make it clear that I did not update the other HornetQ jars in my server to the versions built from trunk. I am building the distribution right now so I can try that, but I was wondering if you think that should matter given the changes that Bill made?

                          • 25. Possible Deadlock Sending to JMS Queue
                            bill.burke

                            Attach it.  I don't think my changes will matter based on your previous posts.

                            • 26. Re: Possible Deadlock Sending to JMS Queue
                              hosier.david

                              Ok, I have a "simpler" test reproducing this. I apologize for the number of steps, but some things just need to be setup to get the new version of HornetQ into JBoss. Here are the steps to setup the test.

                               

                               

                              1. Start with a stock JBoss 6.0.0.Final

                              2. Update to HornetQ 2.2.2.Final using the following steps:

                              - from the HornetQ 2.2.2 distribution lib directory, copy netty.jar, hornetq-bootstrap.jar, hornetq-core.jar, hornetq-jboss-as-integration.jar, hornetq-jms.jar, and hornetq-logging.jar to $JBOSS_HOME/common/lib

                              - from the HornetQ 2.2.2 distribution hornetq-ra.rar.dir, copy hornetq-ra.jar to $JBOSS_HOME/server/default/deploy/jms-ra.rar

                              3. Delete $JBOSS_HOME/server/default/deploy/ROOT.war (I am binding my war to the root context, annoying perhaps for this test, but I wanted to change as little as possible from my real setup)

                              4. Put the attached notification-test.war and hornetq-rest.war into the deploy directory

                              5. Put the attached commons-httpclient-3.1.jar into common/lib (I tried including this in both my wars instead, but the HornetQ rest stuff just kept throwing a ClassNotFoundException on the HttpMethod class from this jar)

                               

                               

                              To run the test, issue the following curl command to get the subscription established.

                              curl -X POST http://localhost:8080/test/subscription

                               

                              Then either run the attached send.js node.js script or create something similar to send the 10,000 messages to the test bean.

                               

                              Using my node.js script, you will see that the subscription stops getting notifications at exactly the 5818th message every time (when using the BLOCK configuration). If you switch to the PAGE configuration, you should see all messages come through on the subscription, but the paging files never get cleaned up.

                              • 27. Re: Possible Deadlock Sending to JMS Queue
                                clebert.suconic

                                Where are you creating the Subscription, where are you receiving/consuming the messages and where are you Acking the messages on the test?

                                • 28. Re: Possible Deadlock Sending to JMS Queue
                                  clebert.suconic

                                  Also: Where are you closing / removing the subscription when you don't need any more? Question is: how many subscriptions are you opening?

                                  • 29. Re: Possible Deadlock Sending to JMS Queue
                                    hosier.david

                                    All the code is in the source attachment to my last post. A little background first. I have the need to do some things that I cannot do by just using the HornetQ REST API from a client, so I actually wrap the creation of the REST subscriptions in my own REST service so that I have more flexibility and to reduce the complexity from my client's perspective. This test approximates what I'm doing in my real code. If you look at the source, you will see my test resource class that provides all the functionality for the test. The resource provides the subscribe method which you must POST to in order to create the queue and the REST subscription as described in my previous post. In my test you just need to POST to the subscribe() method once, so there is only one queue and one subscription, that's it. Looking at SimpleNotificationTestResource.java from my source zip that I attached, the flow is

                                     

                                    1. POST to the subscribe() method, which creates the JMS queue and creates the REST push subscription on that queue with an endpoint that will send to a different method in the same resource. Not real life I know...it's just a test and not how my application actually works. I do get this same hanging behavior with an endpoint that is external to the server as well.

                                    2. POST to the sendTestNotification() method (as demonstrated by the previously attached send.js script), which publishes whatever was POSTed on to the Queue so that it can be delivered to the subscriber.

                                    3. The REST push subscription sends to the receive() method

                                     

                                    You will see a log message print when the sendTestNotification() method gets the POSTed notification, and you will see a log message when the subscriber gets the push from the REST subscription. You should see that the log messages stop coming and the thing is hung.