1 2 3 Previous Next 38 Replies Latest reply on Oct 11, 2010 11:18 AM by clebert.suconic Go to original post
      • 30. Re: What is the expected performance of the STOMP protocol?
        mrbeagle

        since I'm dequeing from stomp, should there still be a concept of window size?

         

         

        Here are my test scripts

         

        enqueue from java

         

        http://pastebin.com/Pw4Uc2GU

         

         

        php deqeue

         

        http://pastebin.com/PNUEGdGS

         

         

        I enqueue a  few thousand rows from java using jms, then run a script in php to dequeue. After between 700-1000 dequeues it locks up the php script and the hornet server itself becomes unresponsive. If I re-run my JMS enqueue test case during this lock up the enqueues time out with the exception I pasted above.

        • 31. Re: What is the expected performance of the STOMP protocol?
          clebert.suconic

          I will have to take a look on early next week.

           

           

          I will have to look at the changes Tim made over for this.

          • 32. Re: What is the expected performance of the STOMP protocol?
            timfox

            Works fine for me using TRUNK. Here are the last few lines of output from dequeue on my laptop:

             

            MSG: This is a test message: 2984
            MSG: This is a test message: 2985
            MSG: This is a test message: 2986
            MSG: This is a test message: 2987
            MSG: This is a test message: 2988
            MSG: This is a test message: 2989
            MSG: This is a test message: 2990
            MSG: This is a test message: 2991
            MSG: This is a test message: 2992
            MSG: This is a test message: 2993
            MSG: This is a test message: 2994
            MSG: This is a test message: 2995
            MSG: This is a test message: 2996
            MSG: This is a test message: 2997
            MSG: This is a test message: 2998
            MSG: This is a test message: 2999
            3000 messages dequeuedMSG: This is a test message: 3000
            1.1247551441193 seconds

             

            I ran your exact JMS client followed by your php script.

            • 33. Re: What is the expected performance of the STOMP protocol?
              timfox

              This doesn't follow. The stack trace you posted shows the client waiting in a call to create consumer:

               

              /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java -Xms2048m -Xmx2048m -Dfile.encoding=UTF-8 -classpath /Applications/IntelliJ IDEA 9.0.3.app/lib/idea_rt.jar:/Applications/IntelliJ IDEA 9.0.3.app/plugins/junit/lib/junit-rt.jar:/System/Library/Frameworks/JavaVM.framework/Versions/A/Resources/Deploy.bundle/Contents/Resources/Java/deploy.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/dt.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/javaws.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/jce.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/management-agent.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/plugin.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/sa-jdi.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/alt-rt.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/charsets.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/classes.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/jconsole.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/jsse.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/ui.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/apple_provider.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/dnsns.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/localedata.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/sunjce_provider.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/sunpkcs11.jar:/Users/jim/Code/InsightsRaven/viral/target/test-classes:/Users/jim/Code/InsightsRaven/viral/target/classes:/Users/jim/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/Users/jim/.m2/repository/org/jsoup/jsoup/1.2.2/jsoup-1.2.2.jar:/Users/jim/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar:/Users/jim/.m2/repository/com/jolbox/bonecp/0.7.0-rc3/bonecp-0.7.0-rc3.jar:/Users/jim/.m2/repository/com/google/collections/google-collections/1.0/google-collections-1.0.jar:/Users/jim/.m2/repository/org/slf4j/slf4j-api/1.5.10/slf4j-api-1.5.10.jar:/Users/jim/Code/InsightsRaven/insights-utils/target/classes:/Users/jim/.m2/repository/org/apache/thrift/libthrift/0.2.0/libthrift-0.2.0.jar:/Users/jim/.m2/repository/org/apache/httpcomponents/httpclient/4.0.1/httpclient-4.0.1.jar:/Users/jim/.m2/repository/org/apache/httpcomponents/httpcore/4.0.1/httpcore-4.0.1.jar:/Users/jim/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/jim/.m2/repository/commons-codec/commons-codec/1.3/commons-codec-1.3.jar:/Users/jim/.m2/repository/com/gravity/utilities/1.0-SNAPSHOT/utilities-1.0-SNAPSHOT.jar:/Users/jim/.m2/repository/org/slf4j/slf4j-log4j12/1.5.10/slf4j-log4j12-1.5.10.jar:/Users/jim/.m2/repository/log4j/log4j/1.2.14/log4j-1.2.14.jar:/Users/jim/.m2/repository/org/jdom/jdom/1.1/jdom-1.1.jar:/Users/jim/.m2/repository/com/google/code/gson/gson/1.4/gson-1.4.jar:/Users/jim/.m2/repository/org/jredis/jredis/1.0-rc1/jredis-1.0-rc1.jar:/Users/jim/.m2/repository/org/mongodb/mongo-java-driver/2.0/mongo-java-driver-2.0.jar:/Users/jim/.m2/repository/mysql/mysql-connector-java/5.1.12/mysql-connector-java-5.1.12.jar:/Users/jim/.m2/repository/javax/jms/jms/1.1/jms-1.1.jar:/Users/jim/.m2/repository/org/hornetq/hornetq-jms-client/2.1.2.Final/hornetq-jms-client-2.1.2.Final.jar:/Users/jim/.m2/repository/org/hornetq/hornetq-core-client/2.1.2.Final/hornetq-core-client-2.1.2.Final.jar:/Users/jim/.m2/repository/org/hornetq/hornetq-transports/2.0.0.GA/hornetq-transports-2.0.0.GA.jar:/Users/jim/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 com.gravity.insights.viral.HornetTest,testHornet
              javax.jms.JMSException: Timed out waiting for response when sending packet 40
              at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
              at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateConsumer(ClientSessionImpl.java:1556)

              javax.jms.JMSException: Timed out waiting for response when sending packet 40

              at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)

              at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateConsumer(ClientSessionImpl.java:1556)

              at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:447)

              at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:413)

              at org.hornetq.core.client.impl.DelegatingSession.createConsumer(DelegatingSession.java:187)

              at org.hornetq.jms.client.HornetQSession.createConsumer(HornetQSession.java:531)

               

              However, the JMS test client you posted doesn't make any calls to create consumer. So you must be running some other client.

               

              Also, FWIW any changes to flow control are currently disabled in TRUNK anyway.

              • 34. Re: What is the expected performance of the STOMP protocol?
                mrbeagle

                Below is notes for 3 scenarios in which we were able to put the server
                into a permanent bad state (multiple threads deadlocked and server no
                longer operating correctly--server does not shut down unless a kill signal
                is sent) with a combination of PHP via Stomp and Java via the HornetQ JMS
                client.  The problem seems to especially happen while Java is enqueuing
                quickly and PHP client is stopped and restarted simultaneously (an event
                that we need to support frequently in our environment).
                SCENARIO 1:
                Java client enqueues 300k items, php client dequeus 3000 items and then
                hangs.  java regularly sends a message count request.  At the same time
                that php hangs, message count starts to send back an error instead of the
                count.  On the server side, get the following exception dumped to the
                console:
                Meanwhile another thread is waiting on a commit:
                SCENARIO 2: Java Client enqueuing, PHP client dequeuing.  Everything goes
                well until the PHP client is stopped and restarted.  At that point the PHP
                client hangs and eventually the connection gets shut down.  Java client
                stops enqueuing and gets timeout exceptions when it tries to retrieve a
                message count.  Deadlock is detected below on the server side.  If I
                disconnect all the clients the deadlock below persists indefinitely.  If I
                reconnect a fresh Java client, I can enqueue, but neither PHP via Stomp
                nor Java via JMS can dequeue until I restart the server.  Each time I try
                to add a new Consumer a thread is fired up that looks just like the thread
                below that's waiting for a consumer.
                Java stack information for the threads listed above:
                ===================================================
                SCENARIO 3: Tried using NIO server.  Problem harder to repro with NIO
                server.  But still possible.  Java enqueuing and dequeuing.  PHP
                dequeueing.  Everything runs fine, but then stop and restart PHP.  PHP
                gets nothing, Java can no longer dequeue and gets periodic timeout
                notifications.  Below deadlock permanently exists on server until shutdown:
                Java stack information for the threads listed above:
                ===================================================

                Below is notes for 3 scenarios in which we were able to put the server

                into a permanent bad state (multiple threads deadlocked and server no

                longer operating correctly--server does not shut down unless a kill signal

                is sent) with a combination of PHP via Stomp and Java via the HornetQ JMS

                client.  The problem seems to especially happen while Java is enqueuing

                quickly and PHP client is stopped and restarted simultaneously (an event

                that we need to support frequently in our environment).

                 

                PHP client here: http://pastebin.com/Midk7nDC

                Java client here: http://pastebin.com/8wXaQabX

                 

                 

                SCENARIO 1:

                 

                Java client enqueues 300k items, php client dequeus 3000 items and then

                hangs.  java regularly sends a message count request.  At the same time

                that php hangs, message count starts to send back an error instead of the

                count.  On the server side, get the following exception dumped to the

                console:

                 

                http://pastebin.com/39dx87ks

                 

                 

                Meanwhile another thread is waiting on a commit:

                 

                http://pastebin.com/ccH1zhfh

                 

                 

                SCENARIO 2: Java Client enqueuing, PHP client dequeuing.  Everything goes

                well until the PHP client is stopped and restarted.  At that point the PHP

                client hangs and eventually the connection gets shut down.  Java client

                stops enqueuing and gets timeout exceptions when it tries to retrieve a

                message count.  Deadlock is detected below on the server side.  If I

                disconnect all the clients the deadlock below persists indefinitely.  If I

                reconnect a fresh Java client, I can enqueue, but neither PHP via Stomp

                nor Java via JMS can dequeue until I restart the server.  Each time I try

                to add a new Consumer a thread is fired up that looks just like the thread

                below that's waiting for a consumer.

                 

                http://pastebin.com/1uqtUJH8

                 

                 

                Java stack information for the threads listed above:

                ===================================================

                http://pastebin.com/S1VziGb3

                 

                 

                 

                SCENARIO 3: Tried using NIO server.  Problem harder to repro with NIO

                server.  But still possible.  Java enqueuing and dequeuing.  PHP

                dequeueing.  Everything runs fine, but then stop and restart PHP.  PHP

                gets nothing, Java can no longer dequeue and gets periodic timeout

                notifications.  Below deadlock permanently exists on server until shutdown:

                 

                http://pastebin.com/08dfWsXD

                 

                Java stack information for the threads listed above:

                ===================================================

                http://pastebin.com/zH47NvyD

                • 35. Re: What is the expected performance of the STOMP protocol?
                  timfox

                  You didn't mention you were calling getMessageCount previously. What happens when you don't?

                   

                  This thread is getting very confused.

                  • 36. Re: What is the expected performance of the STOMP protocol?
                    timfox

                    If these are different issues, start different threads or it's almost impossible to figure out what is going on, and what the latest set of actions you are doing is.

                    • 37. Re: What is the expected performance of the STOMP protocol?
                      timfox

                      There is a known issue with calling getMessageCount when server is under load, I'm just trying to eliminate that as a possibility.

                       

                      Anyhow, I'm out now, I'm sure someone can take a look next week.

                      • 38. Re: What is the expected performance of the STOMP protocol?
                        clebert.suconic

                        @Jimmy if you could please do as Tim said (removing the excessive calls to getCount()) and we will take a look depending on what you tell us.

                        1 2 3 Previous Next