1 2 3 Previous Next 38 Replies Latest reply: Oct 11, 2010 11:18 AM by Clebert Suconic Go to original post RSS
  • 30. Re: What is the expected performance of the STOMP protocol?
    Jimmy Beagle Newbie

    since I'm dequeing from stomp, should there still be a concept of window size?

     

     

    Here are my test scripts

     

    enqueue from java

     

    http://pastebin.com/Pw4Uc2GU

     

     

    php deqeue

     

    http://pastebin.com/PNUEGdGS

     

     

    I enqueue a  few thousand rows from java using jms, then run a script in php to dequeue. After between 700-1000 dequeues it locks up the php script and the hornet server itself becomes unresponsive. If I re-run my JMS enqueue test case during this lock up the enqueues time out with the exception I pasted above.

  • 31. Re: What is the expected performance of the STOMP protocol?
    Clebert Suconic Master

    I will have to take a look on early next week.

     

     

    I will have to look at the changes Tim made over for this.

  • 32. Re: What is the expected performance of the STOMP protocol?
    Tim Fox Master

    Works fine for me using TRUNK. Here are the last few lines of output from dequeue on my laptop:

     

    MSG: This is a test message: 2984
    MSG: This is a test message: 2985
    MSG: This is a test message: 2986
    MSG: This is a test message: 2987
    MSG: This is a test message: 2988
    MSG: This is a test message: 2989
    MSG: This is a test message: 2990
    MSG: This is a test message: 2991
    MSG: This is a test message: 2992
    MSG: This is a test message: 2993
    MSG: This is a test message: 2994
    MSG: This is a test message: 2995
    MSG: This is a test message: 2996
    MSG: This is a test message: 2997
    MSG: This is a test message: 2998
    MSG: This is a test message: 2999
    3000 messages dequeuedMSG: This is a test message: 3000
    1.1247551441193 seconds

     

    I ran your exact JMS client followed by your php script.

  • 33. Re: What is the expected performance of the STOMP protocol?
    Tim Fox Master

    This doesn't follow. The stack trace you posted shows the client waiting in a call to create consumer:

     

    /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java -Xms2048m -Xmx2048m -Dfile.encoding=UTF-8 -classpath /Applications/IntelliJ IDEA 9.0.3.app/lib/idea_rt.jar:/Applications/IntelliJ IDEA 9.0.3.app/plugins/junit/lib/junit-rt.jar:/System/Library/Frameworks/JavaVM.framework/Versions/A/Resources/Deploy.bundle/Contents/Resources/Java/deploy.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/dt.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/javaws.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/jce.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/management-agent.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/plugin.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/sa-jdi.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/alt-rt.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/charsets.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/classes.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/jconsole.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/jsse.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/ui.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/apple_provider.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/dnsns.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/localedata.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/sunjce_provider.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/sunpkcs11.jar:/Users/jim/Code/InsightsRaven/viral/target/test-classes:/Users/jim/Code/InsightsRaven/viral/target/classes:/Users/jim/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/Users/jim/.m2/repository/org/jsoup/jsoup/1.2.2/jsoup-1.2.2.jar:/Users/jim/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar:/Users/jim/.m2/repository/com/jolbox/bonecp/0.7.0-rc3/bonecp-0.7.0-rc3.jar:/Users/jim/.m2/repository/com/google/collections/google-collections/1.0/google-collections-1.0.jar:/Users/jim/.m2/repository/org/slf4j/slf4j-api/1.5.10/slf4j-api-1.5.10.jar:/Users/jim/Code/InsightsRaven/insights-utils/target/classes:/Users/jim/.m2/repository/org/apache/thrift/libthrift/0.2.0/libthrift-0.2.0.jar:/Users/jim/.m2/repository/org/apache/httpcomponents/httpclient/4.0.1/httpclient-4.0.1.jar:/Users/jim/.m2/repository/org/apache/httpcomponents/httpcore/4.0.1/httpcore-4.0.1.jar:/Users/jim/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/jim/.m2/repository/commons-codec/commons-codec/1.3/commons-codec-1.3.jar:/Users/jim/.m2/repository/com/gravity/utilities/1.0-SNAPSHOT/utilities-1.0-SNAPSHOT.jar:/Users/jim/.m2/repository/org/slf4j/slf4j-log4j12/1.5.10/slf4j-log4j12-1.5.10.jar:/Users/jim/.m2/repository/log4j/log4j/1.2.14/log4j-1.2.14.jar:/Users/jim/.m2/repository/org/jdom/jdom/1.1/jdom-1.1.jar:/Users/jim/.m2/repository/com/google/code/gson/gson/1.4/gson-1.4.jar:/Users/jim/.m2/repository/org/jredis/jredis/1.0-rc1/jredis-1.0-rc1.jar:/Users/jim/.m2/repository/org/mongodb/mongo-java-driver/2.0/mongo-java-driver-2.0.jar:/Users/jim/.m2/repository/mysql/mysql-connector-java/5.1.12/mysql-connector-java-5.1.12.jar:/Users/jim/.m2/repository/javax/jms/jms/1.1/jms-1.1.jar:/Users/jim/.m2/repository/org/hornetq/hornetq-jms-client/2.1.2.Final/hornetq-jms-client-2.1.2.Final.jar:/Users/jim/.m2/repository/org/hornetq/hornetq-core-client/2.1.2.Final/hornetq-core-client-2.1.2.Final.jar:/Users/jim/.m2/repository/org/hornetq/hornetq-transports/2.0.0.GA/hornetq-transports-2.0.0.GA.jar:/Users/jim/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 com.gravity.insights.viral.HornetTest,testHornet
    javax.jms.JMSException: Timed out waiting for response when sending packet 40
    at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
    at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateConsumer(ClientSessionImpl.java:1556)

    javax.jms.JMSException: Timed out waiting for response when sending packet 40

    at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)

    at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateConsumer(ClientSessionImpl.java:1556)

    at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:447)

    at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:413)

    at org.hornetq.core.client.impl.DelegatingSession.createConsumer(DelegatingSession.java:187)

    at org.hornetq.jms.client.HornetQSession.createConsumer(HornetQSession.java:531)

     

    However, the JMS test client you posted doesn't make any calls to create consumer. So you must be running some other client.

     

    Also, FWIW any changes to flow control are currently disabled in TRUNK anyway.

  • 34. Re: What is the expected performance of the STOMP protocol?
    Jimmy Beagle Newbie

    Below is notes for 3 scenarios in which we were able to put the server
    into a permanent bad state (multiple threads deadlocked and server no
    longer operating correctly--server does not shut down unless a kill signal
    is sent) with a combination of PHP via Stomp and Java via the HornetQ JMS
    client.  The problem seems to especially happen while Java is enqueuing
    quickly and PHP client is stopped and restarted simultaneously (an event
    that we need to support frequently in our environment).
    SCENARIO 1:
    Java client enqueues 300k items, php client dequeus 3000 items and then
    hangs.  java regularly sends a message count request.  At the same time
    that php hangs, message count starts to send back an error instead of the
    count.  On the server side, get the following exception dumped to the
    console:
    Meanwhile another thread is waiting on a commit:
    SCENARIO 2: Java Client enqueuing, PHP client dequeuing.  Everything goes
    well until the PHP client is stopped and restarted.  At that point the PHP
    client hangs and eventually the connection gets shut down.  Java client
    stops enqueuing and gets timeout exceptions when it tries to retrieve a
    message count.  Deadlock is detected below on the server side.  If I
    disconnect all the clients the deadlock below persists indefinitely.  If I
    reconnect a fresh Java client, I can enqueue, but neither PHP via Stomp
    nor Java via JMS can dequeue until I restart the server.  Each time I try
    to add a new Consumer a thread is fired up that looks just like the thread
    below that's waiting for a consumer.
    Java stack information for the threads listed above:
    ===================================================
    SCENARIO 3: Tried using NIO server.  Problem harder to repro with NIO
    server.  But still possible.  Java enqueuing and dequeuing.  PHP
    dequeueing.  Everything runs fine, but then stop and restart PHP.  PHP
    gets nothing, Java can no longer dequeue and gets periodic timeout
    notifications.  Below deadlock permanently exists on server until shutdown:
    Java stack information for the threads listed above:
    ===================================================

    Below is notes for 3 scenarios in which we were able to put the server

    into a permanent bad state (multiple threads deadlocked and server no

    longer operating correctly--server does not shut down unless a kill signal

    is sent) with a combination of PHP via Stomp and Java via the HornetQ JMS

    client.  The problem seems to especially happen while Java is enqueuing

    quickly and PHP client is stopped and restarted simultaneously (an event

    that we need to support frequently in our environment).

     

    PHP client here: http://pastebin.com/Midk7nDC

    Java client here: http://pastebin.com/8wXaQabX

     

     

    SCENARIO 1:

     

    Java client enqueues 300k items, php client dequeus 3000 items and then

    hangs.  java regularly sends a message count request.  At the same time

    that php hangs, message count starts to send back an error instead of the

    count.  On the server side, get the following exception dumped to the

    console:

     

    http://pastebin.com/39dx87ks

     

     

    Meanwhile another thread is waiting on a commit:

     

    http://pastebin.com/ccH1zhfh

     

     

    SCENARIO 2: Java Client enqueuing, PHP client dequeuing.  Everything goes

    well until the PHP client is stopped and restarted.  At that point the PHP

    client hangs and eventually the connection gets shut down.  Java client

    stops enqueuing and gets timeout exceptions when it tries to retrieve a

    message count.  Deadlock is detected below on the server side.  If I

    disconnect all the clients the deadlock below persists indefinitely.  If I

    reconnect a fresh Java client, I can enqueue, but neither PHP via Stomp

    nor Java via JMS can dequeue until I restart the server.  Each time I try

    to add a new Consumer a thread is fired up that looks just like the thread

    below that's waiting for a consumer.

     

    http://pastebin.com/1uqtUJH8

     

     

    Java stack information for the threads listed above:

    ===================================================

    http://pastebin.com/S1VziGb3

     

     

     

    SCENARIO 3: Tried using NIO server.  Problem harder to repro with NIO

    server.  But still possible.  Java enqueuing and dequeuing.  PHP

    dequeueing.  Everything runs fine, but then stop and restart PHP.  PHP

    gets nothing, Java can no longer dequeue and gets periodic timeout

    notifications.  Below deadlock permanently exists on server until shutdown:

     

    http://pastebin.com/08dfWsXD

     

    Java stack information for the threads listed above:

    ===================================================

    http://pastebin.com/zH47NvyD

  • 35. Re: What is the expected performance of the STOMP protocol?
    Tim Fox Master

    You didn't mention you were calling getMessageCount previously. What happens when you don't?

     

    This thread is getting very confused.

  • 36. Re: What is the expected performance of the STOMP protocol?
    Tim Fox Master

    If these are different issues, start different threads or it's almost impossible to figure out what is going on, and what the latest set of actions you are doing is.

  • 37. Re: What is the expected performance of the STOMP protocol?
    Tim Fox Master

    There is a known issue with calling getMessageCount when server is under load, I'm just trying to eliminate that as a possibility.

     

    Anyhow, I'm out now, I'm sure someone can take a look next week.

  • 38. Re: What is the expected performance of the STOMP protocol?
    Clebert Suconic Master

    @Jimmy if you could please do as Tim said (removing the excessive calls to getCount()) and we will take a look depending on what you tell us.

1 2 3 Previous Next