6 Replies Latest reply on Apr 10, 2014 11:05 AM by jbertram

    ambiguous and unclear definition in Hornetq user manual

    vasco

      I ran into the following text while investigating a scaling problem in HornetQ:

      17.3. Configuring Asynchronous Connection Execution

       

      By default, packets received on the server side are executed on the remoting thread.

      It is possible instead to use a thread from a thread pool to handle some packets so that the remoting thread is not tied up for too long. However, please note that processing operations asynchronously on another thread adds a little more latency. Please note that most short running operations are always handled on the remoting thread for performance reasons. To enable asynchronous connection execution, set the parameterasync-connection-execution-enabled in hornetq-configuration.xml to true (default value is true).

      I'm not sure how to interpret this. The first statement is that by default packets are received are handled by the remoting thread but then later it states that this behavior can be changed by setting async-connection-execution to true, which is set to true by default. Apart from this contradiction, the statement about the short running operations confuses me even more i.e. how do we know upfront if something is either short running or not?

      The reason I am asking is, first off all to clarify the user manual, but also because in my load testing I can not get the broker to scale out vertically. I think this primarily because all the message filtering is done by only on thread and I'm looking for a way to utilize more of the available CPU power to do the filtering.

        • 1. Re: ambiguous and unclear definition in Hornetq user manual
          ataylor

          moved to correct forum

          • 2. Re: ambiguous and unclear definition in Hornetq user manual
            jbertram

            The first statement is that by default packets are received are handled by the remoting thread but then later it states that this behavior can be changed by setting async-connection-execution to true, which is set to true by default. Apart from this contradiction...

            I will update the documentation to be more clear.

             

            ...the statement about the short running operations confuses me even more i.e. how do we know upfront if something is either short running or not?

            Whether or not the operation is "short running" is determined by the type of packet it is.  These packets will be executed asynchronously if async-connection-execution is true:

            • RollbackMessage
            • SessionCloseMessage
            • SessionCommitMessage
            • SessionXACommitMessage
            • SessionXAPrepareMessage
            • SessionXARollbackMessage

             

            Every other packet will be executed on the remoting thread regardless of the setting for async-connection-execution.

             

            ...in my load testing I can not get the broker to scale out vertically. I think this primarily because all the message filtering is done by only on thread and I'm looking for a way to utilize more of the available CPU power to do the filtering.

            Couple of points/questions...

            • Your bottleneck may not be CPU.  Oftentimes the bottleneck is the disk.  Do you have specific evidence that indicates CPU is the bottleneck?  If so, what is it?
            • From what I understand the filtering should be done by more than one thread.  Do you have specific evidence that indicates only one thread is being used for filtering?  If so, what is it?
            • What exactly are you doing in your load test?
            • 3. Re: ambiguous and unclear definition in Hornetq user manual
              vasco

              Whether or not the operation is "short running" is determined by the type of packet it is.  These packets will be executed asynchronously if async-connection-execution is true:

              • RollbackMessage
              • SessionCloseMessage
              • SessionCommitMessage
              • SessionXACommitMessage
              • SessionXAPrepareMessage
              • SessionXARollbackMessage

              What about "normal" HornetQBytesMessage messages do they fall under one of these categories?

              Your bottleneck may not be CPU.  Oftentimes the bottleneck is the disk.  Do you have specific evidence that indicates CPU is the bottleneck?  If so, what is it?

               

               

              I don't think disk could be a bottle neck in our case since we are not using persistent messaging so I do no expect anything is written to disc.

              From what I understand the filtering should be done by more than one thread.  Do you have specific evidence that indicates only one thread is being used for filtering?  If so, what is it?

              I have profiled our running load test using a profiler (yourkit) and what I see is that the remoting thread is constantly bussy filtering  and other threads in the pool are sporadically active. I also introduced a breakpoint that prints the thread name every time the filter was evaluated and this was constantly the remoting-thread.

               

              What exactly are you doing in your load test?

               

              We are sending messages to a broker and measuring the time it takes for it to arrive at a client that subscribes using a message selector.

              • 4. Re: ambiguous and unclear definition in Hornetq user manual
                jbertram

                What about "normal" HornetQBytesMessage messages do they fall under one of these categories?

                You're confusing the JMS implementation with internal HornetQ network packets.  All the "messages" I enumerated previously are special "packets" which go over the wire (i.e. over the network) between the client and server to accomplish specific tasks (e.g. close a session, commit a session, rollback an XA transaction, etc.).  These messages are categorically different from anything which implements javax.jms.Message.

                 

                We are sending messages to a broker and measuring the time it takes for it to arrive at a client that subscribes using a message selector.

                • How many producers do you have?
                • How many consumers do you have?
                • At what rate are you sending the messages?
                • Have you investigated direct-deliver?
                • 5. Re: ambiguous and unclear definition in Hornetq user manual
                  vasco
                  How many producers do you have?

                  36

                  How many consumers do you have?

                  36 creating a total 1300 subscriptions.

                   

                  At what rate are you sending the messages?

                  750 msg/sec

                  Have you investigated direct-deliver?

                  We are using direct delivery.

                   

                  I don't think this is relevant for the fact the the broker filters message in one thread right?

                  • 6. Re: ambiguous and unclear definition in Hornetq user manual
                    jbertram

                    I don't think this is relevant for the fact the the broker filters message in one thread right?

                    Of course it is relevant.  Understanding the overall use-case is critical when investigating issues like this.

                     

                    We are using direct delivery.

                    Do you mean that you're using the default value for direct-deliver (i.e. true)?  If so, I would suggest setting it to "false" since you're trying to maximize throughput.

                     

                     

                    In any case, the way I read the code indicates to me that the filtering will be done on the remoting thread servicing the connection from the client.  There doesn't appear to be anything inherently single-threaded about it so at this point it looks like there's something about your use-case that is triggering this behavior.  Do you have any kind of test you could share which demonstrates this behavior?