6 Replies Latest reply on Nov 30, 2010 3:18 PM by clebert.suconic

    jms perf example application does not perform well

    stefan.b.

      I have just downloaded HornetQ-2.1.2.Final and tried to run the performance test in .../examples/jms/perf.

      The test has been done on Windows XP (32 bit), Java is 1.6.0_22.

       

      First of all, the JVM memory settings for the "runServer" target in build.xml had to be reduced from 2048M to 1024M to avoid a "Could not reserve enough space for object heap" error when trying to run the server.

       

      In perf.properties, I chose num-messages=4000 and num-warmup-messages=100; everything else is left at the default value from the binary distribution file.

       

      The final step is to start the server, the listener, and the sender in three command windows:

       

      build.bat runServer

      build.bat runListener

      build.bat runSender

       

      Sending and receiving these 4000 messages takes about three minutes!

      During this time the CPU is near idle.

       

      The exact results are:

       

      Listener: INFO: average: 22,94 msg/s (4000 messages in 174,39s)

      Sender: INFO: average: 24,11 msg/s (4000 messages in 165,94s)

       

      Although this is not a high-performance machine, these numbers are some magnitudes below my expectations.

      Any idea what did I do wrong?

       

      Stefan

        • 1. Re: jms perf example application does not perform well
          ataylor

          what figures were you expecting?

          • 2. Re: jms perf example application does not perform well
            clebert.suconic

            I just added two FAQs on the technical FAQ to answer this:  http://community.jboss.org/wiki/HornetQTechnicalFAQ

             

             

            I'm doing my benchmark and HornetQ doesn't perform well. Why is that?

            It's been quite common users performing their micro benchmarks without fully understanding how systems will achieve performance on enterprise systems.

             

            When you send persistent messages, you are required to perform a sync on the disk before you can deliver any messages or send a response back to the client. In certain cases the client will be waiting a network round trip also, so a single producer case will be limited by the sync capacity and round trip capacity of your hardware.

             

            HornetQ (as many other enterprise systems) will scale up to multiple clients performing multiple requests in a single request to disk and network, making a better usage of the hardware resource.

             

            So, for a proper benchmark you need to be aware of the behavior used by the use case you are dealing with.

             

            I don't intend to list all the details here, but just to give an overall idea:

             

            - Non Persistent Messages : Everything is asynchronous by default. This should be the fastest case

            - Persistent Messages: Both the send and the persistence are synchronous. The client will send a message, wait for a round trip and the sync on disk. You need several clients to measure overall server's performance on this case. (Scalability).

            - Transaction Messages: The messages are sent without any sync and totally asynchronous. However the commit or prepare will require a round trip.

             

            I'm comparing HornetQ to an XYZ software and the performance is bellow what's been advertised.

             

            It's also a common case users comparing oranges to bananas.

             

            HornetQ is strict as possible out of the box in regard to syncs, and we still provide excellent performance with these settings.

             

            Some messaging servers however will perform poorly when increasing sync requirements. They may appear faster when used their configs out of the box, what means you are comparing oranges to bananas.

             

             

            HornetQ also supports more optimistic syncs if that's what you need. Look at the user manual for the journal configuration options. (and on the ConnectionFactory for sync options on message sendin

            • 3. Re: jms perf example application does not perform well
              ataylor

              Since the messages are persistent then your performance is capped by

               

              1 your network, since sends will block a message will take as long as it takes to send 1024 bytes and receive the acknowledement

               

              and

               

              2 How fast your hard disc can sync.

              • 4. Re: jms perf example application does not perform well
                stefan.b.

                Clebert Suconic wrote:

                 

                - Persistent Messages: Both the send and the persistence are synchronous. The client will send a message, wait for a round trip and the sync on disk. You need several clients to measure overall server's performance on this case. (Scalability).

                This is a good point. I suppose that I get much better numbers when running multiple clients. I'll be on a trip tomorrow which means that I can't continue this work before thursday, so don't expect feedback too soon.

                 

                Andy Taylor asked what figures I was expecting.

                 

                Well, this site makes a lot of buzz about HornetQ's performance. There is a paper "JMS_Market_Throughput_Comparison.pdf" linked from this page: http://community.jboss.org/wiki/HornetQ-thePerformanceLeaderinEnterpriseMessaging

                A comparable setup ("G" in the paper) which uses persistent messages passes the test with about 9000 messages per second.

                 

                After reading the paper, I downloaded the software, found the small performance test tool that was included as a part of the package and ran it. I did not expect to see 9000 messages per second, but twenty-something looked pretty lame.

                 

                The major difference between the test setup used in the paper and the ad hoc test done by me is that I was just using a single client whereas the test described there used 40 clients. That matches Clebert's hint that it is required to run multiple clients to see the real strength of HornetQ. I am going to investigate this further in a few days.

                 

                A big thanks to both of you!

                 

                Stefan

                • 5. Re: jms perf example application does not perform well
                  clebert.suconic
                  The major difference between the test setup used in the paper and the ad hoc test done by me is that I was just using a single client whereas the test described there used 40 clients. That matches Clebert's hint that it is required to run multiple clients to see the real strength of HornetQ. I am going to investigate this further in a few days.

                  Yep, that's a common mistake, and we have answered it a few times but never added to the FAQ.

                   

                  Since you asked it now.. I made it to the FAQ :-)

                  • 6. Re: jms perf example application does not perform well
                    clebert.suconic

                    Also, look at the HornetQ's manual at the performance chapter. To guarantee proper failure resilience you need to disable buffering from your disk. If you keep buffering on you will be measuring the time the data takes to arrive at the buffer (OS / Hardware) and not necessarily to the actual disk.

                     

                    There are a few controllers that will have their buffer backed by a battery so their sync times are considerably faster.

                     

                     

                    Those are considerations we always take before making any benchmark, and if you do comparative benchmarks you have to make sure that you're using similar behaviors on each compared software.