1 2 3 Previous Next 36 Replies Latest reply on Mar 22, 2012 8:04 AM by manik

    InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance

    ltepper

      Hi,

       

      I have prepared a test to check infiniSpan performance using a replicated cache on two hosts. (A passive and an active one - please see the attached eclipse project)

      In the test I was using 60 threads, each one running total amount of 100,000 transactions => total of 6 million transactions.

      Each transaction included:  1 put to cache, 10 get requests and 1 remove from cache.

      Each object that was inserted to cahce included 10 ints and 20 Strings.

       

      The following results have shown preformace of 13 transactions per millisecond.

       

       

      Test result log:

      ***************************************************************************************************************************************************************************************************

      2012-02-28 19:35:56,084 INFO [Thread-44] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.

      2012-02-28 19:35:56,085 INFO [Thread-44] infinispan.test.Transaction - Run time is 428967 milliseconds.

      2012-02-28 19:35:56,085 INFO [Thread-44] infinispan.test.Transaction - Amount of transactions in one millisecond is 13.0

      ***************************************************************************************************************************************************************************************************

       

      Comparing to ehCache, these results were quite poor since with ehcahce I got to 275 transactions per second.

       

      My questions is - are these results seems reasonable or was something configured wrongly in my test? (e.g - in my jgroups.xml or in infispan.xml)

       

      Any help would be appriciated.

       

      Regards,

      Liron Tepper.

        • 1. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
          sannegrinovero

          Hi Liron,

           

          indeed during the early 5.1 CRs we spotted some severe performance problems; the performance issues where all fixed before the Final, and we kept improving performance even more every day, so the fastest should be version 5.1.2.CR2 (released yesterday).

           

          Would you mind trying an update? Please update both Infinispan and JGroups.

          • 2. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
            sannegrinovero

            Another thing: if your test runs only 6 million transactions it will finish way before the JVM has warmed up properly: on an average 4 cores Xeons it can peak above the million read transactions per second, but it takes at least 20 minutes of high load before it gets to that. (will be slower for writes: with replication read is always a local lookup).

             

            A comment on your code: I don't see you starting transactions? Consider Infinispan understands "real" JTA transactions, so you could get even better performance wrapping your startTransaction  method in a transaction, as it will batch all update operations across a reduced amount of wire commands.

            • 3. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
              sannegrinovero

              btw doing 13 transactions per millisecond means 13,000 transactions per second.

              I assume you have a typo, as you say ehcache is doing just 275 transactions per second.. looks like too good for Infinispan to be true.

              • 4. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                ltepper

                Hi Sanne,

                 

                First, thank you very much for your replies.

                 

                I have a typo indeed - I meant to write 275 transactions per millisecond in ehcach.

                I will try infiniSpan version 5.1.2.CR2 with the latest JGroups version.

                Also I will try to use real transactions, run a long test over night and post results.

                 

                Regards,

                Liron Tepper.

                • 5. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                  sannegrinovero

                  thanks! Looking forward to see them. We can always improve something

                  • 6. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                    ltepper

                    Here are the new results I got:

                     

                    1) Before placing any new JAR, a long 60 million transactions run:

                     

                    Logs:

                    **********************************************************************************************************************************************************************************************

                    2012-02-29 11:37:17,840 INFO [Thread-15] infinispan.test.Transaction - 60 threads run 1000000 transactions per thread => total of 60000000 transactions.

                    2012-02-29 11:37:17,840 INFO [Thread-15] infinispan.test.Transaction - Run time is 4954327 milliseconds.

                    2012-02-29 11:37:17,840 INFO [Thread-15] infinispan.test.Transaction - Amount of transactions in one millisecond is 12.0

                    **********************************************************************************************************************************************************************************************

                    => As you can see from logs above, a long run of 82 minutes showed a very slight performance descending from 13 transactions per millisecond to 12.

                     

                     

                    2) With new JARs  (infiniSpan version 5.1.2.CR1 with jgroups-3.0.6.Final.jar) and a short 6 Million transactions run:

                     

                    Logs:

                    **********************************************************************************************************************************************************************************************

                    2012-02-29 14:16:02,832 INFO [Thread-48] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.

                    2012-02-29 14:16:02,832 INFO [Thread-48] infinispan.test.Transaction - Run time is 382628 milliseconds.

                    2012-02-29 14:16:02,833 INFO [Thread-48] infinispan.test.Transaction - Average transaction time is 0.0

                    2012-02-29 14:16:02,833 INFO [Thread-48] infinispan.test.Transaction - Amount of transactions in one millisecond is 15.0

                      **********************************************************************************************************************************************************************************************

                     

                    => Using new jars brought us to 15 transactions per second, which is a slight improvement only.

                     

                    3) With real transactions enabled, and a short 6 Million transactions run (and the new JARs) -

                     

                      **********************************************************************************************************************************************************************************************

                    2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.
                    2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - Run time is 425982 milliseconds.
                    2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - Average transaction time is 0.0
                    2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - Amount of transactions in one millisecond is 14.0

                      **********************************************************************************************************************************************************************************************

                    => There wasn't any performance change in this test.

                     

                    The code changes I've made are as followes:

                     

                    ============================================================================================================================

                    private void startTransaction(Integer index) {
                      MyObject myObject = MyObjectFactory.getMyObject();

                      boolean batchStarted = RunLoadTest.testCache.startBatch();
                     
                      RunLoadTest.testCache.get(index);
                      RunLoadTest.testCache.get(index);
                    //  RunLoadTest.testCache.putAsync(index, myObject);
                      RunLoadTest.testCache.put(index, myObject);  //sync !
                      RunLoadTest.testCache.get(index);
                      RunLoadTest.testCache.get(index);
                      RunLoadTest.testCache.get(index);
                      RunLoadTest.testCache.get(index);
                    //  RunLoadTest.testCache.removeAsync(index);
                      RunLoadTest.testCache.remove(index);  // sync !
                      RunLoadTest.testCache.get(index);
                      RunLoadTest.testCache.get(index);
                      RunLoadTest.testCache.get(index);
                      RunLoadTest.testCache.get(index);
                     
                      RunLoadTest.testCache.endBatch(batchStarted);
                      if (!batchStarted) {
                       LogUtils.warn(getClass(), "Batch was not started !");
                      }
                    }

                    ============================================================================================================================

                     

                    And also I've added the following to infispan.xml:

                     

                    ============================================================================================================================

                    <default>

                     


                     

                    <invocationBatching enabled="true" />

                    </default>

                    ============================================================================================================================

                     

                     

                    Any other ideas for improving this would be much appreciated !

                     

                    Regards,

                    Liron Tepper

                    • 7. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                      manik

                      I haven't had a look at your tests as yet, but can you confirm that in both cases you are getting synchronous, 2-phase transactions?  This is the default in Infinispan and I know that it is *not* the default in EHCache.  Similarly, EHCache defaults to asynchronous serialization which again isn't the default in Infinispan.  Just making sure you're doing an apples-to-apples comparison. 

                      • 8. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                        ltepper

                        Hi Manik,

                         

                        This is the configuration I'm using for ehCache:

                         

                        ******************************************************************************************************************************************************

                        <?xml version="1.0" encoding="UTF-8"?>

                        <ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                 xsi:noNamespaceSchemaLocation="ehcache.xsd"
                                 updateCheck="true" monitoring="autodetect"
                                 dynamicConfig="true">

                           
                            <diskStore path="java.io.tmpdir"/>

                            <transactionManagerLookup class="net.sf.ehcache.transaction.manager.DefaultTransactionManagerLookup"
                                                      properties="jndiName=java:/TransactionManager" propertySeparator=";"/>

                            <cacheManagerEventListenerFactory class="" properties=""/>

                        <!-- Liron: This part is added for working on RMI replication mode -->
                                
                        <cacheManagerPeerProviderFactory
                        class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
                        properties="peerDiscovery=manual,
                              rmiUrls=//server2:40001/allotIpSessionCache"/>

                        <cacheManagerPeerListenerFactory
                        class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
                        properties="port=40001,                   
                        socketTimeoutMillis=2000"/>

                        <!-- Liron: End of section. -->


                        <cache name="allotIpSessionCache" maxElementsInMemory="0"
                          eternal="true" overflowToDisk="true" diskPersistent="false"
                          memoryStoreEvictionPolicy="LFU">
                          <cacheEventListenerFactory
                           class="net.sf.ehcache.distribution.RMICacheReplicatorFactory" />
                          <bootstrapCacheLoaderFactory
                           class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory"
                           properties="bootstrapAsynchronously=false" />
                        </cache>

                        </ehcache>

                        ******************************************************************************************************************************************************

                         

                        With infiniSpan, if I set asyncMarshalling to true, I'm getting in a few seconds to a situation in which the infiniSpan internal RMI queue gets exploded.

                        Also documentation says "it can suffer from reordering of operations", so I can't use it.

                         

                        What is the default for transactions with ehCache? And in which way does infiniSpan differs from that?  When I used transaction with infiniSpan I didn't see any improvment,

                        and anyway I know that in our production process we will not be able to use transactions.

                         

                        Regards,

                        Liron Tepper

                        • 9. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                          ltepper

                          Correction - I meant RPC queues, not RMI queues.

                          These are the excptions I got:

                           

                          ***************************************************************************************************************************************************************************************************************************************

                          2012-02-28 19:00:20,033 ERROR [Thread-13] org.infinispan.remoting.rpc.RpcManagerImpl - ISPN000073: Unexpected error while replicating
                          java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@41e0c4c4 rejected from java.util.concurrent.ThreadPoolExecutor@4bfb03f0[Running, pool size = 25, active threads = 25, queued tasks = 100000, completed tasks = 4992]
                                  at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2001) ~[na:1.7.0_02]
                                  at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816) ~[na:1.7.0_02]
                                  at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337) ~[na:1.7.0_02]
                                  at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ~[na:1.7.0_02]
                                  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:104) ~[infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:442) ~[infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:153) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:177) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:222) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:209) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:192) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:185) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:188) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:149) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:190) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:133) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:68) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:201) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.StateTransferLockInterceptor.visitPutKeyValueCommand(StateTransferLockInterceptor.java:131) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:108) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:934) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.CacheImpl.put(CacheImpl.java:647) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.CacheImpl.put(CacheImpl.java:639) [infinispan-core.jar:5.1.0.CR2]
                                  at org.infinispan.CacheSupport.put(CacheSupport.java:52) [infinispan-core.jar:5.1.0.CR2]
                                  at infinispan.test.Transaction.startTransaction(Transaction.java:57) [classes/:na]
                                  at infinispan.test.Transaction.run(Transaction.java:26) [classes/:na]
                                  at java.lang.Thread.run(Thread.java:722) [na:1.7.0_02]
                                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
                                  at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:116)
                                  at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
                                  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61)
                                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                                  at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
                                  at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:934)
                                  at org.infinispan.CacheImpl.put(CacheImpl.java:647)
                                  at org.infinispan.CacheImpl.put(CacheImpl.java:639)
                                  at org.infinispan.CacheSupport.put(CacheSupport.java:52)
                                  at infinispan.test.Transaction.startTransaction(Transaction.java:57)
                                  at infinispan.test.Transaction.run(Transaction.java:26)
                                  at java.lang.Thread.run(Thread.java:722)

                          ***************************************************************************************************************************************************************************************************************************************

                          • 10. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                            galder.zamarreno

                            @Liron, in your <global> section, add the following and try again:

                             

                                  <asyncTransportExecutor factory="org.infinispan.executors.DefaultExecutorFactory">

                                     <properties>

                                        <property name="maxThreads" value="200"/>

                                        <property name="threadNamePrefix" value="AsyncSerializationThread"/>

                                     </properties>

                                  </asyncTransportExecutor>

                             

                            Basically, increase the async transport (or async marshalling) thread pool cos you're max'ing it out.

                            • 11. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                              galder.zamarreno

                              By the way, we're working on coming up with better executor defaults so that Infinispan is easier to consume out of the box.

                              • 12. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                                ltepper

                                I've added the asyncTransportExecutor section that you've mentioned, and set my cache to

                                <

                                async asyncMarshalling="true"

                                />

                                 

                                Still, I'm getting this error:

                                ***************************************************************************************************************************************************************************************************************************************

                                2012-03-01 13:14:24,967 ERROR [Thread-52] o.i.interceptors.InvocationContextInterceptor - ISPN000136: Execution error
                                org.infinispan.CacheException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f74d9e8 rejected from java.util.concurrent.ThreadPoolExecutor@4120bda[Running, pool size = 200, active threads = 200, queued tasks = 100000, completed tasks = 7565]

                                ***************************************************************************************************************************************************************************************************************************************

                                 

                                Also I've tried it also with

                                <async asyncMarshalling="true"/>

                                 

                                And got the following result:

                                ***************************************************************************************************************************************************************************************************************************************

                                2012-03-01 13:24:50,434 INFO [Thread-43] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.
                                2012-03-01 13:24:50,434 INFO [Thread-43] infinispan.test.Transaction - Run time is 393540 milliseconds.

                                 

                                2012-03-01 13:24:50,434 INFO [Thread-43] infinispan.test.Transaction - Amount of transactions in one millisecond is 15.0

                                ***************************************************************************************************************************************************************************************************************************************

                                Which is a very slight performance improvement.

                                Regards,

                                Liron Tepper

                                • 13. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                                  galder.zamarreno

                                  Liron, I dunno what's the difference between the first and the second option.

                                   

                                  Would ot be possible for you to send us the ehcache version of the test so that we can examine it and see what's being compared exactly?

                                  • 14. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
                                    galder.zamarreno

                                    Also Liron, you could be having network issues, so I'd suggest you try the TCP version of JGroups configuration:

                                    https://github.com/infinispan/infinispan/blob/5.1.x/core/src/main/resources/jgroups-tcp.xml

                                     

                                    MPING should discover your nodes using multicast. When you run the test, simply pass -Djgroups.bind_addr=x.x.x.x to each of the machines and that way you don't need to modify the config.

                                     

                                    I've had them myself in the past...

                                    1 2 3 Previous Next