1 2 3 Previous Next 36 Replies Latest reply: Mar 22, 2012 8:04 AM by Manik Surtani RSS

InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance

Liron Tepper Newbie

Hi,

 

I have prepared a test to check infiniSpan performance using a replicated cache on two hosts. (A passive and an active one - please see the attached eclipse project)

In the test I was using 60 threads, each one running total amount of 100,000 transactions => total of 6 million transactions.

Each transaction included:  1 put to cache, 10 get requests and 1 remove from cache.

Each object that was inserted to cahce included 10 ints and 20 Strings.

 

The following results have shown preformace of 13 transactions per millisecond.

 

 

Test result log:

***************************************************************************************************************************************************************************************************

2012-02-28 19:35:56,084 INFO [Thread-44] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.

2012-02-28 19:35:56,085 INFO [Thread-44] infinispan.test.Transaction - Run time is 428967 milliseconds.

2012-02-28 19:35:56,085 INFO [Thread-44] infinispan.test.Transaction - Amount of transactions in one millisecond is 13.0

***************************************************************************************************************************************************************************************************

 

Comparing to ehCache, these results were quite poor since with ehcahce I got to 275 transactions per second.

 

My questions is - are these results seems reasonable or was something configured wrongly in my test? (e.g - in my jgroups.xml or in infispan.xml)

 

Any help would be appriciated.

 

Regards,

Liron Tepper.

  • 1. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Sanne Grinovero Master

    Hi Liron,

     

    indeed during the early 5.1 CRs we spotted some severe performance problems; the performance issues where all fixed before the Final, and we kept improving performance even more every day, so the fastest should be version 5.1.2.CR2 (released yesterday).

     

    Would you mind trying an update? Please update both Infinispan and JGroups.

  • 2. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Sanne Grinovero Master

    Another thing: if your test runs only 6 million transactions it will finish way before the JVM has warmed up properly: on an average 4 cores Xeons it can peak above the million read transactions per second, but it takes at least 20 minutes of high load before it gets to that. (will be slower for writes: with replication read is always a local lookup).

     

    A comment on your code: I don't see you starting transactions? Consider Infinispan understands "real" JTA transactions, so you could get even better performance wrapping your startTransaction  method in a transaction, as it will batch all update operations across a reduced amount of wire commands.

  • 3. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Sanne Grinovero Master

    btw doing 13 transactions per millisecond means 13,000 transactions per second.

    I assume you have a typo, as you say ehcache is doing just 275 transactions per second.. looks like too good for Infinispan to be true.

  • 4. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Liron Tepper Newbie

    Hi Sanne,

     

    First, thank you very much for your replies.

     

    I have a typo indeed - I meant to write 275 transactions per millisecond in ehcach.

    I will try infiniSpan version 5.1.2.CR2 with the latest JGroups version.

    Also I will try to use real transactions, run a long test over night and post results.

     

    Regards,

    Liron Tepper.

  • 5. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Sanne Grinovero Master

    thanks! Looking forward to see them. We can always improve something

  • 6. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Liron Tepper Newbie

    Here are the new results I got:

     

    1) Before placing any new JAR, a long 60 million transactions run:

     

    Logs:

    **********************************************************************************************************************************************************************************************

    2012-02-29 11:37:17,840 INFO [Thread-15] infinispan.test.Transaction - 60 threads run 1000000 transactions per thread => total of 60000000 transactions.

    2012-02-29 11:37:17,840 INFO [Thread-15] infinispan.test.Transaction - Run time is 4954327 milliseconds.

    2012-02-29 11:37:17,840 INFO [Thread-15] infinispan.test.Transaction - Amount of transactions in one millisecond is 12.0

    **********************************************************************************************************************************************************************************************

    => As you can see from logs above, a long run of 82 minutes showed a very slight performance descending from 13 transactions per millisecond to 12.

     

     

    2) With new JARs  (infiniSpan version 5.1.2.CR1 with jgroups-3.0.6.Final.jar) and a short 6 Million transactions run:

     

    Logs:

    **********************************************************************************************************************************************************************************************

    2012-02-29 14:16:02,832 INFO [Thread-48] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.

    2012-02-29 14:16:02,832 INFO [Thread-48] infinispan.test.Transaction - Run time is 382628 milliseconds.

    2012-02-29 14:16:02,833 INFO [Thread-48] infinispan.test.Transaction - Average transaction time is 0.0

    2012-02-29 14:16:02,833 INFO [Thread-48] infinispan.test.Transaction - Amount of transactions in one millisecond is 15.0

      **********************************************************************************************************************************************************************************************

     

    => Using new jars brought us to 15 transactions per second, which is a slight improvement only.

     

    3) With real transactions enabled, and a short 6 Million transactions run (and the new JARs) -

     

      **********************************************************************************************************************************************************************************************

    2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.
    2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - Run time is 425982 milliseconds.
    2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - Average transaction time is 0.0
    2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - Amount of transactions in one millisecond is 14.0

      **********************************************************************************************************************************************************************************************

    => There wasn't any performance change in this test.

     

    The code changes I've made are as followes:

     

    ============================================================================================================================

    private void startTransaction(Integer index) {
      MyObject myObject = MyObjectFactory.getMyObject();

      boolean batchStarted = RunLoadTest.testCache.startBatch();
     
      RunLoadTest.testCache.get(index);
      RunLoadTest.testCache.get(index);
    //  RunLoadTest.testCache.putAsync(index, myObject);
      RunLoadTest.testCache.put(index, myObject);  //sync !
      RunLoadTest.testCache.get(index);
      RunLoadTest.testCache.get(index);
      RunLoadTest.testCache.get(index);
      RunLoadTest.testCache.get(index);
    //  RunLoadTest.testCache.removeAsync(index);
      RunLoadTest.testCache.remove(index);  // sync !
      RunLoadTest.testCache.get(index);
      RunLoadTest.testCache.get(index);
      RunLoadTest.testCache.get(index);
      RunLoadTest.testCache.get(index);
     
      RunLoadTest.testCache.endBatch(batchStarted);
      if (!batchStarted) {
       LogUtils.warn(getClass(), "Batch was not started !");
      }
    }

    ============================================================================================================================

     

    And also I've added the following to infispan.xml:

     

    ============================================================================================================================

    <default>

     


     

    <invocationBatching enabled="true" />

    </default>

    ============================================================================================================================

     

     

    Any other ideas for improving this would be much appreciated !

     

    Regards,

    Liron Tepper

  • 7. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Manik Surtani Master

    I haven't had a look at your tests as yet, but can you confirm that in both cases you are getting synchronous, 2-phase transactions?  This is the default in Infinispan and I know that it is *not* the default in EHCache.  Similarly, EHCache defaults to asynchronous serialization which again isn't the default in Infinispan.  Just making sure you're doing an apples-to-apples comparison. 

  • 8. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Liron Tepper Newbie

    Hi Manik,

     

    This is the configuration I'm using for ehCache:

     

    ******************************************************************************************************************************************************

    <?xml version="1.0" encoding="UTF-8"?>

    <ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:noNamespaceSchemaLocation="ehcache.xsd"
             updateCheck="true" monitoring="autodetect"
             dynamicConfig="true">

       
        <diskStore path="java.io.tmpdir"/>

        <transactionManagerLookup class="net.sf.ehcache.transaction.manager.DefaultTransactionManagerLookup"
                                  properties="jndiName=java:/TransactionManager" propertySeparator=";"/>

        <cacheManagerEventListenerFactory class="" properties=""/>

    <!-- Liron: This part is added for working on RMI replication mode -->
            
    <cacheManagerPeerProviderFactory
    class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
    properties="peerDiscovery=manual,
          rmiUrls=//server2:40001/allotIpSessionCache"/>

    <cacheManagerPeerListenerFactory
    class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
    properties="port=40001,                   
    socketTimeoutMillis=2000"/>

    <!-- Liron: End of section. -->


    <cache name="allotIpSessionCache" maxElementsInMemory="0"
      eternal="true" overflowToDisk="true" diskPersistent="false"
      memoryStoreEvictionPolicy="LFU">
      <cacheEventListenerFactory
       class="net.sf.ehcache.distribution.RMICacheReplicatorFactory" />
      <bootstrapCacheLoaderFactory
       class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory"
       properties="bootstrapAsynchronously=false" />
    </cache>

    </ehcache>

    ******************************************************************************************************************************************************

     

    With infiniSpan, if I set asyncMarshalling to true, I'm getting in a few seconds to a situation in which the infiniSpan internal RMI queue gets exploded.

    Also documentation says "it can suffer from reordering of operations", so I can't use it.

     

    What is the default for transactions with ehCache? And in which way does infiniSpan differs from that?  When I used transaction with infiniSpan I didn't see any improvment,

    and anyway I know that in our production process we will not be able to use transactions.

     

    Regards,

    Liron Tepper

  • 9. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Liron Tepper Newbie

    Correction - I meant RPC queues, not RMI queues.

    These are the excptions I got:

     

    ***************************************************************************************************************************************************************************************************************************************

    2012-02-28 19:00:20,033 ERROR [Thread-13] org.infinispan.remoting.rpc.RpcManagerImpl - ISPN000073: Unexpected error while replicating
    java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@41e0c4c4 rejected from java.util.concurrent.ThreadPoolExecutor@4bfb03f0[Running, pool size = 25, active threads = 25, queued tasks = 100000, completed tasks = 4992]
            at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2001) ~[na:1.7.0_02]
            at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816) ~[na:1.7.0_02]
            at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337) ~[na:1.7.0_02]
            at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ~[na:1.7.0_02]
            at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:104) ~[infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:442) ~[infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:153) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:177) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:222) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:209) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:192) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:185) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:188) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:149) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:190) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:133) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:68) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:201) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.StateTransferLockInterceptor.visitPutKeyValueCommand(StateTransferLockInterceptor.java:131) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:108) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:934) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.CacheImpl.put(CacheImpl.java:647) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.CacheImpl.put(CacheImpl.java:639) [infinispan-core.jar:5.1.0.CR2]
            at org.infinispan.CacheSupport.put(CacheSupport.java:52) [infinispan-core.jar:5.1.0.CR2]
            at infinispan.test.Transaction.startTransaction(Transaction.java:57) [classes/:na]
            at infinispan.test.Transaction.run(Transaction.java:26) [classes/:na]
            at java.lang.Thread.run(Thread.java:722) [na:1.7.0_02]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
            at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:116)
            at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
            at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61)
            at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
            at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
            at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:934)
            at org.infinispan.CacheImpl.put(CacheImpl.java:647)
            at org.infinispan.CacheImpl.put(CacheImpl.java:639)
            at org.infinispan.CacheSupport.put(CacheSupport.java:52)
            at infinispan.test.Transaction.startTransaction(Transaction.java:57)
            at infinispan.test.Transaction.run(Transaction.java:26)
            at java.lang.Thread.run(Thread.java:722)

    ***************************************************************************************************************************************************************************************************************************************

  • 10. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Galder Zamarreño Master

    @Liron, in your <global> section, add the following and try again:

     

          <asyncTransportExecutor factory="org.infinispan.executors.DefaultExecutorFactory">

             <properties>

                <property name="maxThreads" value="200"/>

                <property name="threadNamePrefix" value="AsyncSerializationThread"/>

             </properties>

          </asyncTransportExecutor>

     

    Basically, increase the async transport (or async marshalling) thread pool cos you're max'ing it out.

  • 11. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Galder Zamarreño Master

    By the way, we're working on coming up with better executor defaults so that Infinispan is easier to consume out of the box.

  • 12. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Liron Tepper Newbie

    I've added the asyncTransportExecutor section that you've mentioned, and set my cache to

    <

    async asyncMarshalling="true"

    />

     

    Still, I'm getting this error:

    ***************************************************************************************************************************************************************************************************************************************

    2012-03-01 13:14:24,967 ERROR [Thread-52] o.i.interceptors.InvocationContextInterceptor - ISPN000136: Execution error
    org.infinispan.CacheException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f74d9e8 rejected from java.util.concurrent.ThreadPoolExecutor@4120bda[Running, pool size = 200, active threads = 200, queued tasks = 100000, completed tasks = 7565]

    ***************************************************************************************************************************************************************************************************************************************

     

    Also I've tried it also with

    <async asyncMarshalling="true"/>

     

    And got the following result:

    ***************************************************************************************************************************************************************************************************************************************

    2012-03-01 13:24:50,434 INFO [Thread-43] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.
    2012-03-01 13:24:50,434 INFO [Thread-43] infinispan.test.Transaction - Run time is 393540 milliseconds.

     

    2012-03-01 13:24:50,434 INFO [Thread-43] infinispan.test.Transaction - Amount of transactions in one millisecond is 15.0

    ***************************************************************************************************************************************************************************************************************************************

    Which is a very slight performance improvement.

    Regards,

    Liron Tepper

  • 13. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Galder Zamarreño Master

    Liron, I dunno what's the difference between the first and the second option.

     

    Would ot be possible for you to send us the ehcache version of the test so that we can examine it and see what's being compared exactly?

  • 14. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
    Galder Zamarreño Master

    Also Liron, you could be having network issues, so I'd suggest you try the TCP version of JGroups configuration:

    https://github.com/infinispan/infinispan/blob/5.1.x/core/src/main/resources/jgroups-tcp.xml

     

    MPING should discover your nodes using multicast. When you run the test, simply pass -Djgroups.bind_addr=x.x.x.x to each of the machines and that way you don't need to modify the config.

     

    I've had them myself in the past...

1 2 3 Previous Next