-
1. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
sannegrinovero Feb 28, 2012 1:12 PM (in response to ltepper)Hi Liron,
indeed during the early 5.1 CRs we spotted some severe performance problems; the performance issues where all fixed before the Final, and we kept improving performance even more every day, so the fastest should be version 5.1.2.CR2 (released yesterday).
Would you mind trying an update? Please update both Infinispan and JGroups.
-
2. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
sannegrinovero Feb 28, 2012 1:24 PM (in response to sannegrinovero)Another thing: if your test runs only 6 million transactions it will finish way before the JVM has warmed up properly: on an average 4 cores Xeons it can peak above the million read transactions per second, but it takes at least 20 minutes of high load before it gets to that. (will be slower for writes: with replication read is always a local lookup).
A comment on your code: I don't see you starting transactions? Consider Infinispan understands "real" JTA transactions, so you could get even better performance wrapping your startTransaction method in a transaction, as it will batch all update operations across a reduced amount of wire commands.
-
3. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
sannegrinovero Feb 28, 2012 1:31 PM (in response to sannegrinovero)btw doing 13 transactions per millisecond means 13,000 transactions per second.
I assume you have a typo, as you say ehcache is doing just 275 transactions per second.. looks like too good for Infinispan to be true.
-
4. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
ltepper Feb 29, 2012 3:05 AM (in response to sannegrinovero)Hi Sanne,
First, thank you very much for your replies.
I have a typo indeed - I meant to write 275 transactions per millisecond in ehcach.
I will try infiniSpan version 5.1.2.CR2 with the latest JGroups version.
Also I will try to use real transactions, run a long test over night and post results.
Regards,
Liron Tepper.
-
5. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
sannegrinovero Feb 29, 2012 8:32 AM (in response to ltepper)thanks! Looking forward to see them. We can always improve something
-
6. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
ltepper Feb 29, 2012 10:11 AM (in response to sannegrinovero)Here are the new results I got:
1) Before placing any new JAR, a long 60 million transactions run:
Logs:
**********************************************************************************************************************************************************************************************
2012-02-29 11:37:17,840 INFO [Thread-15] infinispan.test.Transaction - 60 threads run 1000000 transactions per thread => total of 60000000 transactions.
2012-02-29 11:37:17,840 INFO [Thread-15] infinispan.test.Transaction - Run time is 4954327 milliseconds.
2012-02-29 11:37:17,840 INFO [Thread-15] infinispan.test.Transaction - Amount of transactions in one millisecond is 12.0
**********************************************************************************************************************************************************************************************
=> As you can see from logs above, a long run of 82 minutes showed a very slight performance descending from 13 transactions per millisecond to 12.
2) With new JARs (infiniSpan version 5.1.2.CR1 with jgroups-3.0.6.Final.jar) and a short 6 Million transactions run:
Logs:
**********************************************************************************************************************************************************************************************
2012-02-29 14:16:02,832 INFO [Thread-48] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.
2012-02-29 14:16:02,832 INFO [Thread-48] infinispan.test.Transaction - Run time is 382628 milliseconds.
2012-02-29 14:16:02,833 INFO [Thread-48] infinispan.test.Transaction - Average transaction time is 0.0
2012-02-29 14:16:02,833 INFO [Thread-48] infinispan.test.Transaction - Amount of transactions in one millisecond is 15.0
**********************************************************************************************************************************************************************************************
=> Using new jars brought us to 15 transactions per second, which is a slight improvement only.
3) With real transactions enabled, and a short 6 Million transactions run (and the new JARs) -
**********************************************************************************************************************************************************************************************
2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.
2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - Run time is 425982 milliseconds.
2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - Average transaction time is 0.0
2012-02-29 15:02:40,489 INFO [Thread-50] infinispan.test.Transaction - Amount of transactions in one millisecond is 14.0**********************************************************************************************************************************************************************************************
=> There wasn't any performance change in this test.
The code changes I've made are as followes:
============================================================================================================================
private void startTransaction(Integer index) {
MyObject myObject = MyObjectFactory.getMyObject();boolean batchStarted = RunLoadTest.testCache.startBatch();
RunLoadTest.testCache.get(index);
RunLoadTest.testCache.get(index);
// RunLoadTest.testCache.putAsync(index, myObject);
RunLoadTest.testCache.put(index, myObject); //sync !
RunLoadTest.testCache.get(index);
RunLoadTest.testCache.get(index);
RunLoadTest.testCache.get(index);
RunLoadTest.testCache.get(index);
// RunLoadTest.testCache.removeAsync(index);
RunLoadTest.testCache.remove(index); // sync !
RunLoadTest.testCache.get(index);
RunLoadTest.testCache.get(index);
RunLoadTest.testCache.get(index);
RunLoadTest.testCache.get(index);
RunLoadTest.testCache.endBatch(batchStarted);
if (!batchStarted) {
LogUtils.warn(getClass(), "Batch was not started !");
}
}============================================================================================================================
And also I've added the following to infispan.xml:
============================================================================================================================
<default>
<invocationBatching enabled="true" />
</default>
============================================================================================================================
Any other ideas for improving this would be much appreciated !
Regards,
Liron Tepper
-
7. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
manik Feb 29, 2012 1:34 PM (in response to ltepper)I haven't had a look at your tests as yet, but can you confirm that in both cases you are getting synchronous, 2-phase transactions? This is the default in Infinispan and I know that it is *not* the default in EHCache. Similarly, EHCache defaults to asynchronous serialization which again isn't the default in Infinispan. Just making sure you're doing an apples-to-apples comparison.
-
8. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
ltepper Mar 1, 2012 3:39 AM (in response to manik)Hi Manik,
This is the configuration I'm using for ehCache:
******************************************************************************************************************************************************
<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="ehcache.xsd"
updateCheck="true" monitoring="autodetect"
dynamicConfig="true">
<diskStore path="java.io.tmpdir"/><transactionManagerLookup class="net.sf.ehcache.transaction.manager.DefaultTransactionManagerLookup"
properties="jndiName=java:/TransactionManager" propertySeparator=";"/><cacheManagerEventListenerFactory class="" properties=""/>
<!-- Liron: This part is added for working on RMI replication mode -->
<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=manual,
rmiUrls=//server2:40001/allotIpSessionCache"/><cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="port=40001,
socketTimeoutMillis=2000"/><!-- Liron: End of section. -->
<cache name="allotIpSessionCache" maxElementsInMemory="0"
eternal="true" overflowToDisk="true" diskPersistent="false"
memoryStoreEvictionPolicy="LFU">
<cacheEventListenerFactory
class="net.sf.ehcache.distribution.RMICacheReplicatorFactory" />
<bootstrapCacheLoaderFactory
class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory"
properties="bootstrapAsynchronously=false" />
</cache></ehcache>
******************************************************************************************************************************************************
With infiniSpan, if I set asyncMarshalling to true, I'm getting in a few seconds to a situation in which the infiniSpan internal RMI queue gets exploded.
Also documentation says "it can suffer from reordering of operations", so I can't use it.
What is the default for transactions with ehCache? And in which way does infiniSpan differs from that? When I used transaction with infiniSpan I didn't see any improvment,
and anyway I know that in our production process we will not be able to use transactions.
Regards,
Liron Tepper
-
9. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
ltepper Mar 1, 2012 4:12 AM (in response to ltepper)Correction - I meant RPC queues, not RMI queues.
These are the excptions I got:
***************************************************************************************************************************************************************************************************************************************
2012-02-28 19:00:20,033 ERROR [Thread-13] org.infinispan.remoting.rpc.RpcManagerImpl - ISPN000073: Unexpected error while replicating
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@41e0c4c4 rejected from java.util.concurrent.ThreadPoolExecutor@4bfb03f0[Running, pool size = 25, active threads = 25, queued tasks = 100000, completed tasks = 4992]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2001) ~[na:1.7.0_02]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816) ~[na:1.7.0_02]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337) ~[na:1.7.0_02]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ~[na:1.7.0_02]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:104) ~[infinispan-core.jar:5.1.0.CR2]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:442) ~[infinispan-core.jar:5.1.0.CR2]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:153) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:177) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:222) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:209) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:192) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:185) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:188) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:149) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:190) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:133) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:68) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:201) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.StateTransferLockInterceptor.visitPutKeyValueCommand(StateTransferLockInterceptor.java:131) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:108) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:116) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:934) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.CacheImpl.put(CacheImpl.java:647) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.CacheImpl.put(CacheImpl.java:639) [infinispan-core.jar:5.1.0.CR2]
at org.infinispan.CacheSupport.put(CacheSupport.java:52) [infinispan-core.jar:5.1.0.CR2]
at infinispan.test.Transaction.startTransaction(Transaction.java:57) [classes/:na]
at infinispan.test.Transaction.run(Transaction.java:26) [classes/:na]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_02]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:116)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:934)
at org.infinispan.CacheImpl.put(CacheImpl.java:647)
at org.infinispan.CacheImpl.put(CacheImpl.java:639)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
at infinispan.test.Transaction.startTransaction(Transaction.java:57)
at infinispan.test.Transaction.run(Transaction.java:26)
at java.lang.Thread.run(Thread.java:722)***************************************************************************************************************************************************************************************************************************************
-
10. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
galder.zamarreno Mar 1, 2012 6:02 AM (in response to ltepper)@Liron, in your <global> section, add the following and try again:
<asyncTransportExecutor factory="org.infinispan.executors.DefaultExecutorFactory">
<properties>
<property name="maxThreads" value="200"/>
<property name="threadNamePrefix" value="AsyncSerializationThread"/>
</properties>
</asyncTransportExecutor>
Basically, increase the async transport (or async marshalling) thread pool cos you're max'ing it out.
-
11. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
galder.zamarreno Mar 1, 2012 6:15 AM (in response to galder.zamarreno)By the way, we're working on coming up with better executor defaults so that Infinispan is easier to consume out of the box.
-
12. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
ltepper Mar 1, 2012 6:28 AM (in response to galder.zamarreno)I've added the asyncTransportExecutor section that you've mentioned, and set my cache to
<
async asyncMarshalling="true"
/>
Still, I'm getting this error:
***************************************************************************************************************************************************************************************************************************************
2012-03-01 13:14:24,967 ERROR [Thread-52] o.i.interceptors.InvocationContextInterceptor - ISPN000136: Execution error
org.infinispan.CacheException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f74d9e8 rejected from java.util.concurrent.ThreadPoolExecutor@4120bda[Running, pool size = 200, active threads = 200, queued tasks = 100000, completed tasks = 7565]***************************************************************************************************************************************************************************************************************************************
Also I've tried it also with
<async asyncMarshalling="true"/>
And got the following result:
***************************************************************************************************************************************************************************************************************************************
2012-03-01 13:24:50,434 INFO [Thread-43] infinispan.test.Transaction - 60 threads run 100000 transactions per thread => total of 6000000 transactions.
2012-03-01 13:24:50,434 INFO [Thread-43] infinispan.test.Transaction - Run time is 393540 milliseconds.2012-03-01 13:24:50,434 INFO [Thread-43] infinispan.test.Transaction - Amount of transactions in one millisecond is 15.0
***************************************************************************************************************************************************************************************************************************************
Which is a very slight performance improvement.
Regards,
Liron Tepper
-
13. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
galder.zamarreno Mar 1, 2012 9:53 AM (in response to ltepper)Liron, I dunno what's the difference between the first and the second option.
Would ot be possible for you to send us the ehcache version of the test so that we can examine it and see what's being compared exactly?
-
14. Re: InfiniSpan 5.1.0CR2 with Jgroups 3.0.1 performance
galder.zamarreno Mar 1, 2012 9:56 AM (in response to galder.zamarreno)Also Liron, you could be having network issues, so I'd suggest you try the TCP version of JGroups configuration:
https://github.com/infinispan/infinispan/blob/5.1.x/core/src/main/resources/jgroups-tcp.xml
MPING should discover your nodes using multicast. When you run the test, simply pass -Djgroups.bind_addr=x.x.x.x to each of the machines and that way you don't need to modify the config.
I've had them myself in the past...