3 Replies Latest reply: Apr 26, 2012 7:04 AM by Galder Zamarreño RSS

Getting  ISPN000172 errors in clustered environment in CR4

Sandeep Kumar Newbie

We are using tcp and tcping for discovery and we are trying to run performance tests using a scripts....

Infinispan version 5.1 CR4

We start caches using startCaches api and do not use configuration file.

We get the following errors when running:

ERROR [org.infispan.cacheviews.CacheViewsManagerImpl] ISPN000172: Failed to prepare view Cachiew{viewId=7, members=[tcpf-02-9025, tcpf-01-59958]} for cache  Cach98, rolling back to view CacheView{viewId=3, members=[tcpf-02-9025]}

  1. java.util.concurrent.TimeoutException

at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)

at java.util.concurrent.FutureTask.get(FutureTask.java:91)

at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(cheViewsManagerImpl.java:303)

at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(cheViewsManagerImpl.java:249)

at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTascall(CacheViewsManagerImpl.java:856)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

at java.util.concurrent.FutureTask.run(FutureTask.java:138)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut.java:908)

at java.lang.Thread.run(Thread.java:619)

  • 1. Re: Getting  ISPN000172 errors in clustered environment in CR4
    Dan Berindei Apprentice

    Sandeep, is this the only error you are seeing in the logs? This one isn't very helpful I'm afraid...

     

    Could you either upload your test code or run the test again with TRACE logging enabled for the org.infinispan category and upload the (zipped) log?

  • 2. Re: Getting  ISPN000172 errors in clustered environment in CR4
    dex chen Novice

    We are seeing the similar issue in 5.1.3.Final. I posted exception  in other thread.

     

    In our case, later on we see OutOfMemoryError from RpcManagerImpl:

    2012-04-20/22:13:53.821/EDT [transport-thread-2] ERROR org.infinispan.remoting.rpc.RpcManagerImpl[156] - ISPN000073: Unexpected error while replicating

    java.lang.OutOfMemoryError: Java heap space

     

    What casue this OutOfMemoryError?

     

    The JVM has a big heap space in this case.

     

    Here is the exception we have:

    2012-04-20/21:31:49.397/EDT [CacheViewTrigger,node1-x] ERROR org.infinispan.cacheviews.CacheViewInfo[143] - ISPN000164: Rolling back to cache view 54, but last committed view is 57

    2012-04-20/21:56:39.586/EDT [CacheViewInstaller-6,node1-x] ERROR org.infinispan.cacheviews.CacheViewsManagerImpl[264] - ISPN000172: Failed to prepare view CacheView{viewId=69, members=[node2-x, node1-x} for cache  session, rolling back to view CacheView{viewId=58, members=[node1-x]}

    java.util.concurrent.TimeoutException

            at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)

            at java.util.concurrent.FutureTask.get(FutureTask.java:119)

            at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:322)

            at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)

            at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)

            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

            at java.util.concurrent.FutureTask.run(FutureTask.java:166)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

            at java.lang.Thread.run(Thread.java:722)

    2012-04-20/22:02:37.989/EDT [CacheViewInstaller-6,node1-x] ERROR org.infinispan.cacheviews.CacheViewInfo[143] - ISPN000164: Rolling back to cache view 58, but last committed view is 72

    2012-04-20/22:07:49.216/EDT [CacheViewInstaller-9,node1-x] ERROR org.infinispan.cacheviews.CacheViewsManagerImpl[264] - ISPN000172: Failed to prepare view CacheView{viewId=74, members=[node2-x, node1-x]} for cache  session, rolling back to view CacheView{viewId=73, members=node1-x]}

    java.util.concurrent.TimeoutException

            at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)

            at java.util.concurrent.FutureTask.get(FutureTask.java:119)

            at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:322)

            at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)

            at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)

            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

            at java.util.concurrent.FutureTask.run(FutureTask.java:166)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

            at java.lang.Thread.run(Thread.java:722)

    2012-04-20/22:13:53.821/EDT [transport-thread-2] ERROR org.infinispan.remoting.rpc.RpcManagerImpl[156] - ISPN000073: Unexpected error while replicating

    java.lang.OutOfMemoryError: Java heap space

            at org.infinispan.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:91)

            at org.jboss.marshalling.OutputStreamByteOutput.write(OutputStreamByteOutput.java:56)

            at org.jboss.marshalling.SimpleDataOutput.flush(SimpleDataOutput.java:311)

            at org.jboss.marshalling.SimpleDataOutput.finish(SimpleDataOutput.java:353)

            at org.jboss.marshalling.AbstractMarshaller.finish(AbstractMarshaller.java:134)

            at org.infinispan.marshall.jboss.ExtendedRiverMarshaller.finish(ExtendedRiverMarshaller.java:47)

            at org.infinispan.marshall.jboss.AbstractJBossMarshaller.finishObjectOutput(AbstractJBossMarshaller.java:116)

            at org.infinispan.marshall.VersionAwareMarshaller.finishObjectOutput(VersionAwareMarshaller.java:

  • 3. Re: Getting  ISPN000172 errors in clustered environment in CR4
    Galder Zamarreño Master

    Dex, if you get any OOME, add these options to generate a heap dump:

     

    JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/java_heap"

    or

    JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/z/Tmp/java_heap"

     

    We can then take the dump (from dropbox?) and analise it. Thx