3 Replies Latest reply on Apr 26, 2012 7:04 AM by galder.zamarreno

    Getting  ISPN000172 errors in clustered environment in CR4

    sandkuma

      We are using tcp and tcping for discovery and we are trying to run performance tests using a scripts....

      Infinispan version 5.1 CR4

      We start caches using startCaches api and do not use configuration file.

      We get the following errors when running:

      ERROR [org.infispan.cacheviews.CacheViewsManagerImpl] ISPN000172: Failed to prepare view Cachiew{viewId=7, members=[tcpf-02-9025, tcpf-01-59958]} for cache  Cach98, rolling back to view CacheView{viewId=3, members=[tcpf-02-9025]}

      1. java.util.concurrent.TimeoutException

      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)

      at java.util.concurrent.FutureTask.get(FutureTask.java:91)

      at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(cheViewsManagerImpl.java:303)

      at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(cheViewsManagerImpl.java:249)

      at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTascall(CacheViewsManagerImpl.java:856)

      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

      at java.util.concurrent.FutureTask.run(FutureTask.java:138)

      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExutor.java:886)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut.java:908)

      at java.lang.Thread.run(Thread.java:619)

        • 1. Re: Getting  ISPN000172 errors in clustered environment in CR4
          dan.berindei

          Sandeep, is this the only error you are seeing in the logs? This one isn't very helpful I'm afraid...

           

          Could you either upload your test code or run the test again with TRACE logging enabled for the org.infinispan category and upload the (zipped) log?

          • 2. Re: Getting  ISPN000172 errors in clustered environment in CR4
            dex80526

            We are seeing the similar issue in 5.1.3.Final. I posted exception  in other thread.

             

            In our case, later on we see OutOfMemoryError from RpcManagerImpl:

            2012-04-20/22:13:53.821/EDT [transport-thread-2] ERROR org.infinispan.remoting.rpc.RpcManagerImpl[156] - ISPN000073: Unexpected error while replicating

            java.lang.OutOfMemoryError: Java heap space

             

            What casue this OutOfMemoryError?

             

            The JVM has a big heap space in this case.

             

            Here is the exception we have:

            2012-04-20/21:31:49.397/EDT [CacheViewTrigger,node1-x] ERROR org.infinispan.cacheviews.CacheViewInfo[143] - ISPN000164: Rolling back to cache view 54, but last committed view is 57

            2012-04-20/21:56:39.586/EDT [CacheViewInstaller-6,node1-x] ERROR org.infinispan.cacheviews.CacheViewsManagerImpl[264] - ISPN000172: Failed to prepare view CacheView{viewId=69, members=[node2-x, node1-x} for cache  session, rolling back to view CacheView{viewId=58, members=[node1-x]}

            java.util.concurrent.TimeoutException

                    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)

                    at java.util.concurrent.FutureTask.get(FutureTask.java:119)

                    at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:322)

                    at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)

                    at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)

                    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

                    at java.util.concurrent.FutureTask.run(FutureTask.java:166)

                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

                    at java.lang.Thread.run(Thread.java:722)

            2012-04-20/22:02:37.989/EDT [CacheViewInstaller-6,node1-x] ERROR org.infinispan.cacheviews.CacheViewInfo[143] - ISPN000164: Rolling back to cache view 58, but last committed view is 72

            2012-04-20/22:07:49.216/EDT [CacheViewInstaller-9,node1-x] ERROR org.infinispan.cacheviews.CacheViewsManagerImpl[264] - ISPN000172: Failed to prepare view CacheView{viewId=74, members=[node2-x, node1-x]} for cache  session, rolling back to view CacheView{viewId=73, members=node1-x]}

            java.util.concurrent.TimeoutException

                    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)

                    at java.util.concurrent.FutureTask.get(FutureTask.java:119)

                    at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:322)

                    at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)

                    at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876)

                    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

                    at java.util.concurrent.FutureTask.run(FutureTask.java:166)

                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

                    at java.lang.Thread.run(Thread.java:722)

            2012-04-20/22:13:53.821/EDT [transport-thread-2] ERROR org.infinispan.remoting.rpc.RpcManagerImpl[156] - ISPN000073: Unexpected error while replicating

            java.lang.OutOfMemoryError: Java heap space

                    at org.infinispan.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:91)

                    at org.jboss.marshalling.OutputStreamByteOutput.write(OutputStreamByteOutput.java:56)

                    at org.jboss.marshalling.SimpleDataOutput.flush(SimpleDataOutput.java:311)

                    at org.jboss.marshalling.SimpleDataOutput.finish(SimpleDataOutput.java:353)

                    at org.jboss.marshalling.AbstractMarshaller.finish(AbstractMarshaller.java:134)

                    at org.infinispan.marshall.jboss.ExtendedRiverMarshaller.finish(ExtendedRiverMarshaller.java:47)

                    at org.infinispan.marshall.jboss.AbstractJBossMarshaller.finishObjectOutput(AbstractJBossMarshaller.java:116)

                    at org.infinispan.marshall.VersionAwareMarshaller.finishObjectOutput(VersionAwareMarshaller.java:

            • 3. Re: Getting  ISPN000172 errors in clustered environment in CR4
              galder.zamarreno

              Dex, if you get any OOME, add these options to generate a heap dump:

               

              JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/java_heap"

              or

              JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/z/Tmp/java_heap"

               

              We can then take the dump (from dropbox?) and analise it. Thx