5 Replies Latest reply on May 11, 2012 11:27 AM by galder.zamarreno

    ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError

    dex80526

      We are seeing a lot of ISPN000136, ISPN000172 errors in a 3 node cluster in replication with TCP (Infinispan 5.1.3.Final). After a while of repeating these errors, it starts to throw OutOfMemoryError as shown below:

      (Note: I posted a few similar or related questions to the forum).

       

      ERROR org.infinispan.interceptors.InvocationContextInterceptor[141] - ISPN000136: Execution error

      org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 55

       

      ISPN000172: Failed to prepare view CacheView ....

       

      ERROR org.infinispan.remoting.rpc.RpcManagerImpl[156] - ISPN000073: Unexpected error while replicating

      java.lang.OutOfMemoryError: Java heap space ..

       

      ..

      and

       

      RROR org.infinispan.cacheviews.CacheViewInfo[143] - ISPN000164: Rolling back to cache view 80, but last committed view is 85

      2012-04-20/22:44:21.432/EDT [Timer-3,x] ERROR org.jgroups.util.TimeScheduler2[538] - failed executing task FixedIntervalTask: task=org.jgroups.protocols.UNICAST2$1@3e7c272a, cancelled=false

      java.lang.OutOfMemoryError: GC overhead limit exceeded

       

      The systems are having 8GB RAM and the JVM --Xmx is set to 5.4 GB.

       

      Does Infinispan creats temporay objects in memory during failed state transfer and vew rollback?

       

      Of course, what are the possible causes which lead to this?

        • 1. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
          galder.zamarreno

          Same thing:

           

          JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/java_heap"

          or

          JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/z/Tmp/java_heap"

           

          OOME -> generate dump -> upload it -> send us the link

          • 2. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
            dex80526

            Galder:  the OutOfMemory issue hit again on the same IDR. The  JVM/Tomcat does not crash. but the log from Infinispan says:

             

            2012-05-07/12:38:59.383/EDT [transport-thread-18] ERROR org.infinispan.remoting.rpc.RpcManagerImpl[156] - ISPN000073: Unexpected error while replicating

            java.lang.OutOfMemoryError: Java heap space

                    at org.jboss.marshalling.util.IdentityIntMap.resize(IdentityIntMap.java:167)

                    at org.jboss.marshalling.util.IdentityIntMap.put(IdentityIntMap.java:146)

                    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:878)

                    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:805)

                    at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1057)

                    at org.jboss.marshalling.river.RiverObjectOutputStream.defaultWriteObject(RiverObjectOutputStream.java:159)

                    at java.util.concurrent.ConcurrentHashMap.writeObject(ConcurrentHashMap.java:1442)

                    at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                    at java.lang.reflect.Method.invoke(Method.java:601)

                    at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:173)

                    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1001)

                    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:879)

                    at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1057)

                    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1013)

                    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:879)

                    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

                    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

                    at org.infinispan.marshall.MarshallUtil.marshallMap(MarshallUtil.java:60)

             

            I did a java stack dump from the running instance.

            I attached dump here.

            • 3. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
              dex80526

              Did anyone take a look at the dump?  I saw a few threads were blocked in the dump. I'll try to provide more info if need. thanks.

              • 4. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
                galder.zamarreno

                Dude, that's a thread dump, not a heap dump

                 

                If you add the options I suggested earlier, when the JVM crashes with an OOME, one will be generated for you.

                 

                Google is your friend...

                • 5. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
                  galder.zamarreno

                  It doesn't matter that the JVM crashes. With HeapDumpOnOutOfMemoryError, if OOME, then heap dump.