5 Replies Latest reply: May 11, 2012 11:27 AM by Galder Zamarreño RSS

ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError

dex chen Novice

We are seeing a lot of ISPN000136, ISPN000172 errors in a 3 node cluster in replication with TCP (Infinispan 5.1.3.Final). After a while of repeating these errors, it starts to throw OutOfMemoryError as shown below:

(Note: I posted a few similar or related questions to the forum).

 

ERROR org.infinispan.interceptors.InvocationContextInterceptor[141] - ISPN000136: Execution error

org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 55

 

ISPN000172: Failed to prepare view CacheView ....

 

ERROR org.infinispan.remoting.rpc.RpcManagerImpl[156] - ISPN000073: Unexpected error while replicating

java.lang.OutOfMemoryError: Java heap space ..

 

..

and

 

RROR org.infinispan.cacheviews.CacheViewInfo[143] - ISPN000164: Rolling back to cache view 80, but last committed view is 85

2012-04-20/22:44:21.432/EDT [Timer-3,x] ERROR org.jgroups.util.TimeScheduler2[538] - failed executing task FixedIntervalTask: task=org.jgroups.protocols.UNICAST2$1@3e7c272a, cancelled=false

java.lang.OutOfMemoryError: GC overhead limit exceeded

 

The systems are having 8GB RAM and the JVM --Xmx is set to 5.4 GB.

 

Does Infinispan creats temporay objects in memory during failed state transfer and vew rollback?

 

Of course, what are the possible causes which lead to this?

  • 1. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
    Galder Zamarreño Master

    Same thing:

     

    JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/java_heap"

    or

    JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/z/Tmp/java_heap"

     

    OOME -> generate dump -> upload it -> send us the link

  • 2. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
    dex chen Novice

    Galder:  the OutOfMemory issue hit again on the same IDR. The  JVM/Tomcat does not crash. but the log from Infinispan says:

     

    2012-05-07/12:38:59.383/EDT [transport-thread-18] ERROR org.infinispan.remoting.rpc.RpcManagerImpl[156] - ISPN000073: Unexpected error while replicating

    java.lang.OutOfMemoryError: Java heap space

            at org.jboss.marshalling.util.IdentityIntMap.resize(IdentityIntMap.java:167)

            at org.jboss.marshalling.util.IdentityIntMap.put(IdentityIntMap.java:146)

            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:878)

            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:805)

            at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1057)

            at org.jboss.marshalling.river.RiverObjectOutputStream.defaultWriteObject(RiverObjectOutputStream.java:159)

            at java.util.concurrent.ConcurrentHashMap.writeObject(ConcurrentHashMap.java:1442)

            at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

            at java.lang.reflect.Method.invoke(Method.java:601)

            at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:173)

            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1001)

            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:879)

            at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1057)

            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1013)

            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:879)

            at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

            at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

            at org.infinispan.marshall.MarshallUtil.marshallMap(MarshallUtil.java:60)

     

    I did a java stack dump from the running instance.

    I attached dump here.

  • 3. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
    dex chen Novice

    Did anyone take a look at the dump?  I saw a few threads were blocked in the dump. I'll try to provide more info if need. thanks.

  • 4. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
    Galder Zamarreño Master

    Dude, that's a thread dump, not a heap dump

     

    If you add the options I suggested earlier, when the JVM crashes with an OOME, one will be generated for you.

     

    Google is your friend...

  • 5. Re: ISPN000136 (Execution error), ISPN000172 (Failed to prepare view) and OutOfMemoryError
    Galder Zamarreño Master

    It doesn't matter that the JVM crashes. With HeapDumpOnOutOfMemoryError, if OOME, then heap dump.